Quantcast

High Performance Load

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

High Performance Load

Colin DuPlantis
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


Hi, all, I'm bumping into some problems with connections closing under
very heavy load (100k's messages/s). The machine CPU (8-core) is pegged
processing these messages (not QFJ, our business app).

I see, every now and then, a connection close after the counter sends a
35=1. My working assumption right now is that the thread(s) assigned to
the QFJ acceptor/receiver are starving and not able to send 35=0 at
agreed-upon intervals.

Has anybody mucked around with thread affinity or the like to make sure
the QFJ sockets and their minders are fed and happy?

Also, are there any opinions about the benefits of using JDBC message
store or file message store for high performance?

Thanks.

--
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
888.868.4884 +1.541.306.6556
http://www.marketcetera.org


------------------------------------------------------------------------------
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High Performance Load

Robert Engels-2
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


If it is so starved that you cannot respond to the heartbeat then you are not really alive, and so you shouldn’t respond… thus terminating the connection.

Even if you could partition it in such a way to ensure you respond to the heartbeats, what if that connection you are responding to then sends a request - you clearly can’t handle it because you are starved for CPU resources, thus you should not be responding to the heartbeat either…

The only thing in this case is to make your application threads more efficient, and/or better hardware, and/or scale out (more machines).

We use thread affinity and core isolation in our application with QFJ and it works very well, but these techniques are designed to improve latency, not throughput.

You have a throughput problem. Not a latency problem.

Regards,
Robert Engels
OptionsCIty Software Inc.


> On Oct 30, 2015, at 4:20 PM, Colin DuPlantis <[hidden email]> wrote:
>
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> Hi, all, I'm bumping into some problems with connections closing under
> very heavy load (100k's messages/s). The machine CPU (8-core) is pegged
> processing these messages (not QFJ, our business app).
>
> I see, every now and then, a connection close after the counter sends a
> 35=1. My working assumption right now is that the thread(s) assigned to
> the QFJ acceptor/receiver are starving and not able to send 35=0 at
> agreed-upon intervals.
>
> Has anybody mucked around with thread affinity or the like to make sure
> the QFJ sockets and their minders are fed and happy?
>
> Also, are there any opinions about the benefits of using JDBC message
> store or file message store for high performance?
>
> Thanks.
>
> --
> Colin DuPlantis
> Chief Architect, Marketcetera
> Download, Run, Trade
> 888.868.4884 +1.541.306.6556
> http://www.marketcetera.org
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users


------------------------------------------------------------------------------
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High Performance Load

Charles Briquel
In reply to this post by Colin DuPlantis
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


Hi,
You are receiving or sending theses messages ?
Did you look at your gc ?



> Le 30 oct. 2015 à 22:20, Colin DuPlantis <[hidden email]> a écrit :
>
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> Hi, all, I'm bumping into some problems with connections closing under
> very heavy load (100k's messages/s). The machine CPU (8-core) is pegged
> processing these messages (not QFJ, our business app).
>
> I see, every now and then, a connection close after the counter sends a
> 35=1. My working assumption right now is that the thread(s) assigned to
> the QFJ acceptor/receiver are starving and not able to send 35=0 at
> agreed-upon intervals.
>
> Has anybody mucked around with thread affinity or the like to make sure
> the QFJ sockets and their minders are fed and happy?
>
> Also, are there any opinions about the benefits of using JDBC message
> store or file message store for high performance?
>
> Thanks.
>
> --
> Colin DuPlantis
> Chief Architect, Marketcetera
> Download, Run, Trade
> 888.868.4884 +1.541.306.6556
> http://www.marketcetera.org
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users


------------------------------------------------------------------------------
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High Performance Load

Robert Engels-2
In reply to this post by Robert Engels-2
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


In thinking about this some more, my answer may have been too stringent.

Say you did isolate the ‘application processing request threads to 7 of the 8 cores. You could then put all of the fix processing threads on one core. It can easily handle thousands or low volume connections.

As long as the fix processor hands off the handling of the request to the application queues, you will not time out.

BUT, you have to know if that is acceptable…. The user could be firing thousands of orders (or other requests) and they are just backing up in the queues waiting to be processed…

Leading to VERY HIGH latency to the point of the system not being usable.


> On Oct 30, 2015, at 4:32 PM, Robert Engels <[hidden email]> wrote:
>
> If it is so starved that you cannot respond to the heartbeat then you are not really alive, and so you shouldn’t respond… thus terminating the connection.
>
> Even if you could partition it in such a way to ensure you respond to the heartbeats, what if that connection you are responding to then sends a request - you clearly can’t handle it because you are starved for CPU resources, thus you should not be responding to the heartbeat either…
>
> The only thing in this case is to make your application threads more efficient, and/or better hardware, and/or scale out (more machines).
>
> We use thread affinity and core isolation in our application with QFJ and it works very well, but these techniques are designed to improve latency, not throughput.
>
> You have a throughput problem. Not a latency problem.
>
> Regards,
> Robert Engels
> OptionsCIty Software Inc.
>
>
>> On Oct 30, 2015, at 4:20 PM, Colin DuPlantis <[hidden email]> wrote:
>>
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>
>>
>> Hi, all, I'm bumping into some problems with connections closing under
>> very heavy load (100k's messages/s). The machine CPU (8-core) is pegged
>> processing these messages (not QFJ, our business app).
>>
>> I see, every now and then, a connection close after the counter sends a
>> 35=1. My working assumption right now is that the thread(s) assigned to
>> the QFJ acceptor/receiver are starving and not able to send 35=0 at
>> agreed-upon intervals.
>>
>> Has anybody mucked around with thread affinity or the like to make sure
>> the QFJ sockets and their minders are fed and happy?
>>
>> Also, are there any opinions about the benefits of using JDBC message
>> store or file message store for high performance?
>>
>> Thanks.
>>
>> --
>> Colin DuPlantis
>> Chief Architect, Marketcetera
>> Download, Run, Trade
>> 888.868.4884 +1.541.306.6556
>> http://www.marketcetera.org
>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Quickfixj-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>


------------------------------------------------------------------------------
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High Performance Load

Colin DuPlantis
In reply to this post by Charles Briquel
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


Receiving.

GC is not causing issues, AFAIK. Something like 1 major, a few minor,
total time about 2s, say.

All CPUs running 80-90%+

I can reduce CPU consumption by easing up on the number of messages we
process in parallel, but I'd like to see if I can keep that number as
high as possible to maximize throughput. Reducing the number of worker
threads reduces CPU, which makes this problem go away.

On 10/30/2015 02:32 PM, Charles Briquel wrote:

> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> Hi,
> You are receiving or sending theses messages ?
> Did you look at your gc ?
>
>
>
>> Le 30 oct. 2015 à 22:20, Colin DuPlantis <[hidden email]> a écrit :
>>
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>
>>
>> Hi, all, I'm bumping into some problems with connections closing under
>> very heavy load (100k's messages/s). The machine CPU (8-core) is pegged
>> processing these messages (not QFJ, our business app).
>>
>> I see, every now and then, a connection close after the counter sends a
>> 35=1. My working assumption right now is that the thread(s) assigned to
>> the QFJ acceptor/receiver are starving and not able to send 35=0 at
>> agreed-upon intervals.
>>
>> Has anybody mucked around with thread affinity or the like to make sure
>> the QFJ sockets and their minders are fed and happy?
>>
>> Also, are there any opinions about the benefits of using JDBC message
>> store or file message store for high performance?
>>
>> Thanks.
>>
>> --
>> Colin DuPlantis
>> Chief Architect, Marketcetera
>> Download, Run, Trade
>> 888.868.4884 +1.541.306.6556
>> http://www.marketcetera.org
>>
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Quickfixj-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

--
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
888.868.4884 +1.541.306.6556
http://www.marketcetera.org


------------------------------------------------------------------------------
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High Performance Load

Robert Engels-2
In reply to this post by Robert Engels-2
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


One final note, you’ll need to be careful because of the persistent nature of fix messages. Since you are going to hand off the actual processing to be performed somewhere else, but you need to capture the inbound message because you will be responding that you received the message (seqnum advancement), so you will need to be able to recover these messages in case of failure.

If you process the requests synchronously you can be a bit more light-weight in the persistence but it is highly system dependent.


> On Oct 30, 2015, at 4:37 PM, Robert Engels <[hidden email]> wrote:
>
> In thinking about this some more, my answer may have been too stringent.
>
> Say you did isolate the ‘application processing request threads to 7 of the 8 cores. You could then put all of the fix processing threads on one core. It can easily handle thousands or low volume connections.
>
> As long as the fix processor hands off the handling of the request to the application queues, you will not time out.
>
> BUT, you have to know if that is acceptable…. The user could be firing thousands of orders (or other requests) and they are just backing up in the queues waiting to be processed…
>
> Leading to VERY HIGH latency to the point of the system not being usable.
>
>
>> On Oct 30, 2015, at 4:32 PM, Robert Engels <[hidden email]> wrote:
>>
>> If it is so starved that you cannot respond to the heartbeat then you are not really alive, and so you shouldn’t respond… thus terminating the connection.
>>
>> Even if you could partition it in such a way to ensure you respond to the heartbeats, what if that connection you are responding to then sends a request - you clearly can’t handle it because you are starved for CPU resources, thus you should not be responding to the heartbeat either…
>>
>> The only thing in this case is to make your application threads more efficient, and/or better hardware, and/or scale out (more machines).
>>
>> We use thread affinity and core isolation in our application with QFJ and it works very well, but these techniques are designed to improve latency, not throughput.
>>
>> You have a throughput problem. Not a latency problem.
>>
>> Regards,
>> Robert Engels
>> OptionsCIty Software Inc.
>>
>>
>>> On Oct 30, 2015, at 4:20 PM, Colin DuPlantis <[hidden email]> wrote:
>>>
>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>
>>>
>>> Hi, all, I'm bumping into some problems with connections closing under
>>> very heavy load (100k's messages/s). The machine CPU (8-core) is pegged
>>> processing these messages (not QFJ, our business app).
>>>
>>> I see, every now and then, a connection close after the counter sends a
>>> 35=1. My working assumption right now is that the thread(s) assigned to
>>> the QFJ acceptor/receiver are starving and not able to send 35=0 at
>>> agreed-upon intervals.
>>>
>>> Has anybody mucked around with thread affinity or the like to make sure
>>> the QFJ sockets and their minders are fed and happy?
>>>
>>> Also, are there any opinions about the benefits of using JDBC message
>>> store or file message store for high performance?
>>>
>>> Thanks.
>>>
>>> --
>>> Colin DuPlantis
>>> Chief Architect, Marketcetera
>>> Download, Run, Trade
>>> 888.868.4884 +1.541.306.6556
>>> http://www.marketcetera.org
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> _______________________________________________
>>> Quickfixj-users mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>>
>


------------------------------------------------------------------------------
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High Performance Load

Charles Briquel
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


May be another road to explore, I noticed that in our app, the quickfix thread that is receiving messages give them to another one.
Depending on your config, you can have one thread for all or one thread by session that handle messages.
The usage we had of this thread is just to put the message in our queue, in the fromApp method
I made a change to drop this thread and make the receiving thread to directly give the message to our pool of threads.
But before giving messages to the app, it is processed and this work is not done by the receiving thread. And during this work, heartbeats are processed.
I think, but I may be wrong, that there is two threads for I/O, one for sending, the other for receiving.
In this case I should be ok to do like that, and may be it would solve your issue. But your receiving thread will do much more work, this can be a new pbm.


> Le 30 oct. 2015 à 22:42, Robert Engels <[hidden email]> a écrit :
>
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> One final note, you’ll need to be careful because of the persistent nature of fix messages. Since you are going to hand off the actual processing to be performed somewhere else, but you need to capture the inbound message because you will be responding that you received the message (seqnum advancement), so you will need to be able to recover these messages in case of failure.
>
> If you process the requests synchronously you can be a bit more light-weight in the persistence but it is highly system dependent.
>
>
>> On Oct 30, 2015, at 4:37 PM, Robert Engels <[hidden email]> wrote:
>>
>> In thinking about this some more, my answer may have been too stringent.
>>
>> Say you did isolate the ‘application processing request threads to 7 of the 8 cores. You could then put all of the fix processing threads on one core. It can easily handle thousands or low volume connections.
>>
>> As long as the fix processor hands off the handling of the request to the application queues, you will not time out.
>>
>> BUT, you have to know if that is acceptable…. The user could be firing thousands of orders (or other requests) and they are just backing up in the queues waiting to be processed…
>>
>> Leading to VERY HIGH latency to the point of the system not being usable.
>>
>>
>>> On Oct 30, 2015, at 4:32 PM, Robert Engels <[hidden email]> wrote:
>>>
>>> If it is so starved that you cannot respond to the heartbeat then you are not really alive, and so you shouldn’t respond… thus terminating the connection.
>>>
>>> Even if you could partition it in such a way to ensure you respond to the heartbeats, what if that connection you are responding to then sends a request - you clearly can’t handle it because you are starved for CPU resources, thus you should not be responding to the heartbeat either…
>>>
>>> The only thing in this case is to make your application threads more efficient, and/or better hardware, and/or scale out (more machines).
>>>
>>> We use thread affinity and core isolation in our application with QFJ and it works very well, but these techniques are designed to improve latency, not throughput.
>>>
>>> You have a throughput problem. Not a latency problem.
>>>
>>> Regards,
>>> Robert Engels
>>> OptionsCIty Software Inc.
>>>
>>>
>>>> On Oct 30, 2015, at 4:20 PM, Colin DuPlantis <[hidden email]> wrote:
>>>>
>>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>>
>>>>
>>>> Hi, all, I'm bumping into some problems with connections closing under
>>>> very heavy load (100k's messages/s). The machine CPU (8-core) is pegged
>>>> processing these messages (not QFJ, our business app).
>>>>
>>>> I see, every now and then, a connection close after the counter sends a
>>>> 35=1. My working assumption right now is that the thread(s) assigned to
>>>> the QFJ acceptor/receiver are starving and not able to send 35=0 at
>>>> agreed-upon intervals.
>>>>
>>>> Has anybody mucked around with thread affinity or the like to make sure
>>>> the QFJ sockets and their minders are fed and happy?
>>>>
>>>> Also, are there any opinions about the benefits of using JDBC message
>>>> store or file message store for high performance?
>>>>
>>>> Thanks.
>>>>
>>>> --
>>>> Colin DuPlantis
>>>> Chief Architect, Marketcetera
>>>> Download, Run, Trade
>>>> 888.868.4884 +1.541.306.6556
>>>> http://www.marketcetera.org
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> _______________________________________________
>>>> Quickfixj-users mailing list
>>>> [hidden email]
>>>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>>>
>>
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users


------------------------------------------------------------------------------
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High Performance Load

Colin DuPlantis
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


We immediately hand off the messages to another thread, in fact,
hundreds of them. This is in large part why the machine CPU is
completely consumed.

The QFJ thread itself is very lightly used, it's just starving.

Thanks.

On 10/30/15 3:15 PM, Charles Briquel wrote:

> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> May be another road to explore, I noticed that in our app, the quickfix thread that is receiving messages give them to another one.
> Depending on your config, you can have one thread for all or one thread by session that handle messages.
> The usage we had of this thread is just to put the message in our queue, in the fromApp method
> I made a change to drop this thread and make the receiving thread to directly give the message to our pool of threads.
> But before giving messages to the app, it is processed and this work is not done by the receiving thread. And during this work, heartbeats are processed.
> I think, but I may be wrong, that there is two threads for I/O, one for sending, the other for receiving.
> In this case I should be ok to do like that, and may be it would solve your issue. But your receiving thread will do much more work, this can be a new pbm.
>
>
>> Le 30 oct. 2015 à 22:42, Robert Engels <[hidden email]> a écrit :
>>
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>
>>
>> One final note, you’ll need to be careful because of the persistent nature of fix messages. Since you are going to hand off the actual processing to be performed somewhere else, but you need to capture the inbound message because you will be responding that you received the message (seqnum advancement), so you will need to be able to recover these messages in case of failure.
>>
>> If you process the requests synchronously you can be a bit more light-weight in the persistence but it is highly system dependent.
>>
>>
>>> On Oct 30, 2015, at 4:37 PM, Robert Engels <[hidden email]> wrote:
>>>
>>> In thinking about this some more, my answer may have been too stringent.
>>>
>>> Say you did isolate the ‘application processing request threads to 7 of the 8 cores. You could then put all of the fix processing threads on one core. It can easily handle thousands or low volume connections.
>>>
>>> As long as the fix processor hands off the handling of the request to the application queues, you will not time out.
>>>
>>> BUT, you have to know if that is acceptable…. The user could be firing thousands of orders (or other requests) and they are just backing up in the queues waiting to be processed…
>>>
>>> Leading to VERY HIGH latency to the point of the system not being usable.
>>>
>>>
>>>> On Oct 30, 2015, at 4:32 PM, Robert Engels <[hidden email]> wrote:
>>>>
>>>> If it is so starved that you cannot respond to the heartbeat then you are not really alive, and so you shouldn’t respond… thus terminating the connection.
>>>>
>>>> Even if you could partition it in such a way to ensure you respond to the heartbeats, what if that connection you are responding to then sends a request - you clearly can’t handle it because you are starved for CPU resources, thus you should not be responding to the heartbeat either…
>>>>
>>>> The only thing in this case is to make your application threads more efficient, and/or better hardware, and/or scale out (more machines).
>>>>
>>>> We use thread affinity and core isolation in our application with QFJ and it works very well, but these techniques are designed to improve latency, not throughput.
>>>>
>>>> You have a throughput problem. Not a latency problem.
>>>>
>>>> Regards,
>>>> Robert Engels
>>>> OptionsCIty Software Inc.
>>>>
>>>>
>>>>> On Oct 30, 2015, at 4:20 PM, Colin DuPlantis <[hidden email]> wrote:
>>>>>
>>>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>>>
>>>>>
>>>>> Hi, all, I'm bumping into some problems with connections closing under
>>>>> very heavy load (100k's messages/s). The machine CPU (8-core) is pegged
>>>>> processing these messages (not QFJ, our business app).
>>>>>
>>>>> I see, every now and then, a connection close after the counter sends a
>>>>> 35=1. My working assumption right now is that the thread(s) assigned to
>>>>> the QFJ acceptor/receiver are starving and not able to send 35=0 at
>>>>> agreed-upon intervals.
>>>>>
>>>>> Has anybody mucked around with thread affinity or the like to make sure
>>>>> the QFJ sockets and their minders are fed and happy?
>>>>>
>>>>> Also, are there any opinions about the benefits of using JDBC message
>>>>> store or file message store for high performance?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> --
>>>>> Colin DuPlantis
>>>>> Chief Architect, Marketcetera
>>>>> Download, Run, Trade
>>>>> 888.868.4884 +1.541.306.6556
>>>>> http://www.marketcetera.org
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> _______________________________________________
>>>>> Quickfixj-users mailing list
>>>>> [hidden email]
>>>>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Quickfixj-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

--
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
800.819.2928 x101 +1.541.306.6556
http://www.marketcetera.org


------------------------------------------------------------------------------
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High Performance Load

Colin DuPlantis
In reply to this post by Robert Engels-2
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


Thanks for your thoughts, Robert.

We are trying to solve a "liveness" problem, not a throughput or latency
problem. It is understood that a certain piece of hardware can manage a
certain rate of throughput. If that rate is exceeded in a burst, it is
acceptable that latency will increase. What is not acceptable is
dropping the connection in these conditions.

Affining one or more threads to maintain sufficient liveness to prevent
disconnection would probably decrease throughput and latency. This is
also understood and acceptable.

We already manage the persistence of inbound messages in case of
recovery - this part is working fine. We're just trying to
deterministically establish the upper bound of throughput for a certain
hardware configuration. We can throttle the number of simultaneous
threads until the disconnection event no longer occurs, but this is a
probabilistic solution, subject to some degree to the whimsy of
circumstance. My preference is a deterministic solution even at the cost
of a certain percentage of throughput and latency.

On 10/30/15 2:42 PM, Robert Engels wrote:

> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> One final note, you’ll need to be careful because of the persistent nature of fix messages. Since you are going to hand off the actual processing to be performed somewhere else, but you need to capture the inbound message because you will be responding that you received the message (seqnum advancement), so you will need to be able to recover these messages in case of failure.
>
> If you process the requests synchronously you can be a bit more light-weight in the persistence but it is highly system dependent.
>
>
>> On Oct 30, 2015, at 4:37 PM, Robert Engels <[hidden email]> wrote:
>>
>> In thinking about this some more, my answer may have been too stringent.
>>
>> Say you did isolate the ‘application processing request threads to 7 of the 8 cores. You could then put all of the fix processing threads on one core. It can easily handle thousands or low volume connections.
>>
>> As long as the fix processor hands off the handling of the request to the application queues, you will not time out.
>>
>> BUT, you have to know if that is acceptable…. The user could be firing thousands of orders (or other requests) and they are just backing up in the queues waiting to be processed…
>>
>> Leading to VERY HIGH latency to the point of the system not being usable.
>>
>>
>>> On Oct 30, 2015, at 4:32 PM, Robert Engels <[hidden email]> wrote:
>>>
>>> If it is so starved that you cannot respond to the heartbeat then you are not really alive, and so you shouldn’t respond… thus terminating the connection.
>>>
>>> Even if you could partition it in such a way to ensure you respond to the heartbeats, what if that connection you are responding to then sends a request - you clearly can’t handle it because you are starved for CPU resources, thus you should not be responding to the heartbeat either…
>>>
>>> The only thing in this case is to make your application threads more efficient, and/or better hardware, and/or scale out (more machines).
>>>
>>> We use thread affinity and core isolation in our application with QFJ and it works very well, but these techniques are designed to improve latency, not throughput.
>>>
>>> You have a throughput problem. Not a latency problem.
>>>
>>> Regards,
>>> Robert Engels
>>> OptionsCIty Software Inc.
>>>
>>>
>>>> On Oct 30, 2015, at 4:20 PM, Colin DuPlantis <[hidden email]> wrote:
>>>>
>>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>>
>>>>
>>>> Hi, all, I'm bumping into some problems with connections closing under
>>>> very heavy load (100k's messages/s). The machine CPU (8-core) is pegged
>>>> processing these messages (not QFJ, our business app).
>>>>
>>>> I see, every now and then, a connection close after the counter sends a
>>>> 35=1. My working assumption right now is that the thread(s) assigned to
>>>> the QFJ acceptor/receiver are starving and not able to send 35=0 at
>>>> agreed-upon intervals.
>>>>
>>>> Has anybody mucked around with thread affinity or the like to make sure
>>>> the QFJ sockets and their minders are fed and happy?
>>>>
>>>> Also, are there any opinions about the benefits of using JDBC message
>>>> store or file message store for high performance?
>>>>
>>>> Thanks.
>>>>
>>>> --
>>>> Colin DuPlantis
>>>> Chief Architect, Marketcetera
>>>> Download, Run, Trade
>>>> 888.868.4884 +1.541.306.6556
>>>> http://www.marketcetera.org
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> _______________________________________________
>>>> Quickfixj-users mailing list
>>>> [hidden email]
>>>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

--
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
800.819.2928 x101 +1.541.306.6556
http://www.marketcetera.org


------------------------------------------------------------------------------
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High Performance Load

Robert Engels-2
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


I assume that “liveness” is some sort of combination of latency and throughput, but I’m not familiar with any specific measurement in this regard.

Without even using affinity, you can on most (some?) platforms achieve what you desire by just using proper thread priorities. By making the FIX & IO threads higher priority than the application worker threads, you will achieve what you desire, and reach the highest level of throughput and concurrent connections possible.

You will need to review the literature on your JVM if/how the ‘java thread’ priorities map to native thread priorities. If your JVM does not support this, then you can do it with native code.

Your other option is to use a pool of worker threads, capped at 7 (or number of cores - number of IO threads), then you will always have a cpu available to respond to fix/IO. The IO decoding and processing (and FIX) is usually a small fraction of the resources required by an application “event” - but maybe not in your case.

But, I suggest you look at the problem using limits. As long as you can calculate the avg cost of an IO receive and decode and send, and the cost of an application “event” it is fairly trivial to calculate the upper bound on the number of events and concurrent clients (more simply, client requests per second) you can support. Then by sizing the number of threads in each pool you can weight number of clients vs. response latency.

It sounds like your “application events” are cpu bound, which makes this far easier to calculate. If they are IO bound, there are far many more variables you need to account for.


> On Oct 30, 2015, at 6:32 PM, Colin DuPlantis <[hidden email]> wrote:
>
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> Thanks for your thoughts, Robert.
>
> We are trying to solve a "liveness" problem, not a throughput or latency
> problem. It is understood that a certain piece of hardware can manage a
> certain rate of throughput. If that rate is exceeded in a burst, it is
> acceptable that latency will increase. What is not acceptable is
> dropping the connection in these conditions.
>
> Affining one or more threads to maintain sufficient liveness to prevent
> disconnection would probably decrease throughput and latency. This is
> also understood and acceptable.
>
> We already manage the persistence of inbound messages in case of
> recovery - this part is working fine. We're just trying to
> deterministically establish the upper bound of throughput for a certain
> hardware configuration. We can throttle the number of simultaneous
> threads until the disconnection event no longer occurs, but this is a
> probabilistic solution, subject to some degree to the whimsy of
> circumstance. My preference is a deterministic solution even at the cost
> of a certain percentage of throughput and latency.
>
> On 10/30/15 2:42 PM, Robert Engels wrote:
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>
>>
>> One final note, you’ll need to be careful because of the persistent nature of fix messages. Since you are going to hand off the actual processing to be performed somewhere else, but you need to capture the inbound message because you will be responding that you received the message (seqnum advancement), so you will need to be able to recover these messages in case of failure.
>>
>> If you process the requests synchronously you can be a bit more light-weight in the persistence but it is highly system dependent.
>>
>>
>>> On Oct 30, 2015, at 4:37 PM, Robert Engels <[hidden email]> wrote:
>>>
>>> In thinking about this some more, my answer may have been too stringent.
>>>
>>> Say you did isolate the ‘application processing request threads to 7 of the 8 cores. You could then put all of the fix processing threads on one core. It can easily handle thousands or low volume connections.
>>>
>>> As long as the fix processor hands off the handling of the request to the application queues, you will not time out.
>>>
>>> BUT, you have to know if that is acceptable…. The user could be firing thousands of orders (or other requests) and they are just backing up in the queues waiting to be processed…
>>>
>>> Leading to VERY HIGH latency to the point of the system not being usable.
>>>
>>>
>>>> On Oct 30, 2015, at 4:32 PM, Robert Engels <[hidden email]> wrote:
>>>>
>>>> If it is so starved that you cannot respond to the heartbeat then you are not really alive, and so you shouldn’t respond… thus terminating the connection.
>>>>
>>>> Even if you could partition it in such a way to ensure you respond to the heartbeats, what if that connection you are responding to then sends a request - you clearly can’t handle it because you are starved for CPU resources, thus you should not be responding to the heartbeat either…
>>>>
>>>> The only thing in this case is to make your application threads more efficient, and/or better hardware, and/or scale out (more machines).
>>>>
>>>> We use thread affinity and core isolation in our application with QFJ and it works very well, but these techniques are designed to improve latency, not throughput.
>>>>
>>>> You have a throughput problem. Not a latency problem.
>>>>
>>>> Regards,
>>>> Robert Engels
>>>> OptionsCIty Software Inc.
>>>>
>>>>
>>>>> On Oct 30, 2015, at 4:20 PM, Colin DuPlantis <[hidden email]> wrote:
>>>>>
>>>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>>>
>>>>>
>>>>> Hi, all, I'm bumping into some problems with connections closing under
>>>>> very heavy load (100k's messages/s). The machine CPU (8-core) is pegged
>>>>> processing these messages (not QFJ, our business app).
>>>>>
>>>>> I see, every now and then, a connection close after the counter sends a
>>>>> 35=1. My working assumption right now is that the thread(s) assigned to
>>>>> the QFJ acceptor/receiver are starving and not able to send 35=0 at
>>>>> agreed-upon intervals.
>>>>>
>>>>> Has anybody mucked around with thread affinity or the like to make sure
>>>>> the QFJ sockets and their minders are fed and happy?
>>>>>
>>>>> Also, are there any opinions about the benefits of using JDBC message
>>>>> store or file message store for high performance?
>>>>>
>>>>> Thanks.
>>>>>
>>>>> --
>>>>> Colin DuPlantis
>>>>> Chief Architect, Marketcetera
>>>>> Download, Run, Trade
>>>>> 888.868.4884 +1.541.306.6556
>>>>> http://www.marketcetera.org
>>>>>
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> _______________________________________________
>>>>> Quickfixj-users mailing list
>>>>> [hidden email]
>>>>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>>
>> ------------------------------------------------------------------------------
>> _______________________________________________
>> Quickfixj-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>
> --
> Colin DuPlantis
> Chief Architect, Marketcetera
> Download, Run, Trade
> 800.819.2928 x101 +1.541.306.6556
> http://www.marketcetera.org
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users


------------------------------------------------------------------------------
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High Performance Load

ChristopherH
In reply to this post by Colin DuPlantis
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



Just my thoughts ..

I suspect your best solution is to drastically cut the number of threads. More threads shouldn't equal throughput as you only have 8 cores as the number of threads rises application memory and context switching overhead will climb.

I recently re-factored a very large server for scaling reasons from potentially thousands to very small fixed pools (number of cores or less) , latency outliers fell dramatically, GC fell dramatically  and throughput improved, pretty much win,  win, win) 

I can see why thread affinity is attractive to you in this scenario (and its very useful in tuning high performance apps in general)  but if there is any option to re-factor your code I would go that way first and perhaps make use of thread affinity for later latency tuning as has been previously suggested..

With regard to filestore vs JDBC ? I've got by with filestore elsewhere but if you need greater performance you could consider memory mapped file storage perhaps there are open source libraries for this I believe
that could be adapted for QFJ in theory and greatly improve performance . They should I believe write even if the app crashes though OS crash / power failure would still have to be considered.

What is your heartbeat interval ? With the test request I presume there is no large send request going on at the same time as I've seen that one as an issue before with a different engine ? (unable to reply to the  test request due to large  resend in progress).

Chris

> To: [hidden email]

> From: [hidden email]
> Date: Fri, 30 Oct 2015 14:20:32 -0700
> Subject: [Quickfixj-users] High Performance Load
>
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> Hi, all, I'm bumping into some problems with connections closing under
> very heavy load (100k's messages/s). The machine CPU (8-core) is pegged
> processing these messages (not QFJ, our business app).
>
> I see, every now and then, a connection close after the counter sends a
> 35=1. My working assumption right now is that the thread(s) assigned to
> the QFJ acceptor/receiver are starving and not able to send 35=0 at
> agreed-upon intervals.
>
> Has anybody mucked around with thread affinity or the like to make sure
> the QFJ sockets and their minders are fed and happy?
>
> Also, are there any opinions about the benefits of using JDBC message
> store or file message store for high performance?
>
> Thanks.
>
> --
> Colin DuPlantis
> Chief Architect, Marketcetera
> Download, Run, Trade
> 888.868.4884 +1.541.306.6556
> http://www.marketcetera.org
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

------------------------------------------------------------------------------

_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High Performance Load

Robert Nicholson
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



In theory you can send heartbeat from any thread.

Otherwise you should think about decoupling receipt of the message from processing the message.

Sent from my iPhone

On Oct 30, 2015, at 9:04 PM, Chris Hurst <[hidden email]> wrote:

Just my thoughts ..

I suspect your best solution is to drastically cut the number of threads. More threads shouldn't equal throughput as you only have 8 cores as the number of threads rises application memory and context switching overhead will climb.

I recently re-factored a very large server for scaling reasons from potentially thousands to very small fixed pools (number of cores or less) , latency outliers fell dramatically, GC fell dramatically  and throughput improved, pretty much win,  win, win) 

I can see why thread affinity is attractive to you in this scenario (and its very useful in tuning high performance apps in general)  but if there is any option to re-factor your code I would go that way first and perhaps make use of thread affinity for later latency tuning as has been previously suggested..

With regard to filestore vs JDBC ? I've got by with filestore elsewhere but if you need greater performance you could consider memory mapped file storage perhaps there are open source libraries for this I believe
that could be adapted for QFJ in theory and greatly improve performance . They should I believe write even if the app crashes though OS crash / power failure would still have to be considered.

What is your heartbeat interval ? With the test request I presume there is no large send request going on at the same time as I've seen that one as an issue before with a different engine ? (unable to reply to the  test request due to large  resend in progress).

Chris

> To: [hidden email]

> From: [hidden email]
> Date: Fri, 30 Oct 2015 14:20:32 -0700
> Subject: [Quickfixj-users] High Performance Load
>
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> Hi, all, I'm bumping into some problems with connections closing under
> very heavy load (100k's messages/s). The machine CPU (8-core) is pegged
> processing these messages (not QFJ, our business app).
>
> I see, every now and then, a connection close after the counter sends a
> 35=1. My working assumption right now is that the thread(s) assigned to
> the QFJ acceptor/receiver are starving and not able to send 35=0 at
> agreed-upon intervals.
>
> Has anybody mucked around with thread affinity or the like to make sure
> the QFJ sockets and their minders are fed and happy?
>
> Also, are there any opinions about the benefits of using JDBC message
> store or file message store for high performance?
>
> Thanks.
>
> --
> Colin DuPlantis
> Chief Architect, Marketcetera
> Download, Run, Trade
> 888.868.4884 +1.541.306.6556
> http://www.marketcetera.org
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
------------------------------------------------------------------------------
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users

------------------------------------------------------------------------------

_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High Performance Load

Charles Briquel
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



In practice, it seems to be the « QFJ Message Processor » that is processing heartbeat, in the method ‘Session::next(Message)
So if the heartbeat response is not send, it’s either that this thread is blocked (which is likely to happens if is get blocked by the queue in which application make him drop the incoming messages, but I suppose he already checked this),
or this is the thread that is performing the send, in mina layer, that get trouble, I don’t know which one is it.
He said that he receive test request, so the receiving thread is not blocked, and he do not send so much messages.
That’s why, dropping the QFJ Message Processor thread may help, it’s more easy and quick to try than looking into cpu binding… And will lead to little less competition for the scheduler.
But if it’s lead to blocking the receiving thread when he want to pass the messages to the application thread pool, the problem will still remains.
I don’t think there is a reason why the receiving thread won’t get blocked because of too much cpu charge, will the Message Processor will be blocked for long times. Except if this is the application that is blocking this thread…




Le 31 oct. 2015 à 06:13, Robert Nicholson <[hidden email]> a écrit :

QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


In theory you can send heartbeat from any thread.

Otherwise you should think about decoupling receipt of the message from processing the message.

Sent from my iPhone

On Oct 30, 2015, at 9:04 PM, Chris Hurst <[hidden email]> wrote:

QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


Just my thoughts ..

I suspect your best solution is to drastically cut the number of threads. More threads shouldn't equal throughput as you only have 8 cores as the number of threads rises application memory and context switching overhead will climb.

I recently re-factored a very large server for scaling reasons from potentially thousands to very small fixed pools (number of cores or less) , latency outliers fell dramatically, GC fell dramatically  and throughput improved, pretty much win,  win, win) 

I can see why thread affinity is attractive to you in this scenario (and its very useful in tuning high performance apps in general)  but if there is any option to re-factor your code I would go that way first and perhaps make use of thread affinity for later latency tuning as has been previously suggested..

With regard to filestore vs JDBC ? I've got by with filestore elsewhere but if you need greater performance you could consider memory mapped file storage perhaps there are open source libraries for this I believe
that could be adapted for QFJ in theory and greatly improve performance . They should I believe write even if the app crashes though OS crash / power failure would still have to be considered.

What is your heartbeat interval ? With the test request I presume there is no large send request going on at the same time as I've seen that one as an issue before with a different engine ? (unable to reply to the  test request due to large  resend in progress).

Chris

> To: [hidden email]

> From: [hidden email]
> Date: Fri, 30 Oct 2015 14:20:32 -0700
> Subject: [Quickfixj-users] High Performance Load
> 
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
> 
> 
> Hi, all, I'm bumping into some problems with connections closing under 
> very heavy load (100k's messages/s). The machine CPU (8-core) is pegged 
> processing these messages (not QFJ, our business app).
> 
> I see, every now and then, a connection close after the counter sends a 
> 35=1. My working assumption right now is that the thread(s) assigned to 
> the QFJ acceptor/receiver are starving and not able to send 35=0 at 
> agreed-upon intervals.
> 
> Has anybody mucked around with thread affinity or the like to make sure 
> the QFJ sockets and their minders are fed and happy?
> 
> Also, are there any opinions about the benefits of using JDBC message 
> store or file message store for high performance?
> 
> Thanks.
> 
> -- 
> Colin DuPlantis
> Chief Architect, Marketcetera
> Download, Run, Trade
> 888.868.4884 +1.541.306.6556
> http://www.marketcetera.org
> 
> 
> ------------------------------------------------------------------------------
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
------------------------------------------------------------------------------
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
------------------------------------------------------------------------------
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users


------------------------------------------------------------------------------

_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High Performance Load

Colin DuPlantis
In reply to this post by ChristopherH
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



Yes, the problem can be resolved by reducing the number of active thread pool members. The optimal number isn't as low as exactly the number of cores you have because sometimes threads are blocked by I/O and another thread could be served during the gap.

My problem isn't one of "how do I prevent this from happening" so much as "how do I prevent this from happening and still squeeze every last cycle out of the available hardware". Thus my interest in affinity. As I increase the number of active thread pool members, throughput increases and latency decreases to a certain point. However, floating around near that optimal point, we sometimes stray too far and get the disconnect. I would sacrifice a little latency/throughput to achieve deterministic behavior near the maximum performance point.

Anyway, thanks, everybody for the thoughtful conversation, I really appreciate it.

On 10/30/15 7:04 PM, Chris Hurst wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




Just my thoughts ..

I suspect your best solution is to drastically cut the number of threads. More threads shouldn't equal throughput as you only have 8 cores as the number of threads rises application memory and context switching overhead will climb.

I recently re-factored a very large server for scaling reasons from potentially thousands to very small fixed pools (number of cores or less) , latency outliers fell dramatically, GC fell dramatically  and throughput improved, pretty much win,  win, win) 

I can see why thread affinity is attractive to you in this scenario (and its very useful in tuning high performance apps in general)  but if there is any option to re-factor your code I would go that way first and perhaps make use of thread affinity for later latency tuning as has been previously suggested..

With regard to filestore vs JDBC ? I've got by with filestore elsewhere but if you need greater performance you could consider memory mapped file storage perhaps there are open source libraries for this I believe
that could be adapted for QFJ in theory and greatly improve performance . They should I believe write even if the app crashes though OS crash / power failure would still have to be considered.

What is your heartbeat interval ? With the test request I presume there is no large send request going on at the same time as I've seen that one as an issue before with a different engine ? (unable to reply to the  test request due to large  resend in progress).

Chris

> To: [hidden email]
> From: [hidden email]
> Date: Fri, 30 Oct 2015 14:20:32 -0700
> Subject: [Quickfixj-users] High Performance Load
>
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> Hi, all, I'm bumping into some problems with connections closing under
> very heavy load (100k's messages/s). The machine CPU (8-core) is pegged
> processing these messages (not QFJ, our business app).
>
> I see, every now and then, a connection close after the counter sends a
> 35=1. My working assumption right now is that the thread(s) assigned to
> the QFJ acceptor/receiver are starving and not able to send 35=0 at
> agreed-upon intervals.
>
> Has anybody mucked around with thread affinity or the like to make sure
> the QFJ sockets and their minders are fed and happy?
>
> Also, are there any opinions about the benefits of using JDBC message
> store or file message store for high performance?
>
> Thanks.
>
> --
> Colin DuPlantis
> Chief Architect, Marketcetera
> Download, Run, Trade
> 888.868.4884 +1.541.306.6556
> http://www.marketcetera.org
>
>
> ------------------------------------------------------------------------------
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users


------------------------------------------------------------------------------


_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users

-- 
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
800.819.2928 x101 +1.541.306.6556
http://www.marketcetera.org

------------------------------------------------------------------------------

_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: High Performance Load

Tod Harter
In reply to this post by Colin DuPlantis
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




This is one of those problems where you really just have to work through it. In general your application threads shouldn't need to number more than the available cores (or 2x that if you have hyperthreading, etc). More threads than that just means context switching. You probably need to sequence the processing of messages anyhow, so its dubious whether or not you can actually handle more than some specific number in any case.

The hitch here is blocking. If you have 8 cores and 8 threads, and one goes off to do some database INSERT that blocks, then you have idle CPU, but the best way to handle THAT is making it asynchronous. That is hand off the database write to a separate thread pool, and have those threads drop the results back into your main work pool. It means you WILL have more threads than cores, but the extras will spend their whole lives blocked, meaning you won't get a lot of context switches. The main threads can go right back to the work queue for another job and stay live practically 100% of the time.

The thing is, you really will have to just experiment with all these techniques. Beyond that, sad to say, QuickFIX/J is mighty convenient, but it isn't anywhere close to the most performant FIX library out there. There are proprietary Java FIX libraries that can reduce messaging overhead by a large margin, you may want to investigate.

“Men occasionally stumble over the truth, but most of them pick themselves up and hurry off as if nothing had happened.” -Winston Churchill

"Sometimes I wonder whether the world is being run by smart people who are putting us on, or by imbeciles who really mean it." - Mark Twain

------------------------------------------------------------------------------

_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Loading...