Transactionality of quickfix.Application callback methods

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Transactionality of quickfix.Application callback methods

Diana Susca
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



Hi again,

Still building an acceptor on 1.6.2 and I have come across a new issue:
How do you ensure that messages aren't lost when an exception occurs in the fromApp() method in the Application?

I have performed a couple of tests, and if I throw an exception in fromApp(), the new order message is no longer saved in the database, so it's lost. But there is no error regarding sequence numbers, so following messages are accepted without problems, no resend requests are issued.

Isn't the message receiving supposed to be an atomic operation? I think that from the moment the message enters through the socket until it is completely processed (end of the fromApp() method) everything should be inside one transaction, and if at any point it crashes - for example due to a power outage, it should be as if the message was never there, so that it gets resent.

Otherwise, how can we recover from such an inconsistent state?

Do you have any suggestions on how to deal with this?

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transactionality of quickfix.Application callback methods

Christoph John
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


Hi,

if you throw an unchecked exception in your fromApp() then the sequence number is NOT incremented.
At least it shouldn't be. We had a similar case on the mailing list just yesterday
(StringIndexOutOfBoundsException) where the seq num was not incremented.

How does your fromApp code look? Can you paste someting from the message and event log?

And no, there is nothing atomic about that operation. It's just very likely that you will not
increment the seq num in case of errors so that the messages will be re-transmitted.
If something happens between reception of the message and fromApp() then the seq num is not
incremented. Seq num is incremented after fromApp() has been processed.

Chris.


On 14/02/17 09:52, Diana Susca wrote:

> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
>
>
> Hi again,
>
> Still building an acceptor on 1.6.2 and I have come across a new issue:
> How do you ensure that messages aren't lost when an exception occurs in the fromApp() method in
> the Application?
>
> I have performed a couple of tests, and if I throw an exception in fromApp(), the new order
> message is no longer saved in the database, so it's lost. But there is no error regarding sequence
> numbers, so following messages are accepted without problems, no resend requests are issued.
>
> Isn't the message receiving supposed to be an atomic operation? I think that from the moment the
> message enters through the socket until it is completely processed (end of the fromApp() method)
> everything should be inside one transaction, and if at any point it crashes - for example due to a
> power outage, it should be as if the message was never there, so that it gets resent.
>
> Otherwise, how can we recover from such an inconsistent state?
>
> Do you have any suggestions on how to deal with this?
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>
>
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

--
Christoph John
Development & Support
Direct: +49 241 557080-28
Mailto:[hidden email]
       


http://www.macd.com <http://www.macd.com/>
----------------------------------------------------------------------------------------------------
       
----------------------------------------------------------------------------------------------------
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
Tel: +49 241 557080-0 | Fax: +49 241 557080-10
         Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663

Geschäftsführer: George Macdonald
----------------------------------------------------------------------------------------------------
       
----------------------------------------------------------------------------------------------------

take care of the environment - print only if necessary

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transactionality of quickfix.Application callback methods

Guido Medina
In reply to this post by Diana Susca
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



I believe you should catch the exception and somehow let the other side know about your failure on processing that message,
maybe the JavaDoc can help you fake it:

/**
* This callback receives messages for the application. This is one of the
* core entry points for your FIX application. Every application level
* request will come through here. If, for example, your application is a
* sell-side OMS, this is where you will get your new order requests. If you
* were a buy side, you would get your execution reports here. If a
* FieldNotFound exception is thrown, the counterparty will receive a reject
* indicating a conditionally required field is missing. The Message class
* will throw this exception when trying to retrieve a missing field, so you
* will rarely need the throw this explicitly. You can also throw an
* UnsupportedMessageType exception. This will result in the counterparty
* getting a business reject informing them your application cannot process
* those types of messages. An IncorrectTagValue can also be thrown if a
* field contains a value that is out of range or you do not support.
*
* @param message QuickFIX message
* @param sessionId QuickFIX session ID
* @throws FieldNotFound
* @throws IncorrectDataFormat
* @throws IncorrectTagValue
* @throws UnsupportedMessageType
*/
void fromApp(Message message, SessionID sessionId) throws FieldNotFound, IncorrectDataFormat,
IncorrectTagValue, UnsupportedMessageType;

On Tue, Feb 14, 2017 at 8:52 AM, Diana Susca <[hidden email]> wrote:
QuickFIX/J Documentation: <a href="http://www.quickfixj.org/documentation/ QuickFIX/J" rel="noreferrer" target="_blank">http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



Hi again,

Still building an acceptor on 1.6.2 and I have come across a new issue:
How do you ensure that messages aren't lost when an exception occurs in the fromApp() method in the Application?

I have performed a couple of tests, and if I throw an exception in fromApp(), the new order message is no longer saved in the database, so it's lost. But there is no error regarding sequence numbers, so following messages are accepted without problems, no resend requests are issued.

Isn't the message receiving supposed to be an atomic operation? I think that from the moment the message enters through the socket until it is completely processed (end of the fromApp() method) everything should be inside one transaction, and if at any point it crashes - for example due to a power outage, it should be as if the message was never there, so that it gets resent.

Otherwise, how can we recover from such an inconsistent state?

Do you have any suggestions on how to deal with this?

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transactionality of quickfix.Application callback methods

Guido Medina
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



What I meant by "letting the other side know" was by throwing one of the documented exceptions referred by the JavaDoc,
maybe there are more exceptions more adecuated to your case that are not listed?

On Tue, Feb 14, 2017 at 10:16 AM, Guido Medina <[hidden email]> wrote:
I believe you should catch the exception and somehow let the other side know about your failure on processing that message,
maybe the JavaDoc can help you fake it:

/**
* This callback receives messages for the application. This is one of the
* core entry points for your FIX application. Every application level
* request will come through here. If, for example, your application is a
* sell-side OMS, this is where you will get your new order requests. If you
* were a buy side, you would get your execution reports here. If a
* FieldNotFound exception is thrown, the counterparty will receive a reject
* indicating a conditionally required field is missing. The Message class
* will throw this exception when trying to retrieve a missing field, so you
* will rarely need the throw this explicitly. You can also throw an
* UnsupportedMessageType exception. This will result in the counterparty
* getting a business reject informing them your application cannot process
* those types of messages. An IncorrectTagValue can also be thrown if a
* field contains a value that is out of range or you do not support.
*
* @param message QuickFIX message
* @param sessionId QuickFIX session ID
* @throws FieldNotFound
* @throws IncorrectDataFormat
* @throws IncorrectTagValue
* @throws UnsupportedMessageType
*/
void fromApp(Message message, SessionID sessionId) throws FieldNotFound, IncorrectDataFormat,
IncorrectTagValue, UnsupportedMessageType;

On Tue, Feb 14, 2017 at 8:52 AM, Diana Susca <[hidden email]> wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J
Support: http://www.quickfixj.org/support/



Hi again,

Still building an acceptor on 1.6.2 and I have come across a new issue:
How do you ensure that messages aren't lost when an exception occurs in the fromApp() method in the Application?

I have performed a couple of tests, and if I throw an exception in fromApp(), the new order message is no longer saved in the database, so it's lost. But there is no error regarding sequence numbers, so following messages are accepted without problems, no resend requests are issued.

Isn't the message receiving supposed to be an atomic operation? I think that from the moment the message enters through the socket until it is completely processed (end of the fromApp() method) everything should be inside one transaction, and if at any point it crashes - for example due to a power outage, it should be as if the message was never there, so that it gets resent.

Otherwise, how can we recover from such an inconsistent state?

Do you have any suggestions on how to deal with this?

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transactionality of quickfix.Application callback methods

Christoph John
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


You could test if RejectMessageOnUnhandledException=Y helps in that regard.

Chris.


On 14/02/17 11:18, Guido Medina wrote:

> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
>
>
> What I meant by "letting the other side know" was by throwing one of the documented exceptions
> referred by the JavaDoc,
> maybe there are more exceptions more adecuated to your case that are not listed?
>
> On Tue, Feb 14, 2017 at 10:16 AM, Guido Medina <[hidden email] <mailto:[hidden email]>> wrote:
>
>     I believe you should catch the exception and somehow let the other side know about your
>     failure on processing that message,
>     maybe the JavaDoc can help you fake it:
>
>     /** * This callback receives messages for the application. This is one of the * core entry
>     points for your FIX application. Every application level * request will come through here. If,
>     for example, your application is a * sell-side OMS, this is where you will get your new order
>     requests. If you * were a buy side, you would get your execution reports here. If a *
>     FieldNotFound exception is thrown, the counterparty will receive a reject * indicating a
>     conditionally required field is missing. The Message class * will throw this exception when
>     trying to retrieve a missing field, so you * will rarely need the throw this explicitly. You
>     can also throw an * UnsupportedMessageType exception. This will result in the counterparty *
>     getting a business reject informing them your application cannot process * those types of
>     messages. An IncorrectTagValue can also be thrown if a * field contains a value that is out of
>     range or you do not support. * * @param message QuickFIX message * @param sessionId QuickFIX
>     session ID * @throws FieldNotFound * @throws IncorrectDataFormat * @throws IncorrectTagValue *
>     @throws UnsupportedMessageType */ void fromApp(Message message, SessionID sessionId)throws FieldNotFound, IncorrectDataFormat, IncorrectTagValue, UnsupportedMessageType;
>
>
>     On Tue, Feb 14, 2017 at 8:52 AM, Diana Susca <[hidden email]
>     <mailto:[hidden email]>> wrote:
>
>         QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>         QuickFIX/J <http://www.quickfixj.org/documentation/QuickFIX/J> Support:
>         http://www.quickfixj.org/support/ <http://www.quickfixj.org/support/>
>
>
>
>         Hi again,
>
>         Still building an acceptor on 1.6.2 and I have come across a new issue:
>         How do you ensure that messages aren't lost when an exception occurs in the fromApp()
>         method in the Application?
>
>         I have performed a couple of tests, and if I throw an exception in fromApp(), the new
>         order message is no longer saved in the database, so it's lost. But there is no error
>         regarding sequence numbers, so following messages are accepted without problems, no resend
>         requests are issued.
>
>         Isn't the message receiving supposed to be an atomic operation? I think that from the
>         moment the message enters through the socket until it is completely processed (end of the
>         fromApp() method) everything should be inside one transaction, and if at any point it
>         crashes - for example due to a power outage, it should be as if the message was never
>         there, so that it gets resent.
>
>         Otherwise, how can we recover from such an inconsistent state?
>
>         Do you have any suggestions on how to deal with this?
>
>         ------------------------------------------------------------------------------
>         Check out the vibrant tech community on one of the world's most
>         engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>         _______________________________________________
>         Quickfixj-users mailing list
>         [hidden email] <mailto:[hidden email]>
>         https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>         <https://lists.sourceforge.net/lists/listinfo/quickfixj-users>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, SlashDot.org! http://sdm.link/slashdot
>
>
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

--
Christoph John
Development & Support
Direct: +49 241 557080-28
Mailto:[hidden email]
       


http://www.macd.com <http://www.macd.com/>
----------------------------------------------------------------------------------------------------
       
----------------------------------------------------------------------------------------------------
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
Tel: +49 241 557080-0 | Fax: +49 241 557080-10
         Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663

Geschäftsführer: George Macdonald
----------------------------------------------------------------------------------------------------
       
----------------------------------------------------------------------------------------------------

take care of the environment - print only if necessary

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transactionality of quickfix.Application callback methods

Diana Susca
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



I just tested the Acceptor with  RejectMessageOnUnhandledException=Y and I got the same result. The message was silently dropped.

I will also try Guido's approach, but that only works when there is an exception thrown. What if there is a disastrous event, such as a hard-drive failure?




On Tue, Feb 14, 2017 at 12:23 PM, Christoph John <[hidden email]> wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J
Support: http://www.quickfixj.org/support/


You could test if RejectMessageOnUnhandledException=Y helps in that regard.

Chris.



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transactionality of quickfix.Application callback methods

Christoph John
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



Without some source code it will be very hard to tell why the exception was not properly handled.
What does your fromApp() method look like?
IIRC RejectMessageOnUnhandledException catches a Throwable, so it seems to me that you somehow catch your Exception earlier.

Hard-drive failure: when the seqnum cannot be persisted then it will trigger a resend on the next incoming message. Hopefully you replaced your hard-drive until then. ;)


On 14/02/17 11:49, Diana Susca wrote:
I just tested the Acceptor with  RejectMessageOnUnhandledException=Y and I got the same result. The message was silently dropped.

I will also try Guido's approach, but that only works when there is an exception thrown. What if there is a disastrous event, such as a hard-drive failure?




On Tue, Feb 14, 2017 at 12:23 PM, Christoph John <[hidden email]> wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J
Support: http://www.quickfixj.org/support/


You could test if RejectMessageOnUnhandledException=Y helps in that regard.

Chris.



--
Christoph John
Development & Support
Direct: +49 241 557080-28
Mailto:Christoph.John@...



http://www.macd.com


MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
Tel: +49 241 557080-0 | Fax: +49 241 557080-10
 Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663

Geschäftsführer: George Macdonald


take care of the environment - print only if necessary

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transactionality of quickfix.Application callback methods

Guido Medina
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



Even better, why pay for Oracle if you can get a high available distributed DB for free like Riak or Cassandra,
I have been working with Riak for years and sometimes a node goes down for good and we have noticed it DAYS later without the operation being affected because data is distributed,
specially for key-value data like session ID and FIX text messages.

On Tue, Feb 14, 2017 at 10:52 AM, Christoph John <[hidden email]> wrote:
QuickFIX/J Documentation: <a href="http://www.quickfixj.org/documentation/ QuickFIX/J" rel="noreferrer" target="_blank">http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



Without some source code it will be very hard to tell why the exception was not properly handled.
What does your fromApp() method look like?
IIRC RejectMessageOnUnhandledException catches a Throwable, so it seems to me that you somehow catch your Exception earlier.

Hard-drive failure: when the seqnum cannot be persisted then it will trigger a resend on the next incoming message. Hopefully you replaced your hard-drive until then. ;)


On 14/02/17 11:49, Diana Susca wrote:
I just tested the Acceptor with  RejectMessageOnUnhandledException=Y and I got the same result. The message was silently dropped.

I will also try Guido's approach, but that only works when there is an exception thrown. What if there is a disastrous event, such as a hard-drive failure?




On Tue, Feb 14, 2017 at 12:23 PM, Christoph John <[hidden email]> wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J
Support: http://www.quickfixj.org/support/


You could test if RejectMessageOnUnhandledException=Y helps in that regard.

Chris.



--
Christoph John
Development & Support
Direct: <a href="tel:+49%20241%2055708028" value="+4924155708028" target="_blank">+49 241 557080-28
Mailto:Christoph.John@...



http://www.macd.com


MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
Tel: +49 241 557080-0 | Fax: <a href="tel:+49%20241%2055708010" value="+4924155708010" target="_blank">+49 241 557080-10
 Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663

Geschäftsführer: George Macdonald


take care of the environment - print only if necessary

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transactionality of quickfix.Application callback methods

Colin DuPlantis
In reply to this post by Diana Susca
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



Once your application accepts the message, you are responsible for delivering it, or, at least, disposing of it properly.

One way you could do that is to catch an exception in fromApp() and reject the message, so at least the counter party knows you couldn't digest it.

For more complicated issues like, as you mentioned, hard drive failure, you will need to restart the session. You can either do that immediately with some kind of clustered set up or manually once you find the problem.

Regardless, on session start, your application will need to be able to determine what the last message it successfully processed was. Let's say you store that information in a database. You read the last message you can confirm you handled and compare that to the sequence number in the QFJ store. If there's a gap, you need to reset the sequence number in the store before the counter logs on. This will force the counter to resend the missing messages, which you can now process.

Of course, you can't actually guarantee that you didn't process those messages, because it's possible that the transaction to write to the DB after processing is what failed. Therefore, you also need to have baked into your processing loop the possibility that you have already processed this message.

HTH

On 02/14/2017 12:52 AM, Diana Susca wrote:

QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




Hi again,

Still building an acceptor on 1.6.2 and I have come across a new issue:
How do you ensure that messages aren't lost when an exception occurs in the fromApp() method in the Application?

I have performed a couple of tests, and if I throw an exception in fromApp(), the new order message is no longer saved in the database, so it's lost. But there is no error regarding sequence numbers, so following messages are accepted without problems, no resend requests are issued.

Isn't the message receiving supposed to be an atomic operation? I think that from the moment the message enters through the socket until it is completely processed (end of the fromApp() method) everything should be inside one transaction, and if at any point it crashes - for example due to a power outage, it should be as if the message was never there, so that it gets resent.

Otherwise, how can we recover from such an inconsistent state?

Do you have any suggestions on how to deal with this?


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot


_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transactionality of quickfix.Application callback methods

Colin DuPlantis
In reply to this post by Diana Susca
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



Once your application accepts the message, you are responsible for delivering it, or, at least, disposing of it properly.

One way you could do that is to catch an exception in fromApp() and reject the message, so at least the counter party knows you couldn't digest it.

For more complicated issues like, as you mentioned, hard drive failure, you will need to restart the session. You can either do that immediately with some kind of clustered set up or manually once you find the problem.

Regardless, on session start, your application will need to be able to determine what the last message it successfully processed was. Let's say you store that information in a database. You read the last message you can confirm you handled and compare that to the sequence number in the QFJ store. If there's a gap, you need to reset the sequence number in the store before the counter logs on. This will force the counter to resend the missing messages, which you can now process.

Of course, you can't actually guarantee that you didn't process those messages, because it's possible that the transaction to write to the DB after processing is what failed. Therefore, you also need to have baked into your processing loop the possibility that you have already processed this message.

HTH

On 02/14/2017 12:52 AM, Diana Susca wrote:

QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




Hi again,

Still building an acceptor on 1.6.2 and I have come across a new issue:
How do you ensure that messages aren't lost when an exception occurs in the fromApp() method in the Application?

I have performed a couple of tests, and if I throw an exception in fromApp(), the new order message is no longer saved in the database, so it's lost. But there is no error regarding sequence numbers, so following messages are accepted without problems, no resend requests are issued.

Isn't the message receiving supposed to be an atomic operation? I think that from the moment the message enters through the socket until it is completely processed (end of the fromApp() method) everything should be inside one transaction, and if at any point it crashes - for example due to a power outage, it should be as if the message was never there, so that it gets resent.

Otherwise, how can we recover from such an inconsistent state?

Do you have any suggestions on how to deal with this?


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot


_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transactionality of quickfix.Application callback methods

Diana Susca
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



Thanks for all the answers, they are all useful.
Indeed, I used to catch all exceptions, which is why the RejectMessageOnUnhandledException=Y didn't work.


On Tue, Feb 14, 2017 at 4:37 PM, Colin DuPlantis <[hidden email]> wrote:
QuickFIX/J Documentation: <a href="http://www.quickfixj.org/documentation/ QuickFIX/J" rel="noreferrer" target="_blank">http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



Once your application accepts the message, you are responsible for delivering it, or, at least, disposing of it properly.

One way you could do that is to catch an exception in fromApp() and reject the message, so at least the counter party knows you couldn't digest it.

For more complicated issues like, as you mentioned, hard drive failure, you will need to restart the session. You can either do that immediately with some kind of clustered set up or manually once you find the problem.

Regardless, on session start, your application will need to be able to determine what the last message it successfully processed was. Let's say you store that information in a database. You read the last message you can confirm you handled and compare that to the sequence number in the QFJ store. If there's a gap, you need to reset the sequence number in the store before the counter logs on. This will force the counter to resend the missing messages, which you can now process.

Of course, you can't actually guarantee that you didn't process those messages, because it's possible that the transaction to write to the DB after processing is what failed. Therefore, you also need to have baked into your processing loop the possibility that you have already processed this message.

HTH

On 02/14/2017 12:52 AM, Diana Susca wrote:

QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




Hi again,

Still building an acceptor on 1.6.2 and I have come across a new issue:
How do you ensure that messages aren't lost when an exception occurs in the fromApp() method in the Application?

I have performed a couple of tests, and if I throw an exception in fromApp(), the new order message is no longer saved in the database, so it's lost. But there is no error regarding sequence numbers, so following messages are accepted without problems, no resend requests are issued.

Isn't the message receiving supposed to be an atomic operation? I think that from the moment the message enters through the socket until it is completely processed (end of the fromApp() method) everything should be inside one transaction, and if at any point it crashes - for example due to a power outage, it should be as if the message was never there, so that it gets resent.

Otherwise, how can we recover from such an inconsistent state?

Do you have any suggestions on how to deal with this?


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot


_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transactionality of quickfix.Application callback methods

Tod Harter
In reply to this post by Diana Susca
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



What you should understand is that QF/J and FIX's architecture, already gives you all you need to deal with these issues.

QF/J will only record the increment of a sequence number and record a message in the message store if your fromApp() succeeds, so all you have to do to guarantee retransmission of messages is throw an exception. Even if your machine simply crashes and doesn't store the new sequence number (thus there's no reject message) a resend request will happen after restart because of the detected sequence number gap.

It IS conceivable that application logic could handle an incoming message and THEN something causes a reject or a crash happens, etc. Whenever a message is resent it will come with a posdup flag = 'y' and/or a resend='y', indicating that this message could be one that has been seen by application logic before. QF/J obviously can't determine this or know what to do in that case, so it would be up to application logic to decide what to do next.

Typically application level issues are handled using BusinessMessageReject or a more specialized form of rejection, NOT a reject message or resend request. There should be no case where application logic would be generating low level rejects, it should simply throw an exception (ideally one of the supported exception types) if a message cannot be processed at all, or otherwise an application level reject.

Following these procedures should meet all requirements. Obviously if you need HA there are some additional considerations (replication of the message store and provision of some kind of failover mechanism for the initiator). Using SQL message stores is obviously one technique, though ideally you need to couple it with RDBMS level replication as well (to handle loss of database events).

The point is, QF/J should be able to handle the whole 'which was the last message processed and get me the next one' part of the equation itself. Application logic should only need to deal with business level rejection or else tossing exceptions in cases of lower level failure (missing tags and such).

On Wed, Feb 15, 2017 at 4:41 AM, <[hidden email]> wrote:
Date: Tue, 14 Feb 2017 06:37:03 -0800
From: Colin DuPlantis <[hidden email]>
Subject: Re: [Quickfixj-users] Transactionality of
        quickfix.Application callback methods
To: [hidden email]
Message-ID: <[hidden email]>
Content-Type: text/plain; charset="windows-1252"

Once your application accepts the message, you are responsible for
delivering it, or, at least, disposing of it properly.

One way you could do that is to catch an exception in fromApp() and
reject the message, so at least the counter party knows you couldn't
digest it.

For more complicated issues like, as you mentioned, hard drive failure,
you will need to restart the session. You can either do that immediately
with some kind of clustered set up or manually once you find the problem.

Regardless, on session start, your application will need to be able to
determine what the last message it successfully processed was. Let's say
you store that information in a database. You read the last message you
can confirm you handled and compare that to the sequence number in the
QFJ store. If there's a gap, you need to reset the sequence number in
the store before the counter logs on. This will force the counter to
resend the missing messages, which you can now process.

Of course, you can't actually guarantee that you didn't process those
messages, because it's possible that the transaction to write to the DB
after processing is what failed. Therefore, you also need to have baked
into your processing loop the possibility that you have already
processed this message.

HTH



I will no longer accept the things I cannot change. I am changing the things I cannot accept.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Transactionality of quickfix.Application callback methods

Colin DuPlantis
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



You are correct in everything you say.

I was handwaving in my explanation over more complex behavior when you use more than one thread to process incoming messages.


On 02/15/2017 10:40 AM, Tod Harter wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




What you should understand is that QF/J and FIX's architecture, already gives you all you need to deal with these issues.

QF/J will only record the increment of a sequence number and record a message in the message store if your fromApp() succeeds, so all you have to do to guarantee retransmission of messages is throw an exception. Even if your machine simply crashes and doesn't store the new sequence number (thus there's no reject message) a resend request will happen after restart because of the detected sequence number gap.

It IS conceivable that application logic could handle an incoming message and THEN something causes a reject or a crash happens, etc. Whenever a message is resent it will come with a posdup flag = 'y' and/or a resend='y', indicating that this message could be one that has been seen by application logic before. QF/J obviously can't determine this or know what to do in that case, so it would be up to application logic to decide what to do next.

Typically application level issues are handled using BusinessMessageReject or a more specialized form of rejection, NOT a reject message or resend request. There should be no case where application logic would be generating low level rejects, it should simply throw an exception (ideally one of the supported exception types) if a message cannot be processed at all, or otherwise an application level reject.

Following these procedures should meet all requirements. Obviously if you need HA there are some additional considerations (replication of the message store and provision of some kind of failover mechanism for the initiator). Using SQL message stores is obviously one technique, though ideally you need to couple it with RDBMS level replication as well (to handle loss of database events).

The point is, QF/J should be able to handle the whole 'which was the last message processed and get me the next one' part of the equation itself. Application logic should only need to deal with business level rejection or else tossing exceptions in cases of lower level failure (missing tags and such).

On Wed, Feb 15, 2017 at 4:41 AM, <[hidden email]> wrote:
Date: Tue, 14 Feb 2017 06:37:03 -0800
From: Colin DuPlantis <[hidden email]>
Subject: Re: [Quickfixj-users] Transactionality of
        quickfix.Application callback methods
To: [hidden email]
Message-ID: <[hidden email]>
Content-Type: text/plain; charset="windows-1252"

Once your application accepts the message, you are responsible for
delivering it, or, at least, disposing of it properly.

One way you could do that is to catch an exception in fromApp() and
reject the message, so at least the counter party knows you couldn't
digest it.

For more complicated issues like, as you mentioned, hard drive failure,
you will need to restart the session. You can either do that immediately
with some kind of clustered set up or manually once you find the problem.

Regardless, on session start, your application will need to be able to
determine what the last message it successfully processed was. Let's say
you store that information in a database. You read the last message you
can confirm you handled and compare that to the sequence number in the
QFJ store. If there's a gap, you need to reset the sequence number in
the store before the counter logs on. This will force the counter to
resend the missing messages, which you can now process.

Of course, you can't actually guarantee that you didn't process those
messages, because it's possible that the transaction to write to the DB
after processing is what failed. Therefore, you also need to have baked
into your processing loop the possibility that you have already
processed this message.

HTH



I will no longer accept the things I cannot change. I am changing the things I cannot accept.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot


_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Loading...