Rolling Message store?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Rolling Message store?

Nick Fortescue-2
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/

I'd guess this might be becoming an FAQ, but I can't find an answer anywhere.
 
I've got an application that is likely to be reasonably long running, say a week, before being reset. Logging messages is nice and easy, just plug in an appropriate log4j configuration file and I can have all the rolling logs i want. However I can't see a similar way of dealing with MessageStore.
 
I suspect that for fix resend purposes i'll never need more than the last 10 minutes of messages. Is there a standard MessageStore implementation that either just forgets older messages, or a way of getting FileStore to roll old messages over (say every day)? This would allow old messages to be archived or deleted without affecting a running app.
 
If not, would such a message store be a useful addition to the codebase? I'm thinking of a MemoryStore with a time limit threshold, or a FileStore with various rolling configuration options similar to log4j.
 
Nick
 
 

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Rolling Message store?

Stephen Bate
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/
> I suspect that for fix resend purposes i'll never need more than the last
> 10 minutes of messages. Is there a standard MessageStore implementation
> that either just forgets older messages, or a way of getting FileStore
> to roll old messages over (say every day)? This would allow old
> messages to be archived or deleted without affecting a running app.

There's not a rolling file implementation for the message store.

> If not, would such a message store be a useful addition to the codebase?
> I'm thinking of a MemoryStore with a time limit threshold, or a
> FileStore with various rolling configuration options similar to log4j.

I'm not sure how useful a rolling store would be. However, there's
room for improvement in the caching of the messages for resend. Currently,
the file store loads all messages into memory (using MemoryStore). That
can slow down server start time and can consume a lot of memory for long,
active sessions. Another option might be to use something like JBossCache
or EhCache (or similar) to store the messages with a backing file but
configure the cache to delete messages that are old than a specified age. We
would need to do some experimentation to be sure the performance is good
enough.

Steve


-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Rolling Message store?

Brad Harvey
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/

I wonder how hsqldb with a cached table would compare in terms of
performance and reliability?  Should be able to plug it into the
existing jdbc store.  If you run hypersonic in embedded mode it wouldn't
add much system admin overhead over the file store.

Brad.

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of
Steve Bate
Sent: Tuesday, 19 June 2007 4:13 AM
To: [hidden email]
Subject: Re: [Quickfixj-users] Rolling Message store?

QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/
> I suspect that for fix resend purposes i'll never need more than the
> last 10 minutes of messages. Is there a standard MessageStore
> implementation that either just forgets older messages, or a way of
> getting FileStore to roll old messages over (say every day)? This
> would allow old messages to be archived or deleted without affecting a
running app.

There's not a rolling file implementation for the message store.

> If not, would such a message store be a useful addition to the
codebase?
> I'm thinking of a MemoryStore with a time limit threshold, or a
> FileStore with various rolling configuration options similar to log4j.

I'm not sure how useful a rolling store would be. However, there's room
for improvement in the caching of the messages for resend. Currently,
the file store loads all messages into memory (using MemoryStore). That
can slow down server start time and can consume a lot of memory for
long, active sessions. Another option might be to use something like
JBossCache or EhCache (or similar) to store the messages with a backing
file but configure the cache to delete messages that are old than a
specified age. We would need to do some experimentation to be sure the
performance is good enough.

Steve


------------------------------------------------------------------------
-
This SF.net email is sponsored by DB2 Express Download DB2 Express C -
the FREE version of DB2 express and take control of your XML. No limits.
Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Rolling Message store?

Vladimir Kvasov
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


Brad,

I tried to use hsqldb (embedded mode) to perform data processing once. sql queries simplified code a little bit but there was a significant difference in performance compared to processing based on ArrayLists. hsqldb version was much slower. So I'd recommend to write some pilot code to perform load tests before choosing to use hsqldb.

wbr
_________________________________
Vladimir Kvasov
Trading & Market Making
Rates Technology
Deutsche Bank
+7 495 9812289




"Brad Harvey" <[hidden email]>
Sent by: [hidden email]

19.06.2007 02:12

Please respond to
[hidden email]

To
<[hidden email]>
cc
Subject
Re: [Quickfixj-users] Rolling Message store?





QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/

I wonder how hsqldb with a cached table would compare in terms of
performance and reliability?  Should be able to plug it into the
existing jdbc store.  If you run hypersonic in embedded mode it wouldn't
add much system admin overhead over the file store.

Brad.

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of
Steve Bate
Sent: Tuesday, 19 June 2007 4:13 AM
To: [hidden email]
Subject: Re: [Quickfixj-users] Rolling Message store?

QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/
> I suspect that for fix resend purposes i'll never need more than the
> last 10 minutes of messages. Is there a standard MessageStore
> implementation that either just forgets older messages, or a way of
> getting FileStore to roll old messages over (say every day)? This
> would allow old messages to be archived or deleted without affecting a
running app.

There's not a rolling file implementation for the message store.

> If not, would such a message store be a useful addition to the
codebase?
> I'm thinking of a MemoryStore with a time limit threshold, or a
> FileStore with various rolling configuration options similar to log4j.

I'm not sure how useful a rolling store would be. However, there's room
for improvement in the caching of the messages for resend. Currently,
the file store loads all messages into memory (using MemoryStore). That
can slow down server start time and can consume a lot of memory for
long, active sessions. Another option might be to use something like
JBossCache or EhCache (or similar) to store the messages with a backing
file but configure the cache to delete messages that are old than a
specified age. We would need to do some experimentation to be sure the
performance is good enough.

Steve


------------------------------------------------------------------------
-
This SF.net email is sponsored by DB2 Express Download DB2 Express C -
the FREE version of DB2 express and take control of your XML. No limits.
Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users


---

This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Rolling Message store?

colincrist
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/

To perform well you need to get the messages to disk asap and then worry about having them stored in a more useful structured way. Engine startup would flush messages  from this transaction log into the proper store.
 
I'd recommend HOWL as a transaction log, it does what it says on the tin...
 
 
Colin.


From: [hidden email] [mailto:[hidden email]] On Behalf Of Vladimir Kvasov
Sent: 19 June 2007 07:26
To: [hidden email]
Subject: Re: [Quickfixj-users] Rolling Message store?


Brad,

I tried to use hsqldb (embedded mode) to perform data processing once. sql queries simplified code a little bit but there was a significant difference in performance compared to processing based on ArrayLists. hsqldb version was much slower. So I'd recommend to write some pilot code to perform load tests before choosing to use hsqldb.

wbr
_________________________________
Vladimir Kvasov
Trading & Market Making
Rates Technology
Deutsche Bank
+7 495 9812289




"Brad Harvey" <[hidden email]>
Sent by: [hidden email]

19.06.2007 02:12

Please respond to
[hidden email]

To
<[hidden email]>
cc
Subject
Re: [Quickfixj-users] Rolling Message store?





QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/

I wonder how hsqldb with a cached table would compare in terms of
performance and reliability?  Should be able to plug it into the
existing jdbc store.  If you run hypersonic in embedded mode it wouldn't
add much system admin overhead over the file store.

Brad.

-----Original Message-----
From: [hidden email]
[mailto:[hidden email]] On Behalf Of
Steve Bate
Sent: Tuesday, 19 June 2007 4:13 AM
To: [hidden email]
Subject: Re: [Quickfixj-users] Rolling Message store?

QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/
> I suspect that for fix resend purposes i'll never need more than the
> last 10 minutes of messages. Is there a standard MessageStore
> implementation that either just forgets older messages, or a way of
> getting FileStore to roll old messages over (say every day)? This
> would allow old messages to be archived or deleted without affecting a
running app.

There's not a rolling file implementation for the message store.

> If not, would such a message store be a useful addition to the
codebase?
> I'm thinking of a MemoryStore with a time limit threshold, or a
> FileStore with various rolling configuration options similar to log4j.

I'm not sure how useful a rolling store would be. However, there's room
for improvement in the caching of the messages for resend. Currently,
the file store loads all messages into memory (using MemoryStore). That
can slow down server start time and can consume a lot of memory for
long, active sessions. Another option might be to use something like
JBossCache or EhCache (or similar) to store the messages with a backing
file but configure the cache to delete messages that are old than a
specified age. We would need to do some experimentation to be sure the
performance is good enough.

Steve


------------------------------------------------------------------------
-
This SF.net email is sponsored by DB2 Express Download DB2 Express C -
the FREE version of DB2 express and take control of your XML. No limits.
Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users


---

This e-mail may contain confidential and/or privileged information. If you
are not the intended recipient (or have received this e-mail in error)
please notify the sender immediately and destroy this e-mail. Any
unauthorized copying, disclosure or distribution of the material in this
e-mail is strictly forbidden.

-------------------------------------------------------------------------
This SF.net email is sponsored by DB2 Express
Download DB2 Express C - the FREE version of DB2 express and take
control of your XML. No limits. Just data. Click to get it now.
http://sourceforge.net/powerbar/db2/
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users