Fields 355 & 354

classic Classic list List threaded Threaded
19 messages Options
Reply | Threaded
Open this post in threaded view
|

Fields 355 & 354

Colin DuPlantis
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


Howdy, all.

We've got a counter-party that's sending messages with tag 355 and
omitting tag 354. This causes QFJ to be unable to extract the contents
of field 355. We're running 1.6.3 right now, and the trouble comes up in
Message.extractField as the length is assumed to be present.

Are there any opinions about whether 354 is required if 355 is included?
The FIX spec lists both as "not required" as do the data dictionaries,
but, it seems to me that 354 should be conditionally required if 355 is
present.

My question is, what are the opinions of the list members as to how
snarly I can get with the counter about this issue?

Cheers.


--
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
888.868.4884 +1.541.306.6556
http://www.marketcetera.org


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Fields 355 & 354

Grant Birchmeier
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



Per FIX spec, the first element of a repeating group is the delimiter item that signals the start of the group (and the end of the last one), and is always mandatory.  I believe the spec is not ambiguous about this point.

Whether that gets your counterparty to change their impl to behave or not, that's up in the air.

On Wed, May 3, 2017 at 12:11 PM, Colin DuPlantis <[hidden email]> wrote:
QuickFIX/J Documentation: <a href="http://www.quickfixj.org/documentation/ QuickFIX/J" rel="noreferrer" target="_blank">http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


Howdy, all.

We've got a counter-party that's sending messages with tag 355 and
omitting tag 354. This causes QFJ to be unable to extract the contents
of field 355. We're running 1.6.3 right now, and the trouble comes up in
Message.extractField as the length is assumed to be present.

Are there any opinions about whether 354 is required if 355 is included?
The FIX spec lists both as "not required" as do the data dictionaries,
but, it seems to me that 354 should be conditionally required if 355 is
present.

My question is, what are the opinions of the list members as to how
snarly I can get with the counter about this issue?

Cheers.


--
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
888.868.4884 +1.541.306.6556
http://www.marketcetera.org


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Fields 355 & 354

Colin DuPlantis
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



I don't disagree, but 355 et al are not part of a repeating group.


On 05/03/2017 10:55 AM, Grant Birchmeier wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




Per FIX spec, the first element of a repeating group is the delimiter item that signals the start of the group (and the end of the last one), and is always mandatory.  I believe the spec is not ambiguous about this point.

Whether that gets your counterparty to change their impl to behave or not, that's up in the air.

On Wed, May 3, 2017 at 12:11 PM, Colin DuPlantis <[hidden email]> wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J
Support: http://www.quickfixj.org/support/


Howdy, all.

We've got a counter-party that's sending messages with tag 355 and
omitting tag 354. This causes QFJ to be unable to extract the contents
of field 355. We're running 1.6.3 right now, and the trouble comes up in
Message.extractField as the length is assumed to be present.

Are there any opinions about whether 354 is required if 355 is included?
The FIX spec lists both as "not required" as do the data dictionaries,
but, it seems to me that 354 should be conditionally required if 355 is
present.

My question is, what are the opinions of the list members as to how
snarly I can get with the counter about this issue?

Cheers.


--
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
888.868.4884 +1.541.306.6556
http://www.marketcetera.org


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users

-- 
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
888.868.4884 +1.541.306.6556
http://www.marketcetera.org

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Fields 355 & 354

Grant Birchmeier
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



Ah, silly me.  I jumped to that assumption because I've discussed a similar problem on the QF/n list before.  Apologies.

To your actual problem, I'm looking at the FIX spec.  In FIX-5.0_VOL-1.pdf (Dec 2006 version), I see this:

data: Raw data with no format or content restrictions. Data fields are always immediately preceded by a length field. The length field should specify the number of bytes of the value of the UdataU field (up to but not including the terminating SOH). Caution: the value of one of these fields may contain the delimiter (SOH) character. Note that the value specified for this field should be followed by the delimiter (SOH) character as all fields are terminated with an “SOH”.

​And this:​

There is the possibility that an SOH may be included in the character data when using UNICODE encoding. To avoid parsing problems, a FIX engine should use the EncodedLen value to extract the proper number of bytes.

​Both of those passages imply that the length field is optional.  Which is surprising to me.

But you can make the case to them that their message might accidentally include SOH when converted to ASCII, therefore they really should be giving you the length so it can be read correctly.  Without the length, it's very difficult to ignore the accidental SOH and find the correct end of the encoded field.

(Does QF/j support using the Encoded<X>Len field to correctly parse encoded SOHs?  I know that QF/n will choke on those.)

-Grant


On Wed, May 3, 2017 at 1:29 PM, Colin DuPlantis <[hidden email]> wrote:
QuickFIX/J Documentation: <a href="http://www.quickfixj.org/documentation/ QuickFIX/J" rel="noreferrer" target="_blank">http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



I don't disagree, but 355 et al are not part of a repeating group.


On 05/03/2017 10:55 AM, Grant Birchmeier wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




Per FIX spec, the first element of a repeating group is the delimiter item that signals the start of the group (and the end of the last one), and is always mandatory.  I believe the spec is not ambiguous about this point.

Whether that gets your counterparty to change their impl to behave or not, that's up in the air.

On Wed, May 3, 2017 at 12:11 PM, Colin DuPlantis <[hidden email]> wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J
Support: http://www.quickfixj.org/support/


Howdy, all.

We've got a counter-party that's sending messages with tag 355 and
omitting tag 354. This causes QFJ to be unable to extract the contents
of field 355. We're running 1.6.3 right now, and the trouble comes up in
Message.extractField as the length is assumed to be present.

Are there any opinions about whether 354 is required if 355 is included?
The FIX spec lists both as "not required" as do the data dictionaries,
but, it seems to me that 354 should be conditionally required if 355 is
present.

My question is, what are the opinions of the list members as to how
snarly I can get with the counter about this issue?

Cheers.


--
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
<a href="tel:(888)%20868-4884" value="+18888684884" target="_blank">888.868.4884 <a href="tel:(541)%20306-6556" value="+15413066556" target="_blank">+1.541.306.6556
http://www.marketcetera.org


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users

-- 
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
<a href="tel:(888)%20868-4884" value="+18888684884" target="_blank">888.868.4884 <a href="tel:(541)%20306-6556" value="+15413066556" target="_blank">+1.541.306.6556
http://www.marketcetera.org

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users




--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Fields 355 & 354

Colin DuPlantis
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



There is relevant code to handle SOH, but I haven't verified it.

QFJ, at this time, rejects the message as invalid if the len isn't included. I could patch it up to take a swing at the field, assuming no SOH, or just omit the field if the len is omitted.

Thoughts? Opinions? Manifestos?


On 05/03/2017 11:54 AM, Grant Birchmeier wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




Ah, silly me.  I jumped to that assumption because I've discussed a similar problem on the QF/n list before.  Apologies.

To your actual problem, I'm looking at the FIX spec.  In FIX-5.0_VOL-1.pdf (Dec 2006 version), I see this:

data: Raw data with no format or content restrictions. Data fields are always immediately preceded by a length field. The length field should specify the number of bytes of the value of the UdataU field (up to but not including the terminating SOH). Caution: the value of one of these fields may contain the delimiter (SOH) character. Note that the value specified for this field should be followed by the delimiter (SOH) character as all fields are terminated with an “SOH”.

​And this:​

There is the possibility that an SOH may be included in the character data when using UNICODE encoding. To avoid parsing problems, a FIX engine should use the EncodedLen value to extract the proper number of bytes.

​Both of those passages imply that the length field is optional.  Which is surprising to me.

But you can make the case to them that their message might accidentally include SOH when converted to ASCII, therefore they really should be giving you the length so it can be read correctly.  Without the length, it's very difficult to ignore the accidental SOH and find the correct end of the encoded field.

(Does QF/j support using the Encoded<X>Len field to correctly parse encoded SOHs?  I know that QF/n will choke on those.)

-Grant


On Wed, May 3, 2017 at 1:29 PM, Colin DuPlantis <[hidden email]> wrote:

I don't disagree, but 355 et al are not part of a repeating group.


On 05/03/2017 10:55 AM, Grant Birchmeier wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


Per FIX spec, the first element of a repeating group is the delimiter item that signals the start of the group (and the end of the last one), and is always mandatory.  I believe the spec is not ambiguous about this point.
Whether that gets your counterparty to change their impl to behave or not, that's up in the air.
On Wed, May 3, 2017 at 12:11 PM, Colin DuPlantis <[hidden email]> wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/ QuickFIX/J Support: http://www.quickfixj.org/support/ Howdy, all. We've got a counter-party that's sending messages with tag 355 and omitting tag 354. This causes QFJ to be unable to extract the contents of field 355. We're running 1.6.3 right now, and the trouble comes up in Message.extractField as the length is assumed to be present. Are there any opinions about whether 354 is required if 355 is included? The FIX spec lists both as "not required" as do the data dictionaries, but, it seems to me that 354 should be conditionally required if 355 is present. My question is, what are the opinions of the list members as to how snarly I can get with the counter about this issue? Cheers. -- Colin DuPlantis Chief Architect, Marketcetera Download, Run, Trade <a moz-do-not-send="true" href="tel:%28888%29%20868-4884" value="+18888684884" target="_blank">888.868.4884 <a moz-do-not-send="true" href="tel:%28541%29%20306-6556" value="+15413066556" target="_blank">+1.541.306.6556 http://www.marketcetera.org ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Quickfixj-users mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/quickfixj-users
--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
-- 
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
<a moz-do-not-send="true" href="tel:%28888%29%20868-4884" value="+18888684884" target="_blank">888.868.4884 <a moz-do-not-send="true" href="tel:%28541%29%20306-6556" value="+15413066556" target="_blank">+1.541.306.6556
http://www.marketcetera.org
------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ Quickfixj-users mailing list [hidden email] https://lists.sourceforge.net/lists/listinfo/quickfixj-users
--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
-- 
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
888.868.4884 +1.541.306.6556
http://www.marketcetera.org

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Fields 355 & 354

Christoph John
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


When browsing e.g. Advertisement message in fiximate it clearly says about Tag 354:
Must be set if EncodedText field is specified and must immediately precede it.

So IMHO the length field is clearly required.

Cheers,
Chris.
----- Original Message -----
From: Colin DuPlantis <[hidden email]>
To: [hidden email]
Sent: Wed, 03 May 2017 20:58:53 +0200 (CEST)
Subject: Re: [Quickfixj-users] Fields 355 & 354

QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Fields 355 & 354

Colin DuPlantis
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


Chris,

How you feel about a modification to return null for an EncodedText
field if the length is missing:

What are your thoughts on changing Message.extractField to return null
instead of invalidate the message in this case?

Current (1.6.x) code:

         if (dataDictionary != null && dataDictionary.isDataField(tag)) {
             /* Assume length field is 1 less. */
             int lengthField = tag - 1;
             /* Special case for Signature which violates above
assumption. */
             if (tag == 89) {
                 lengthField = 93;
             }
             int fieldLength;
             try {
                 fieldLength = fields.getInt(lengthField);
             } catch (final FieldNotFound e) {
                 throw new InvalidMessage("Tag " + e.field + " not found
in " + messageData);
             }

             // since length is in bytes but data is a string, and it
may also contain an SOH,
             // we find the real field-ending SOH by checking the
encoded bytes length
             // (we avoid re-encoding when the chars length equals the
bytes length, e.g. ASCII text,
             // by assuming the chars length is always smaller than the
encoded bytes length)
             while (sohOffset - equalsOffset - 1 < fieldLength
                     && messageData.substring(equalsOffset + 1,
sohOffset).getBytes(CharsetSupport.getCharsetInstance()).length <
fieldLength) {
                 sohOffset = messageData.indexOf('\001', sohOffset + 1);
                 if (sohOffset == -1) {
                     throw new InvalidMessage("SOH not found at end of
field: " + tag + " in " + messageData);
                 }
             }
         }

Proposed change:

         if (dataDictionary != null && dataDictionary.isDataField(tag)) {
             /* Assume length field is 1 less. */
             int lengthField = tag - 1;
             /* Special case for Signature which violates above
assumption. */
             if (tag == 89) {
                 lengthField = 93;
             }
             int fieldLength;
             try {
                 fieldLength = fields.getInt(lengthField);
             } catch (final FieldNotFound e) {
// proposed change begins
                 return null;
// proposed change ends
             }

             // since length is in bytes but data is a string, and it
may also contain an SOH,
             // we find the real field-ending SOH by checking the
encoded bytes length
             // (we avoid re-encoding when the chars length equals the
bytes length, e.g. ASCII text,
             // by assuming the chars length is always smaller than the
encoded bytes length)
             while (sohOffset - equalsOffset - 1 < fieldLength
                     && messageData.substring(equalsOffset + 1,
sohOffset).getBytes(CharsetSupport.getCharsetInstance()).length <
fieldLength) {
                 sohOffset = messageData.indexOf('\001', sohOffset + 1);
                 if (sohOffset == -1) {
                     throw new InvalidMessage("SOH not found at end of
field: " + tag + " in " + messageData);
                 }
             }
         }



On 05/03/2017 12:26 PM, Christoph John wrote:

> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> When browsing e.g. Advertisement message in fiximate it clearly says about Tag 354:
> Must be set if EncodedText field is specified and must immediately precede it.
>
> So IMHO the length field is clearly required.
>
> Cheers,
> Chris.
> ----- Original Message -----
> From: Colin DuPlantis <[hidden email]>
> To: [hidden email]
> Sent: Wed, 03 May 2017 20:58:53 +0200 (CEST)
> Subject: Re: [Quickfixj-users] Fields 355 & 354
>
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
>
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

--
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
888.868.4884 +1.541.306.6556
http://www.marketcetera.org


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Fields 355 & 354

Christoph John
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


Hi Colin,

hmm, not sure. If you return null there, it looks to me like he is then skipping the parsing of the
data field also.
But to be 100% sure it would be good if we had a unit test which tests the new behaviour with a
missing length field and then with a data field that is OK and one that is not OK (i.e. includes
SOH). I can also help you with the unit test if you like. Should be somewhere in MessageTest IIRC.

Cheers,
Chris.


On 04/05/17 00:54, Colin DuPlantis wrote:

> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> Chris,
>
> How you feel about a modification to return null for an EncodedText
> field if the length is missing:
>
> What are your thoughts on changing Message.extractField to return null
> instead of invalidate the message in this case?
>
> Current (1.6.x) code:
>
>           if (dataDictionary != null && dataDictionary.isDataField(tag)) {
>               /* Assume length field is 1 less. */
>               int lengthField = tag - 1;
>               /* Special case for Signature which violates above
> assumption. */
>               if (tag == 89) {
>                   lengthField = 93;
>               }
>               int fieldLength;
>               try {
>                   fieldLength = fields.getInt(lengthField);
>               } catch (final FieldNotFound e) {
>                   throw new InvalidMessage("Tag " + e.field + " not found
> in " + messageData);
>               }
>
>               // since length is in bytes but data is a string, and it
> may also contain an SOH,
>               // we find the real field-ending SOH by checking the
> encoded bytes length
>               // (we avoid re-encoding when the chars length equals the
> bytes length, e.g. ASCII text,
>               // by assuming the chars length is always smaller than the
> encoded bytes length)
>               while (sohOffset - equalsOffset - 1 < fieldLength
>                       && messageData.substring(equalsOffset + 1,
> sohOffset).getBytes(CharsetSupport.getCharsetInstance()).length <
> fieldLength) {
>                   sohOffset = messageData.indexOf('\001', sohOffset + 1);
>                   if (sohOffset == -1) {
>                       throw new InvalidMessage("SOH not found at end of
> field: " + tag + " in " + messageData);
>                   }
>               }
>           }
>
> Proposed change:
>
>           if (dataDictionary != null && dataDictionary.isDataField(tag)) {
>               /* Assume length field is 1 less. */
>               int lengthField = tag - 1;
>               /* Special case for Signature which violates above
> assumption. */
>               if (tag == 89) {
>                   lengthField = 93;
>               }
>               int fieldLength;
>               try {
>                   fieldLength = fields.getInt(lengthField);
>               } catch (final FieldNotFound e) {
> // proposed change begins
>                   return null;
> // proposed change ends
>               }
>
>               // since length is in bytes but data is a string, and it
> may also contain an SOH,
>               // we find the real field-ending SOH by checking the
> encoded bytes length
>               // (we avoid re-encoding when the chars length equals the
> bytes length, e.g. ASCII text,
>               // by assuming the chars length is always smaller than the
> encoded bytes length)
>               while (sohOffset - equalsOffset - 1 < fieldLength
>                       && messageData.substring(equalsOffset + 1,
> sohOffset).getBytes(CharsetSupport.getCharsetInstance()).length <
> fieldLength) {
>                   sohOffset = messageData.indexOf('\001', sohOffset + 1);
>                   if (sohOffset == -1) {
>                       throw new InvalidMessage("SOH not found at end of
> field: " + tag + " in " + messageData);
>                   }
>               }
>           }
>
>
>
> On 05/03/2017 12:26 PM, Christoph John wrote:
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>
>>
>> When browsing e.g. Advertisement message in fiximate it clearly says about Tag 354:
>> Must be set if EncodedText field is specified and must immediately precede it.
>>
>> So IMHO the length field is clearly required.
>>
>> Cheers,
>> Chris.
>> ----- Original Message -----
>> From: Colin DuPlantis <[hidden email]>
>> To: [hidden email]
>> Sent: Wed, 03 May 2017 20:58:53 +0200 (CEST)
>> Subject: Re: [Quickfixj-users] Fields 355 & 354
>>
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Quickfixj-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

--
Christoph John
Development & Support
Direct: +49 241 557080-28
Mailto:[hidden email]
       


http://www.macd.com <http://www.macd.com/>
----------------------------------------------------------------------------------------------------
       
----------------------------------------------------------------------------------------------------
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
Tel: +49 241 557080-0 | Fax: +49 241 557080-10
         Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663

Geschäftsführer: George Macdonald
----------------------------------------------------------------------------------------------------
       
----------------------------------------------------------------------------------------------------

take care of the environment - print only if necessary

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Fields 355 & 354

Philip Whitehouse
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


This feels like the wrong solution.

If you aren't being sent a real data field, then you can just change
your DataDictionary for that field.

The point of the length is so that if the raw data field contains (for
example) the field separator character, then it's not considered the end
of the field.

If you're using data fields as data fields, then the length is
necessary. If you're actually using it as just a string of data then you
should change your dictionary to expect a string and accept that if the
data contains a field separator character it will break stuff.

Making it so that the extractField behaviour changes is a bad hack.

- Philip Whitehouse


On 2017-05-04 08:26, Christoph John wrote:

> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> Hi Colin,
>
> hmm, not sure. If you return null there, it looks to me like he is
> then skipping the parsing of the
> data field also.
> But to be 100% sure it would be good if we had a unit test which tests
> the new behaviour with a
> missing length field and then with a data field that is OK and one
> that is not OK (i.e. includes
> SOH). I can also help you with the unit test if you like. Should be
> somewhere in MessageTest IIRC.
>
> Cheers,
> Chris.
>
>
> On 04/05/17 00:54, Colin DuPlantis wrote:
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>
>>
>> Chris,
>>
>> How you feel about a modification to return null for an EncodedText
>> field if the length is missing:
>>
>> What are your thoughts on changing Message.extractField to return null
>> instead of invalidate the message in this case?
>>
>> Current (1.6.x) code:
>>
>>           if (dataDictionary != null &&
>> dataDictionary.isDataField(tag)) {
>>               /* Assume length field is 1 less. */
>>               int lengthField = tag - 1;
>>               /* Special case for Signature which violates above
>> assumption. */
>>               if (tag == 89) {
>>                   lengthField = 93;
>>               }
>>               int fieldLength;
>>               try {
>>                   fieldLength = fields.getInt(lengthField);
>>               } catch (final FieldNotFound e) {
>>                   throw new InvalidMessage("Tag " + e.field + " not
>> found
>> in " + messageData);
>>               }
>>
>>               // since length is in bytes but data is a string, and it
>> may also contain an SOH,
>>               // we find the real field-ending SOH by checking the
>> encoded bytes length
>>               // (we avoid re-encoding when the chars length equals
>> the
>> bytes length, e.g. ASCII text,
>>               // by assuming the chars length is always smaller than
>> the
>> encoded bytes length)
>>               while (sohOffset - equalsOffset - 1 < fieldLength
>>                       && messageData.substring(equalsOffset + 1,
>> sohOffset).getBytes(CharsetSupport.getCharsetInstance()).length <
>> fieldLength) {
>>                   sohOffset = messageData.indexOf('\001', sohOffset +
>> 1);
>>                   if (sohOffset == -1) {
>>                       throw new InvalidMessage("SOH not found at end
>> of
>> field: " + tag + " in " + messageData);
>>                   }
>>               }
>>           }
>>
>> Proposed change:
>>
>>           if (dataDictionary != null &&
>> dataDictionary.isDataField(tag)) {
>>               /* Assume length field is 1 less. */
>>               int lengthField = tag - 1;
>>               /* Special case for Signature which violates above
>> assumption. */
>>               if (tag == 89) {
>>                   lengthField = 93;
>>               }
>>               int fieldLength;
>>               try {
>>                   fieldLength = fields.getInt(lengthField);
>>               } catch (final FieldNotFound e) {
>> // proposed change begins
>>                   return null;
>> // proposed change ends
>>               }
>>
>>               // since length is in bytes but data is a string, and it
>> may also contain an SOH,
>>               // we find the real field-ending SOH by checking the
>> encoded bytes length
>>               // (we avoid re-encoding when the chars length equals
>> the
>> bytes length, e.g. ASCII text,
>>               // by assuming the chars length is always smaller than
>> the
>> encoded bytes length)
>>               while (sohOffset - equalsOffset - 1 < fieldLength
>>                       && messageData.substring(equalsOffset + 1,
>> sohOffset).getBytes(CharsetSupport.getCharsetInstance()).length <
>> fieldLength) {
>>                   sohOffset = messageData.indexOf('\001', sohOffset +
>> 1);
>>                   if (sohOffset == -1) {
>>                       throw new InvalidMessage("SOH not found at end
>> of
>> field: " + tag + " in " + messageData);
>>                   }
>>               }
>>           }
>>
>>
>>
>> On 05/03/2017 12:26 PM, Christoph John wrote:
>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>
>>>
>>> When browsing e.g. Advertisement message in fiximate it clearly says
>>> about Tag 354:
>>> Must be set if EncodedText field is specified and must immediately
>>> precede it.
>>>
>>> So IMHO the length field is clearly required.
>>>
>>> Cheers,
>>> Chris.
>>> ----- Original Message -----
>>> From: Colin DuPlantis <[hidden email]>
>>> To: [hidden email]
>>> Sent: Wed, 03 May 2017 20:58:53 +0200 (CEST)
>>> Subject: Re: [Quickfixj-users] Fields 355 & 354
>>>
>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Quickfixj-users mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>
> --
> Christoph John
> Development & Support
> Direct: +49 241 557080-28
> Mailto:[hidden email]
>
>
>
> http://www.macd.com <http://www.macd.com/>
> ----------------------------------------------------------------------------------------------------
>
> ----------------------------------------------------------------------------------------------------
> MACD GmbH
> Oppenhoffallee 103
> D-52066 Aachen
> Tel: +49 241 557080-0 | Fax: +49 241 557080-10
> Amtsgericht Aachen: HRB 8151
> Ust.-Id: DE 813021663
>
> Geschäftsführer: George Macdonald
> ----------------------------------------------------------------------------------------------------
>
> ----------------------------------------------------------------------------------------------------
>
> take care of the environment - print only if necessary
>
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Fields 355 & 354

Colin DuPlantis
In reply to this post by Christoph John
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


Yes, I agree that a missing len field would omit the data field as well.
As of now, a missing len field will invalidate the whole message.

I can certainly add to/adjust the tests as necessary.

Will there be another 1.6.x release, or should this work go into 1.7.x?

Thanks.


On 05/04/2017 12:26 AM, Christoph John wrote:

> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> Hi Colin,
>
> hmm, not sure. If you return null there, it looks to me like he is then skipping the parsing of the
> data field also.
> But to be 100% sure it would be good if we had a unit test which tests the new behaviour with a
> missing length field and then with a data field that is OK and one that is not OK (i.e. includes
> SOH). I can also help you with the unit test if you like. Should be somewhere in MessageTest IIRC.
>
> Cheers,
> Chris.
>
>
> On 04/05/17 00:54, Colin DuPlantis wrote:
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>
>>
>> Chris,
>>
>> How you feel about a modification to return null for an EncodedText
>> field if the length is missing:
>>
>> What are your thoughts on changing Message.extractField to return null
>> instead of invalidate the message in this case?
>>
>> Current (1.6.x) code:
>>
>>            if (dataDictionary != null && dataDictionary.isDataField(tag)) {
>>                /* Assume length field is 1 less. */
>>                int lengthField = tag - 1;
>>                /* Special case for Signature which violates above
>> assumption. */
>>                if (tag == 89) {
>>                    lengthField = 93;
>>                }
>>                int fieldLength;
>>                try {
>>                    fieldLength = fields.getInt(lengthField);
>>                } catch (final FieldNotFound e) {
>>                    throw new InvalidMessage("Tag " + e.field + " not found
>> in " + messageData);
>>                }
>>
>>                // since length is in bytes but data is a string, and it
>> may also contain an SOH,
>>                // we find the real field-ending SOH by checking the
>> encoded bytes length
>>                // (we avoid re-encoding when the chars length equals the
>> bytes length, e.g. ASCII text,
>>                // by assuming the chars length is always smaller than the
>> encoded bytes length)
>>                while (sohOffset - equalsOffset - 1 < fieldLength
>>                        && messageData.substring(equalsOffset + 1,
>> sohOffset).getBytes(CharsetSupport.getCharsetInstance()).length <
>> fieldLength) {
>>                    sohOffset = messageData.indexOf('\001', sohOffset + 1);
>>                    if (sohOffset == -1) {
>>                        throw new InvalidMessage("SOH not found at end of
>> field: " + tag + " in " + messageData);
>>                    }
>>                }
>>            }
>>
>> Proposed change:
>>
>>            if (dataDictionary != null && dataDictionary.isDataField(tag)) {
>>                /* Assume length field is 1 less. */
>>                int lengthField = tag - 1;
>>                /* Special case for Signature which violates above
>> assumption. */
>>                if (tag == 89) {
>>                    lengthField = 93;
>>                }
>>                int fieldLength;
>>                try {
>>                    fieldLength = fields.getInt(lengthField);
>>                } catch (final FieldNotFound e) {
>> // proposed change begins
>>                    return null;
>> // proposed change ends
>>                }
>>
>>                // since length is in bytes but data is a string, and it
>> may also contain an SOH,
>>                // we find the real field-ending SOH by checking the
>> encoded bytes length
>>                // (we avoid re-encoding when the chars length equals the
>> bytes length, e.g. ASCII text,
>>                // by assuming the chars length is always smaller than the
>> encoded bytes length)
>>                while (sohOffset - equalsOffset - 1 < fieldLength
>>                        && messageData.substring(equalsOffset + 1,
>> sohOffset).getBytes(CharsetSupport.getCharsetInstance()).length <
>> fieldLength) {
>>                    sohOffset = messageData.indexOf('\001', sohOffset + 1);
>>                    if (sohOffset == -1) {
>>                        throw new InvalidMessage("SOH not found at end of
>> field: " + tag + " in " + messageData);
>>                    }
>>                }
>>            }
>>
>>
>>
>> On 05/03/2017 12:26 PM, Christoph John wrote:
>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>
>>>
>>> When browsing e.g. Advertisement message in fiximate it clearly says about Tag 354:
>>> Must be set if EncodedText field is specified and must immediately precede it.
>>>
>>> So IMHO the length field is clearly required.
>>>
>>> Cheers,
>>> Chris.
>>> ----- Original Message -----
>>> From: Colin DuPlantis <[hidden email]>
>>> To: [hidden email]
>>> Sent: Wed, 03 May 2017 20:58:53 +0200 (CEST)
>>> Subject: Re: [Quickfixj-users] Fields 355 & 354
>>>
>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Check out the vibrant tech community on one of the world's most
>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>> _______________________________________________
>>> Quickfixj-users mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Fields 355 & 354

Colin DuPlantis
In reply to this post by Philip Whitehouse
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


Philip,

I re-read the FIX spec and I am going to reverse my position and agree
with you.

I was all prepared to quote the spec and argue, but, on review, I think
you're right. I think 354 is required if 355 is present (and similar
relationships for other data/len fields).

I think the best course of action is to leave the current behavior as-is.

Thanks.

- Colin

On 05/04/2017 05:32 AM, Philip Whitehouse wrote:

> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> This feels like the wrong solution.
>
> If you aren't being sent a real data field, then you can just change
> your DataDictionary for that field.
>
> The point of the length is so that if the raw data field contains (for
> example) the field separator character, then it's not considered the end
> of the field.
>
> If you're using data fields as data fields, then the length is
> necessary. If you're actually using it as just a string of data then you
> should change your dictionary to expect a string and accept that if the
> data contains a field separator character it will break stuff.
>
> Making it so that the extractField behaviour changes is a bad hack.
>
> - Philip Whitehouse
>
>
> On 2017-05-04 08:26, Christoph John wrote:
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>
>>
>> Hi Colin,
>>
>> hmm, not sure. If you return null there, it looks to me like he is
>> then skipping the parsing of the
>> data field also.
>> But to be 100% sure it would be good if we had a unit test which tests
>> the new behaviour with a
>> missing length field and then with a data field that is OK and one
>> that is not OK (i.e. includes
>> SOH). I can also help you with the unit test if you like. Should be
>> somewhere in MessageTest IIRC.
>>
>> Cheers,
>> Chris.
>>
>>
>> On 04/05/17 00:54, Colin DuPlantis wrote:
>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>
>>>
>>> Chris,
>>>
>>> How you feel about a modification to return null for an EncodedText
>>> field if the length is missing:
>>>
>>> What are your thoughts on changing Message.extractField to return null
>>> instead of invalidate the message in this case?
>>>
>>> Current (1.6.x) code:
>>>
>>>            if (dataDictionary != null &&
>>> dataDictionary.isDataField(tag)) {
>>>                /* Assume length field is 1 less. */
>>>                int lengthField = tag - 1;
>>>                /* Special case for Signature which violates above
>>> assumption. */
>>>                if (tag == 89) {
>>>                    lengthField = 93;
>>>                }
>>>                int fieldLength;
>>>                try {
>>>                    fieldLength = fields.getInt(lengthField);
>>>                } catch (final FieldNotFound e) {
>>>                    throw new InvalidMessage("Tag " + e.field + " not
>>> found
>>> in " + messageData);
>>>                }
>>>
>>>                // since length is in bytes but data is a string, and it
>>> may also contain an SOH,
>>>                // we find the real field-ending SOH by checking the
>>> encoded bytes length
>>>                // (we avoid re-encoding when the chars length equals
>>> the
>>> bytes length, e.g. ASCII text,
>>>                // by assuming the chars length is always smaller than
>>> the
>>> encoded bytes length)
>>>                while (sohOffset - equalsOffset - 1 < fieldLength
>>>                        && messageData.substring(equalsOffset + 1,
>>> sohOffset).getBytes(CharsetSupport.getCharsetInstance()).length <
>>> fieldLength) {
>>>                    sohOffset = messageData.indexOf('\001', sohOffset +
>>> 1);
>>>                    if (sohOffset == -1) {
>>>                        throw new InvalidMessage("SOH not found at end
>>> of
>>> field: " + tag + " in " + messageData);
>>>                    }
>>>                }
>>>            }
>>>
>>> Proposed change:
>>>
>>>            if (dataDictionary != null &&
>>> dataDictionary.isDataField(tag)) {
>>>                /* Assume length field is 1 less. */
>>>                int lengthField = tag - 1;
>>>                /* Special case for Signature which violates above
>>> assumption. */
>>>                if (tag == 89) {
>>>                    lengthField = 93;
>>>                }
>>>                int fieldLength;
>>>                try {
>>>                    fieldLength = fields.getInt(lengthField);
>>>                } catch (final FieldNotFound e) {
>>> // proposed change begins
>>>                    return null;
>>> // proposed change ends
>>>                }
>>>
>>>                // since length is in bytes but data is a string, and it
>>> may also contain an SOH,
>>>                // we find the real field-ending SOH by checking the
>>> encoded bytes length
>>>                // (we avoid re-encoding when the chars length equals
>>> the
>>> bytes length, e.g. ASCII text,
>>>                // by assuming the chars length is always smaller than
>>> the
>>> encoded bytes length)
>>>                while (sohOffset - equalsOffset - 1 < fieldLength
>>>                        && messageData.substring(equalsOffset + 1,
>>> sohOffset).getBytes(CharsetSupport.getCharsetInstance()).length <
>>> fieldLength) {
>>>                    sohOffset = messageData.indexOf('\001', sohOffset +
>>> 1);
>>>                    if (sohOffset == -1) {
>>>                        throw new InvalidMessage("SOH not found at end
>>> of
>>> field: " + tag + " in " + messageData);
>>>                    }
>>>                }
>>>            }
>>>
>>>
>>>
>>> On 05/03/2017 12:26 PM, Christoph John wrote:
>>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>>
>>>>
>>>> When browsing e.g. Advertisement message in fiximate it clearly says
>>>> about Tag 354:
>>>> Must be set if EncodedText field is specified and must immediately
>>>> precede it.
>>>>
>>>> So IMHO the length field is clearly required.
>>>>
>>>> Cheers,
>>>> Chris.
>>>> ----- Original Message -----
>>>> From: Colin DuPlantis <[hidden email]>
>>>> To: [hidden email]
>>>> Sent: Wed, 03 May 2017 20:58:53 +0200 (CEST)
>>>> Subject: Re: [Quickfixj-users] Fields 355 & 354
>>>>
>>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Check out the vibrant tech community on one of the world's most
>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>> _______________________________________________
>>>> Quickfixj-users mailing list
>>>> [hidden email]
>>>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>> --
>> Christoph John
>> Development & Support
>> Direct: +49 241 557080-28
>> Mailto:[hidden email]
>>
>>
>>
>> http://www.macd.com <http://www.macd.com/>
>> ----------------------------------------------------------------------------------------------------
>>
>> ----------------------------------------------------------------------------------------------------
>> MACD GmbH
>> Oppenhoffallee 103
>> D-52066 Aachen
>> Tel: +49 241 557080-0 | Fax: +49 241 557080-10
>> Amtsgericht Aachen: HRB 8151
>> Ust.-Id: DE 813021663
>>
>> Geschäftsführer: George Macdonald
>> ----------------------------------------------------------------------------------------------------
>>
>> ----------------------------------------------------------------------------------------------------
>>
>> take care of the environment - print only if necessary
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Quickfixj-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Fields 355 & 354

Christoph John
In reply to this post by Philip Whitehouse
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


I think there is a difference between "bad hack" and "adapting himself to different counterparties".
;) As we all know there are counterparties which are more FIX-compatible than others.
But maybe you are right: changing the data dictionary to treat the specific data field as String
field might be a more feasible solution.

Cheers,
Chris.


On 04/05/17 14:32, Philip Whitehouse wrote:

> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> This feels like the wrong solution.
>
> If you aren't being sent a real data field, then you can just change
> your DataDictionary for that field.
>
> The point of the length is so that if the raw data field contains (for
> example) the field separator character, then it's not considered the end
> of the field.
>
> If you're using data fields as data fields, then the length is
> necessary. If you're actually using it as just a string of data then you
> should change your dictionary to expect a string and accept that if the
> data contains a field separator character it will break stuff.
>
> Making it so that the extractField behaviour changes is a bad hack.
>
> - Philip Whitehouse
>
>
> On 2017-05-04 08:26, Christoph John wrote:
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>
>>
>> Hi Colin,
>>
>> hmm, not sure. If you return null there, it looks to me like he is
>> then skipping the parsing of the
>> data field also.
>> But to be 100% sure it would be good if we had a unit test which tests
>> the new behaviour with a
>> missing length field and then with a data field that is OK and one
>> that is not OK (i.e. includes
>> SOH). I can also help you with the unit test if you like. Should be
>> somewhere in MessageTest IIRC.
>>
>> Cheers,
>> Chris.
>>
>>
>> On 04/05/17 00:54, Colin DuPlantis wrote:
>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>
>>>
>>> Chris,
>>>
>>> How you feel about a modification to return null for an EncodedText
>>> field if the length is missing:
>>>
>>> What are your thoughts on changing Message.extractField to return null
>>> instead of invalidate the message in this case?
>>>
>>> Current (1.6.x) code:
>>>
>>>            if (dataDictionary != null &&
>>> dataDictionary.isDataField(tag)) {
>>>                /* Assume length field is 1 less. */
>>>                int lengthField = tag - 1;
>>>                /* Special case for Signature which violates above
>>> assumption. */
>>>                if (tag == 89) {
>>>                    lengthField = 93;
>>>                }
>>>                int fieldLength;
>>>                try {
>>>                    fieldLength = fields.getInt(lengthField);
>>>                } catch (final FieldNotFound e) {
>>>                    throw new InvalidMessage("Tag " + e.field + " not
>>> found
>>> in " + messageData);
>>>                }
>>>
>>>                // since length is in bytes but data is a string, and it
>>> may also contain an SOH,
>>>                // we find the real field-ending SOH by checking the
>>> encoded bytes length
>>>                // (we avoid re-encoding when the chars length equals
>>> the
>>> bytes length, e.g. ASCII text,
>>>                // by assuming the chars length is always smaller than
>>> the
>>> encoded bytes length)
>>>                while (sohOffset - equalsOffset - 1 < fieldLength
>>>                        && messageData.substring(equalsOffset + 1,
>>> sohOffset).getBytes(CharsetSupport.getCharsetInstance()).length <
>>> fieldLength) {
>>>                    sohOffset = messageData.indexOf('\001', sohOffset +
>>> 1);
>>>                    if (sohOffset == -1) {
>>>                        throw new InvalidMessage("SOH not found at end
>>> of
>>> field: " + tag + " in " + messageData);
>>>                    }
>>>                }
>>>            }
>>>
>>> Proposed change:
>>>
>>>            if (dataDictionary != null &&
>>> dataDictionary.isDataField(tag)) {
>>>                /* Assume length field is 1 less. */
>>>                int lengthField = tag - 1;
>>>                /* Special case for Signature which violates above
>>> assumption. */
>>>                if (tag == 89) {
>>>                    lengthField = 93;
>>>                }
>>>                int fieldLength;
>>>                try {
>>>                    fieldLength = fields.getInt(lengthField);
>>>                } catch (final FieldNotFound e) {
>>> // proposed change begins
>>>                    return null;
>>> // proposed change ends
>>>                }
>>>
>>>                // since length is in bytes but data is a string, and it
>>> may also contain an SOH,
>>>                // we find the real field-ending SOH by checking the
>>> encoded bytes length
>>>                // (we avoid re-encoding when the chars length equals
>>> the
>>> bytes length, e.g. ASCII text,
>>>                // by assuming the chars length is always smaller than
>>> the
>>> encoded bytes length)
>>>                while (sohOffset - equalsOffset - 1 < fieldLength
>>>                        && messageData.substring(equalsOffset + 1,
>>> sohOffset).getBytes(CharsetSupport.getCharsetInstance()).length <
>>> fieldLength) {
>>>                    sohOffset = messageData.indexOf('\001', sohOffset +
>>> 1);
>>>                    if (sohOffset == -1) {
>>>                        throw new InvalidMessage("SOH not found at end
>>> of
>>> field: " + tag + " in " + messageData);
>>>                    }
>>>                }
>>>            }
>>>
>>>
>>>
>>> On 05/03/2017 12:26 PM, Christoph John wrote:
>>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>>
>>>>
>>>> When browsing e.g. Advertisement message in fiximate it clearly says
>>>> about Tag 354:
>>>> Must be set if EncodedText field is specified and must immediately
>>>> precede it.
>>>>
>>>> So IMHO the length field is clearly required.
>>>>
>>>> Cheers,
>>>> Chris.
>>>> ----- Original Message -----
>>>> From: Colin DuPlantis <[hidden email]>
>>>> To: [hidden email]
>>>> Sent: Wed, 03 May 2017 20:58:53 +0200 (CEST)
>>>> Subject: Re: [Quickfixj-users] Fields 355 & 354
>>>>
>>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Check out the vibrant tech community on one of the world's most
>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>> _______________________________________________
>>>> Quickfixj-users mailing list
>>>> [hidden email]
>>>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>> --
>> Christoph John
>> Development & Support
>> Direct: +49 241 557080-28
>> Mailto:[hidden email]
>>
>>
>>
>> http://www.macd.com <http://www.macd.com/>
>> ----------------------------------------------------------------------------------------------------
>>
>> ----------------------------------------------------------------------------------------------------
>> MACD GmbH
>> Oppenhoffallee 103
>> D-52066 Aachen
>> Tel: +49 241 557080-0 | Fax: +49 241 557080-10
>> Amtsgericht Aachen: HRB 8151
>> Ust.-Id: DE 813021663
>>
>> Geschäftsführer: George Macdonald
>> ----------------------------------------------------------------------------------------------------
>>
>> ----------------------------------------------------------------------------------------------------
>>
>> take care of the environment - print only if necessary
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Quickfixj-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

--
Christoph John
Development & Support
Direct: +49 241 557080-28
Mailto:[hidden email]
       


http://www.macd.com <http://www.macd.com/>
----------------------------------------------------------------------------------------------------
       
----------------------------------------------------------------------------------------------------
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
Tel: +49 241 557080-0 | Fax: +49 241 557080-10
         Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663

Geschäftsführer: George Macdonald
----------------------------------------------------------------------------------------------------
       
----------------------------------------------------------------------------------------------------

take care of the environment - print only if necessary

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Fields 355 & 354

Grant Birchmeier
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



> changing the data dictionary to treat the specific data field as String field might be a more feasible solution.

It should be noted that this will malfunction if the encoded text results in an accidental SOH character.  The chances of that happening are not strong, but it's possible.

On Thu, May 4, 2017 at 8:07 AM, Christoph John <[hidden email]> wrote:
QuickFIX/J Documentation: <a href="http://www.quickfixj.org/documentation/ QuickFIX/J" rel="noreferrer" target="_blank">http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


I think there is a difference between "bad hack" and "adapting himself to different counterparties".
;) As we all know there are counterparties which are more FIX-compatible than others.
But maybe you are right: changing the data dictionary to treat the specific data field as String
field might be a more feasible solution.

Cheers,
Chris.


On 04/05/17 14:32, Philip Whitehouse wrote:
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> This feels like the wrong solution.
>
> If you aren't being sent a real data field, then you can just change
> your DataDictionary for that field.
>
> The point of the length is so that if the raw data field contains (for
> example) the field separator character, then it's not considered the end
> of the field.
>
> If you're using data fields as data fields, then the length is
> necessary. If you're actually using it as just a string of data then you
> should change your dictionary to expect a string and accept that if the
> data contains a field separator character it will break stuff.
>
> Making it so that the extractField behaviour changes is a bad hack.
>
> - Philip Whitehouse
>
>
> On 2017-05-04 08:26, Christoph John wrote:
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>
>>
>> Hi Colin,
>>
>> hmm, not sure. If you return null there, it looks to me like he is
>> then skipping the parsing of the
>> data field also.
>> But to be 100% sure it would be good if we had a unit test which tests
>> the new behaviour with a
>> missing length field and then with a data field that is OK and one
>> that is not OK (i.e. includes
>> SOH). I can also help you with the unit test if you like. Should be
>> somewhere in MessageTest IIRC.
>>
>> Cheers,
>> Chris.
>>
>>
>> On 04/05/17 00:54, Colin DuPlantis wrote:
>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>
>>>
>>> Chris,
>>>
>>> How you feel about a modification to return null for an EncodedText
>>> field if the length is missing:
>>>
>>> What are your thoughts on changing Message.extractField to return null
>>> instead of invalidate the message in this case?
>>>
>>> Current (1.6.x) code:
>>>
>>>            if (dataDictionary != null &&
>>> dataDictionary.isDataField(tag)) {
>>>                /* Assume length field is 1 less. */
>>>                int lengthField = tag - 1;
>>>                /* Special case for Signature which violates above
>>> assumption. */
>>>                if (tag == 89) {
>>>                    lengthField = 93;
>>>                }
>>>                int fieldLength;
>>>                try {
>>>                    fieldLength = fields.getInt(lengthField);
>>>                } catch (final FieldNotFound e) {
>>>                    throw new InvalidMessage("Tag " + e.field + " not
>>> found
>>> in " + messageData);
>>>                }
>>>
>>>                // since length is in bytes but data is a string, and it
>>> may also contain an SOH,
>>>                // we find the real field-ending SOH by checking the
>>> encoded bytes length
>>>                // (we avoid re-encoding when the chars length equals
>>> the
>>> bytes length, e.g. ASCII text,
>>>                // by assuming the chars length is always smaller than
>>> the
>>> encoded bytes length)
>>>                while (sohOffset - equalsOffset - 1 < fieldLength
>>>                        && messageData.substring(equalsOffset + 1,
>>> sohOffset).getBytes(CharsetSupport.getCharsetInstance()).length <
>>> fieldLength) {
>>>                    sohOffset = messageData.indexOf('\001', sohOffset +
>>> 1);
>>>                    if (sohOffset == -1) {
>>>                        throw new InvalidMessage("SOH not found at end
>>> of
>>> field: " + tag + " in " + messageData);
>>>                    }
>>>                }
>>>            }
>>>
>>> Proposed change:
>>>
>>>            if (dataDictionary != null &&
>>> dataDictionary.isDataField(tag)) {
>>>                /* Assume length field is 1 less. */
>>>                int lengthField = tag - 1;
>>>                /* Special case for Signature which violates above
>>> assumption. */
>>>                if (tag == 89) {
>>>                    lengthField = 93;
>>>                }
>>>                int fieldLength;
>>>                try {
>>>                    fieldLength = fields.getInt(lengthField);
>>>                } catch (final FieldNotFound e) {
>>> // proposed change begins
>>>                    return null;
>>> // proposed change ends
>>>                }
>>>
>>>                // since length is in bytes but data is a string, and it
>>> may also contain an SOH,
>>>                // we find the real field-ending SOH by checking the
>>> encoded bytes length
>>>                // (we avoid re-encoding when the chars length equals
>>> the
>>> bytes length, e.g. ASCII text,
>>>                // by assuming the chars length is always smaller than
>>> the
>>> encoded bytes length)
>>>                while (sohOffset - equalsOffset - 1 < fieldLength
>>>                        && messageData.substring(equalsOffset + 1,
>>> sohOffset).getBytes(CharsetSupport.getCharsetInstance()).length <
>>> fieldLength) {
>>>                    sohOffset = messageData.indexOf('\001', sohOffset +
>>> 1);
>>>                    if (sohOffset == -1) {
>>>                        throw new InvalidMessage("SOH not found at end
>>> of
>>> field: " + tag + " in " + messageData);
>>>                    }
>>>                }
>>>            }
>>>
>>>
>>>
>>> On 05/03/2017 12:26 PM, Christoph John wrote:
>>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>>
>>>>
>>>> When browsing e.g. Advertisement message in fiximate it clearly says
>>>> about Tag 354:
>>>> Must be set if EncodedText field is specified and must immediately
>>>> precede it.
>>>>
>>>> So IMHO the length field is clearly required.
>>>>
>>>> Cheers,
>>>> Chris.
>>>> ----- Original Message -----
>>>> From: Colin DuPlantis <[hidden email]>
>>>> To: [hidden email]
>>>> Sent: Wed, 03 May 2017 20:58:53 +0200 (CEST)
>>>> Subject: Re: [Quickfixj-users] Fields 355 & 354
>>>>
>>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Check out the vibrant tech community on one of the world's most
>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>> _______________________________________________
>>>> Quickfixj-users mailing list
>>>> [hidden email]
>>>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>> --
>> Christoph John
>> Development & Support
>> Direct: <a href="tel:%2B49%20241%20557080-28" value="+4924155708028">+49 241 557080-28
>> Mailto:[hidden email]
>>
>>
>>
>> http://www.macd.com <http://www.macd.com/>
>> ----------------------------------------------------------------------------------------------------
>>
>> ----------------------------------------------------------------------------------------------------
>> MACD GmbH
>> Oppenhoffallee 103
>> D-52066 Aachen
>> Tel: <a href="tel:%2B49%20241%20557080-0" value="+492415570800">+49 241 557080-0 | Fax: <a href="tel:%2B49%20241%20557080-10" value="+4924155708010">+49 241 557080-10
>>       Amtsgericht Aachen: HRB 8151
>> Ust.-Id: DE 813021663
>>
>> Geschäftsführer: George Macdonald
>> ----------------------------------------------------------------------------------------------------
>>
>> ----------------------------------------------------------------------------------------------------
>>
>> take care of the environment - print only if necessary
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Quickfixj-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

--
Christoph John
Development & Support
Direct: <a href="tel:%2B49%20241%20557080-28" value="+4924155708028">+49 241 557080-28
Mailto:[hidden email]



http://www.macd.com <http://www.macd.com/>
----------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------------
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
Tel: <a href="tel:%2B49%20241%20557080-0" value="+492415570800">+49 241 557080-0 | Fax: <a href="tel:%2B49%20241%20557080-10" value="+4924155708010">+49 241 557080-10
         Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663

Geschäftsführer: George Macdonald
----------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------------

take care of the environment - print only if necessary

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Fields 355 & 354

Øyvind Matheson Wergeland
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



Wouldn't that be true for any solution?


Best regards

Øyvind Matheson Wergeland
CTO


Mobile: (+47) 95 16 16 88
E-mail: [hidden email]

Oslo Market Solutions
PO Box 4, 0051 Oslo, Norway
Telephone: (+47) 40 00 23 13
www.oslomarketsolutions.no

On 04.05.2017 16.24, Grant Birchmeier wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




> changing the data dictionary to treat the specific data field as String field might be a more feasible solution.

It should be noted that this will malfunction if the encoded text results in an accidental SOH character.  The chances of that happening are not strong, but it's possible.

On Thu, May 4, 2017 at 8:07 AM, Christoph John <[hidden email]> wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J
Support: http://www.quickfixj.org/support/


I think there is a difference between "bad hack" and "adapting himself to different counterparties".
;) As we all know there are counterparties which are more FIX-compatible than others.
But maybe you are right: changing the data dictionary to treat the specific data field as String
field might be a more feasible solution.

Cheers,
Chris.


On 04/05/17 14:32, Philip Whitehouse wrote:
> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
> QuickFIX/J Support: http://www.quickfixj.org/support/
>
>
> This feels like the wrong solution.
>
> If you aren't being sent a real data field, then you can just change
> your DataDictionary for that field.
>
> The point of the length is so that if the raw data field contains (for
> example) the field separator character, then it's not considered the end
> of the field.
>
> If you're using data fields as data fields, then the length is
> necessary. If you're actually using it as just a string of data then you
> should change your dictionary to expect a string and accept that if the
> data contains a field separator character it will break stuff.
>
> Making it so that the extractField behaviour changes is a bad hack.
>
> - Philip Whitehouse
>
>
> On 2017-05-04 08:26, Christoph John wrote:
>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>
>>
>> Hi Colin,
>>
>> hmm, not sure. If you return null there, it looks to me like he is
>> then skipping the parsing of the
>> data field also.
>> But to be 100% sure it would be good if we had a unit test which tests
>> the new behaviour with a
>> missing length field and then with a data field that is OK and one
>> that is not OK (i.e. includes
>> SOH). I can also help you with the unit test if you like. Should be
>> somewhere in MessageTest IIRC.
>>
>> Cheers,
>> Chris.
>>
>>
>> On 04/05/17 00:54, Colin DuPlantis wrote:
>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>
>>>
>>> Chris,
>>>
>>> How you feel about a modification to return null for an EncodedText
>>> field if the length is missing:
>>>
>>> What are your thoughts on changing Message.extractField to return null
>>> instead of invalidate the message in this case?
>>>
>>> Current (1.6.x) code:
>>>
>>>            if (dataDictionary != null &&
>>> dataDictionary.isDataField(tag)) {
>>>                /* Assume length field is 1 less. */
>>>                int lengthField = tag - 1;
>>>                /* Special case for Signature which violates above
>>> assumption. */
>>>                if (tag == 89) {
>>>                    lengthField = 93;
>>>                }
>>>                int fieldLength;
>>>                try {
>>>                    fieldLength = fields.getInt(lengthField);
>>>                } catch (final FieldNotFound e) {
>>>                    throw new InvalidMessage("Tag " + e.field + " not
>>> found
>>> in " + messageData);
>>>                }
>>>
>>>                // since length is in bytes but data is a string, and it
>>> may also contain an SOH,
>>>                // we find the real field-ending SOH by checking the
>>> encoded bytes length
>>>                // (we avoid re-encoding when the chars length equals
>>> the
>>> bytes length, e.g. ASCII text,
>>>                // by assuming the chars length is always smaller than
>>> the
>>> encoded bytes length)
>>>                while (sohOffset - equalsOffset - 1 < fieldLength
>>>                        && messageData.substring(equalsOffset + 1,
>>> sohOffset).getBytes(CharsetSupport.getCharsetInstance()).length <
>>> fieldLength) {
>>>                    sohOffset = messageData.indexOf('\001', sohOffset +
>>> 1);
>>>                    if (sohOffset == -1) {
>>>                        throw new InvalidMessage("SOH not found at end
>>> of
>>> field: " + tag + " in " + messageData);
>>>                    }
>>>                }
>>>            }
>>>
>>> Proposed change:
>>>
>>>            if (dataDictionary != null &&
>>> dataDictionary.isDataField(tag)) {
>>>                /* Assume length field is 1 less. */
>>>                int lengthField = tag - 1;
>>>                /* Special case for Signature which violates above
>>> assumption. */
>>>                if (tag == 89) {
>>>                    lengthField = 93;
>>>                }
>>>                int fieldLength;
>>>                try {
>>>                    fieldLength = fields.getInt(lengthField);
>>>                } catch (final FieldNotFound e) {
>>> // proposed change begins
>>>                    return null;
>>> // proposed change ends
>>>                }
>>>
>>>                // since length is in bytes but data is a string, and it
>>> may also contain an SOH,
>>>                // we find the real field-ending SOH by checking the
>>> encoded bytes length
>>>                // (we avoid re-encoding when the chars length equals
>>> the
>>> bytes length, e.g. ASCII text,
>>>                // by assuming the chars length is always smaller than
>>> the
>>> encoded bytes length)
>>>                while (sohOffset - equalsOffset - 1 < fieldLength
>>>                        && messageData.substring(equalsOffset + 1,
>>> sohOffset).getBytes(CharsetSupport.getCharsetInstance()).length <
>>> fieldLength) {
>>>                    sohOffset = messageData.indexOf('\001', sohOffset +
>>> 1);
>>>                    if (sohOffset == -1) {
>>>                        throw new InvalidMessage("SOH not found at end
>>> of
>>> field: " + tag + " in " + messageData);
>>>                    }
>>>                }
>>>            }
>>>
>>>
>>>
>>> On 05/03/2017 12:26 PM, Christoph John wrote:
>>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>>
>>>>
>>>> When browsing e.g. Advertisement message in fiximate it clearly says
>>>> about Tag 354:
>>>> Must be set if EncodedText field is specified and must immediately
>>>> precede it.
>>>>
>>>> So IMHO the length field is clearly required.
>>>>
>>>> Cheers,
>>>> Chris.
>>>> ----- Original Message -----
>>>> From: Colin DuPlantis <[hidden email]>
>>>> To: [hidden email]
>>>> Sent: Wed, 03 May 2017 20:58:53 +0200 (CEST)
>>>> Subject: Re: [Quickfixj-users] Fields 355 & 354
>>>>
>>>> QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
>>>> QuickFIX/J Support: http://www.quickfixj.org/support/
>>>>
>>>>
>>>>
>>>>
>>>> ------------------------------------------------------------------------------
>>>> Check out the vibrant tech community on one of the world's most
>>>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>>>> _______________________________________________
>>>> Quickfixj-users mailing list
>>>> [hidden email]
>>>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
>> --
>> Christoph John
>> Development & Support
>> Direct: <a moz-do-not-send="true" href="tel:%2B49%20241%20557080-28" value="+4924155708028">+49 241 557080-28
>> Mailto:[hidden email]
>>
>>
>>
>> http://www.macd.com <http://www.macd.com/>
>> ----------------------------------------------------------------------------------------------------
>>
>> ----------------------------------------------------------------------------------------------------
>> MACD GmbH
>> Oppenhoffallee 103
>> D-52066 Aachen
>> Tel: <a moz-do-not-send="true" href="tel:%2B49%20241%20557080-0" value="+492415570800">+49 241 557080-0 | Fax: <a moz-do-not-send="true" href="tel:%2B49%20241%20557080-10" value="+4924155708010">+49 241 557080-10
>>       Amtsgericht Aachen: HRB 8151
>> Ust.-Id: DE 813021663
>>
>> Geschäftsführer: George Macdonald
>> ----------------------------------------------------------------------------------------------------
>>
>> ----------------------------------------------------------------------------------------------------
>>
>> take care of the environment - print only if necessary
>>
>> ------------------------------------------------------------------------------
>> Check out the vibrant tech community on one of the world's most
>> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
>> _______________________________________________
>> Quickfixj-users mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/quickfixj-users
> ------------------------------------------------------------------------------
> Check out the vibrant tech community on one of the world's most
> engaging tech sites, Slashdot.org! http://sdm.link/slashdot
> _______________________________________________
> Quickfixj-users mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/quickfixj-users

--
Christoph John
Development & Support
Direct: <a moz-do-not-send="true" href="tel:%2B49%20241%20557080-28" value="+4924155708028">+49 241 557080-28
Mailto:[hidden email]



http://www.macd.com <http://www.macd.com/>
----------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------------
MACD GmbH
Oppenhoffallee 103
D-52066 Aachen
Tel: <a moz-do-not-send="true" href="tel:%2B49%20241%20557080-0" value="+492415570800">+49 241 557080-0 | Fax: <a moz-do-not-send="true" href="tel:%2B49%20241%20557080-10" value="+4924155708010">+49 241 557080-10
         Amtsgericht Aachen: HRB 8151
Ust.-Id: DE 813021663

Geschäftsführer: George Macdonald
----------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------------

take care of the environment - print only if necessary

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users



--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Fields 355 & 354

Colin DuPlantis
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



Not if 354 (encoded len) is included, then you know for sure how long 355 is supposed to be regardless of the presence of a wild SOH.


On 05/04/2017 10:31 AM, Øyvind Matheson Wergeland wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




Wouldn't that be true for any solution?


Best regards

Øyvind Matheson Wergeland
CTO


Mobile: (+47) 95 16 16 88
E-mail: [hidden email]

Oslo Market Solutions
PO Box 4, 0051 Oslo, Norway
Telephone: (+47) 40 00 23 13
www.oslomarketsolutions.no

On 04.05.2017 16.24, Grant Birchmeier wrote:

<snip>

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Fields 355 & 354

Øyvind Matheson Wergeland
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



I see that I was not precise: any solution will fail if the length field is missing and the data field contains a wild SOH.

Hypothetically, you could use some look-ahead to check if the SOH is followed by a tag. But I would really not suggest to go down that road.

Best regards

Øyvind Matheson Wergeland
CTO


Mobile: (+47) 95 16 16 88
E-mail: [hidden email]

Oslo Market Solutions
PO Box 4, 0051 Oslo, Norway
Telephone: (+47) 40 00 23 13
www.oslomarketsolutions.no

On 04.05.2017 19.35, Colin DuPlantis wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




Not if 354 (encoded len) is included, then you know for sure how long 355 is supposed to be regardless of the presence of a wild SOH.


On 05/04/2017 10:31 AM, Øyvind Matheson Wergeland wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




Wouldn't that be true for any solution?


Best regards

Øyvind Matheson Wergeland
CTO


Mobile: (+47) 95 16 16 88
E-mail: [hidden email]

Oslo Market Solutions
PO Box 4, 0051 Oslo, Norway
Telephone: (+47) 40 00 23 13
www.oslomarketsolutions.no

On 04.05.2017 16.24, Grant Birchmeier wrote:

<snip>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Fields 355 & 354

Colin DuPlantis
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



I agree with you. I have come to the opinion that 355 without 354 is just broken.


On 05/04/2017 10:43 AM, Øyvind Matheson Wergeland wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




I see that I was not precise: any solution will fail if the length field is missing and the data field contains a wild SOH.

Hypothetically, you could use some look-ahead to check if the SOH is followed by a tag. But I would really not suggest to go down that road.

Best regards

Øyvind Matheson Wergeland
CTO


Mobile: (+47) 95 16 16 88
E-mail: [hidden email]

Oslo Market Solutions
PO Box 4, 0051 Oslo, Norway
Telephone: (+47) 40 00 23 13
www.oslomarketsolutions.no

On 04.05.2017 19.35, Colin DuPlantis wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




Not if 354 (encoded len) is included, then you know for sure how long 355 is supposed to be regardless of the presence of a wild SOH.


On 05/04/2017 10:31 AM, Øyvind Matheson Wergeland wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




Wouldn't that be true for any solution?


Best regards

Øyvind Matheson Wergeland
CTO


Mobile: (+47) 95 16 16 88
E-mail: [hidden email]

Oslo Market Solutions
PO Box 4, 0051 Oslo, Norway
Telephone: (+47) 40 00 23 13
www.oslomarketsolutions.no

On 04.05.2017 16.24, Grant Birchmeier wrote:

<snip>


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot


_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users

-- 
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
888.868.4884 +1.541.306.6556
http://www.marketcetera.org

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Fields 355 & 354

Grant Birchmeier
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



Yes, that's why I was surprised to find out that the FIX spec, when generically discussing encoded fields, implies that the length field is optional.  (see my earlier email on this chain)

It's only when you look at the specific fields within messages that you see the length fields being described as conditionally-required.  It's just odd.



On Thu, May 4, 2017 at 12:45 PM, Colin DuPlantis <[hidden email]> wrote:

I agree with you. I have come to the opinion that 355 without 354 is just broken.


On 05/04/2017 10:43 AM, Øyvind Matheson Wergeland wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




I see that I was not precise: any solution will fail if the length field is missing and the data field contains a wild SOH.

Hypothetically, you could use some look-ahead to check if the SOH is followed by a tag. But I would really not suggest to go down that road.

Best regards

Øyvind Matheson Wergeland
CTO


Mobile: (+47) 95 16 16 88
E-mail: [hidden email]

Oslo Market Solutions
PO Box 4, 0051 Oslo, Norway
Telephone: (+47) 40 00 23 13
www.oslomarketsolutions.no

On 04.05.2017 19.35, Colin DuPlantis wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
Reply | Threaded
Open this post in threaded view
|

Re: Fields 355 & 354

Colin DuPlantis
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/



Hoist on the same petard - I agree. I was initially thrown by the 'N' instead of 'C' and failed to look at each individual message.

In this case, the counter is just wrong. They're not using 355 for encoded text and won't admit what they're using it for.


On 05/04/2017 10:56 AM, Grant Birchmeier wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/




Yes, that's why I was surprised to find out that the FIX spec, when generically discussing encoded fields, implies that the length field is optional.  (see my earlier email on this chain)

It's only when you look at the specific fields within messages that you see the length fields being described as conditionally-required.  It's just odd.



On Thu, May 4, 2017 at 12:45 PM, Colin DuPlantis <[hidden email]> wrote:

I agree with you. I have come to the opinion that 355 without 354 is just broken.


On 05/04/2017 10:43 AM, Øyvind Matheson Wergeland wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/


I see that I was not precise: any solution will fail if the length field is missing and the data field contains a wild SOH.

Hypothetically, you could use some look-ahead to check if the SOH is followed by a tag. But I would really not suggest to go down that road.

Best regards Øyvind Matheson Wergeland CTO Mobile: (+47) 95 16 16 88 E-mail: [hidden email] Oslo Market Solutions PO Box 4, 0051 Oslo, Norway Telephone: (+47) 40 00 23 13 www.oslomarketsolutions.no
On 04.05.2017 19.35, Colin DuPlantis wrote:
QuickFIX/J Documentation: http://www.quickfixj.org/documentation/
QuickFIX/J Support: http://www.quickfixj.org/support/

--
Grant Birchmeier
Connamara Systems, LLC
Made-To-Measure Trading Solutions.
Exactly what you need. No more. No less.
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users
-- 
Colin DuPlantis
Chief Architect, Marketcetera
Download, Run, Trade
888.868.4884 +1.541.306.6556
http://www.marketcetera.org

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Quickfixj-users mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/quickfixj-users