Repository navigation
Feature Request: Add configuration to allow unknown/custom XML fields #331
Description
We are currently integrating with a Prime Broker that uses custom XML tags not defined in the standard schema. Currently, the library rejects these messages due to unknown fields.
To support integrations with non-standard providers, I suggest introducing a configuration flag: validate_user_defined_fields (or ignore_unknown_fields).
With default as current strict validation; throw an error if an unknown tag is encountered.
Activity
davidsteiner commentedon Apr 16, 2026
@Vlad-Bukatin just to make sure I understand the problem, is it just that the primer broker uses an extension of standard FIX with tags that aren't defined in the spec?
If so, that's very common, we have the same use-case internally. The way we do it is by extending the XML and then passing the custom XML to the engine and codegen. Is this something you tried and it didn't work or you didn't like? If it's just unclear how this can be done, I can show you and perhaps we can add an official example.
Having said that, I'm not opposed to an opt-out for using data dictionaries to provide extra flexibility.
But maintaining the custom XML will have added benefits in the future, as I'm planning to implement complete type generation. So rather than just generating field definitions, you'd get a proper
NewOrderSinglestruct for example with all the fields and theOutboundMessagetrait implementation and parsing logic for inbound messages for free.Vlad-Bukatin commentedon Apr 16, 2026
We done same: we downloaded standard FIX 4.2 spec and added missing fields in there.
But our worries, is that we done it for fields we spot during testing. We didn't check the whole documentation of the prime broker fix spec, so that can backfire in production with error. As we don't really read those field, we don't really care about them.
I see what you mean, could it be, that if we need to read/send those custom fields, value - we have to update the custom xml spec - and then it would be included in generated code (i.e. NewOrderSingle), but if we don't - it is not validated and not included in generated code?
davidsteiner commentedon Apr 16, 2026
I see - that makes sense.
I'm not opposed to adding a config flag to bypass data dictionary-based validation. I can't remember how entangled the validation logic is in the general message parsing logic. It may not be so straightforward to just skip it. I can take a look sometime next week or if you have capacity to have a stab at it, by all means give it a go.
Yeah, I don't see any issue with this.
Also FYI, I'm adding an example for custom fields in #337 just so that feature is better documented.
Vlad-Bukatin commentedon Apr 17, 2026
I hope I will have some time next week, so will try to take a look 👍
davidsteiner commentedon Apr 21, 2026
I haven't had much time to look at the actual code, but I wanted to flag a few thoughts in case you start looking at this issue.
The data dictionary is quite essential to the correct parsing of a FIX message. If a message doesn't have repeating groups, the parsing is straightforward - every tag should only occur once and it's essentially a simple key-value pair mapping. But in presence of repeating groups, the data dictionary is what gives it structure. It defines the
NUMINGROUPtag (which contains the group count), the delimiter tag (the first tag in a repeating group) and the order of tags in the group.This information is used by the parser to know when a new repeating group starts and when the last one ends so we resume parsing further tags as part of the enclosing message (or group).
This doesn't have to stop us from introducing a flag to allow unknown user-defined tags. I just wanted to be clear about the price - we'll have to introduce some uncertainty into the parsing logic. For example if we're currently parsing a group and encounter an unknown tag, we can choose to:
But neither can guarantee that the message will be parsed as intended by the counterparty.
Vlad-Bukatin commentedon Apr 22, 2026
Hello, sorry I haven't had a time to investigate.
Thanks for flagging this. I see the complexity you're highlighting.
I'm inclined to agree that treating unknown tags as part of the current group is the most logical default. If a counterparty requires strict structural integrity for that field, it should be in the spec.
However the one thing I want to mention that I’ve been advised that setting ValidateUserDefinedFields=N is the standard industry workaround.
So I found this setting in QuickFix - https://github.com/quickfix/quickfix/blob/master/doc/html/configuration.html
And before even going to implement I would like to investigate first, how it works there.
I will let you know here if(and when) I start working on it. (But now I doubt it would be this week)
davidsteiner commentedon Apr 22, 2026
Yes, I'm aware of the
ValidateUserDefinedFieldsflag in QuickFIX. I feel like calling it industry standard is a stretch, but as the most widespread open-source implementation, QuickFIX variants have definitely shaped expectations in the industry.So I would agree that taking inspiration from QuickFIX is the right way to go, even if it isn't strictly following the spec - I find it more important that the behaviour and configuration options are familiar to people than strict adherence to the protocol.
I'm less familiar with the original C++ implementation but from memory, QuickFIX/J (the Java implementation) does something similar to what I wrote above as option 3. In your specific case, because you don't even care about the values of unknown user defined fields, you just want it to not error out, I'm fairly confident that heuristic would work in the overwhelming majority of scenarios.
Keen to hear what you find in the C++ implementation.
Vlad-Bukatin commentedon May 6, 2026
Just checked the C++ version.
It is simply second option: Treat it as a tag that ends the group.
The reasoning behind that - parsing first and then validation.
In case of java - they have logic exactly for this case inside parsing mechanism.
What approach you would like to follow in Rust lib?
More specifically what is your opinion on adding a small bit of validation logic inside parsing?
Hard choice really, but I guess I would favour clear segregation in this case 🤔
But option 3 looks smarter and more suitable for my specific case 😄
davidsteiner commentedon May 8, 2026
Right, but depending on how you look at the definitions of parsing and validation, the decision whether you end a group or not could well be a parsing one. I'd prioritise pragmatism over purism in this case. The QuickFIX/J implementation and your case show that there is demand for the smarter logic, so my vote is to go with option 3.
As an aside, the purist in me doesn't even understand the need for supporting unknown tags - the two parties should really know and agree on the protocol and any extensions to it upfront. But I've accepted that this is a useful feature to some.