OFX (v1.x) headers

[phillbaker]

Thanks for all of the work on this project.

Looking at the OFX output, it appears that the DATA:OFXSGML and ENCODING:UTF-8 headers are added. What do you think of adding other common OFX headers such as:

OFXHEADER:100
VERSION:102
CHARSET:1252
SECURITY:NONE
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:NONE

[reubano]

That's a good idea. Do you have a link to somewhere that documents these OFX headers as standard/common?

[phillbaker]

It appears the ofx.net site isn't responding, but the specification of the 1.02 version is available in archive.org in this zip file: https://web.archive.org/web/20170620010052/http://www.ofx.net:80/downloads/ofx1.0.2spec.zip, in the file Ofexfin1.doc.

That lists the following headers:

DATA:OFXSGML
ENCODING:
OFXHEADER:100
VERSION:102
SECURITY:
CHARSET:
COMPRESSION:
OLDFILEUID:
NEWFILEUID:

There's also an example in from the ofxparse library here: https://github.com/jseutter/ofxparse/blob/5d8647e61e732fb44982bd0848e9cc3bd9964b81/tests/fixtures/bank_small.ofx

[reubano]

The only potential issue I foresee is with the CHARSET header. It should normally be utf-8. But that could change depending on the arguments passed to the program. Feel free to submit a PR adding in these new headers, and which properly deals with CHARSET.

[phillbaker]

Sounds good - to clarify @reubano, there's a ENCODING header which is typically utf-8 or usascii, however, all the CHARSET examples I've seen have used 1252.

[reubano]

1252 implies windows-1252 though. Maybe the examples are wrong? Either that, or CHARSET doesn't mean what it seems?

[phillbaker]

It all around seems a bit of a mess 😀

# NOTE: Encoding in OFX, particularly in OFX 1.02,
# is kind of a mess.  The OFX 1.02 spec talks about "UNICODE"
# as a supported encoding, which the OFX 2.0 spec has
# back-rationalized to "UTF-8".  The "US-ASCII" encoding is
# given as "USASCII".  Yet the 1.02 spec acknowledges that
# not everyone speaks English nor uses UNICODE, so they let
# you throw any old encoding in there you'd like. 

(source)

What do you think about a pair of ENCODING:UNICODE and CHARSET:UTF-8? example

[reubano]

Of all those, it seems like wasabe has the most sane reasoning. I think what you suggest is good (taking into account the link above). There should also be some mapping (encoding -> charset) so that if a different output encoding is selected, the appropriate headers will automagically get set.

Edit: I also think you may have it backwards as UTF-8 is the relevant ENCODING.

[csingley]

Hey guys, I happened to stumble upon this project and this discussion. As it happens, I've spent way too much time combing through the OFX specs while working on ofxtools. I'd be glad to help if I can.

If I understand correctly, the scope of your work here is to generate OFX for importing into various financial management applications (e.g. GnuCash). There are a few good reasons to prefer generating OFXv2 (i.e. XML) rather than OFXv1 (i.e. SGML) - character encodings being one of them. If you want to generate OFXv1, you should really prefer to adhere to the OFXv1.6 spec. As Jerry notes in the ofxparse code linked above, OFXv1.0.2 has many warts that were fixed in subsequent versions.

You should always include OFX headers for OFXHEADER and VERSION, even if you're outputting OFXv1.0.2 for some reason. You are wise to set ENCODING: UTF-8, and really you want to output the full set of OFX headers (setting the rest of them to NONE) for an easy gain in parser compliance.

You can look at ofxtools.header for some logic.

Indeed, you might be able to make all this easier to develop & maintain by just using ofxtools to generate the OFX you want... an example of the API can be found in ofxtools.Client

I find it more pleasant to generate OFX like this:

sonrs_status = STATUS(code=0, severity='INFO')
sonrs = SONRS(status=sonrs_status, dtserver=datetime.today(), language='ENG')
signon = SIGNONMSGSRSV1(sonrs=sonrs)
bankmsgs = BANKMSGSRSV1()
ofx = OFX(signonmsgsrsv1=signon, bankmsgsrsv1=bankmsgs)
header = OFXHeader(version=220)
print(str(header) + str(ofx))

[csingley]

P.S. relevant to issues like #17 - ofxtools has fairly extensive support for all kinds of investment transactions - that's what I mostly use it for, personally. BUYDEBT, SELLSTOCK, EXERCISE, SPLIT, RETOFCAP, INVPOSLIST, SECINFO... it's all in there.