Repository navigation
Trailing numerals in SAS Date/Time/DateTime formats are lost upon read #233
Description
Activity
A bit of further exploration (thanks to @ofajardo at pyreadstat).
>>> import pyreadstat >>> df, meta = pyreadstat.read_sas7bdat("datetimes.sas7bdat") >>> meta.variable_display_width["FMT_TIME5"] 0 # I think we expect to see 5
In this case, all variables have a display width of 0, including all the formats in the table above. I tried a few other fields in the
pyreadstatmetadata, but nothing seemed to capture these trailing numerals.Hi, thank you for the extensive report. It appears that internally, these format strings are stored without the length information. Two possibilities are that this length is stored in a separate record, or that SAS softwware knows what the lengths should be (e.g. if TIME is always TIME5). The latter possibility seems to be contradicted by the existence of
DDMMYYN6andDDMMYYN8, which are both stored asDDMMYY. I'll need to investigate this further.- added a commit that references this issue
on Mar 19, 2021 Hi, please try the commit here:
It would help to have the same script save a 32-bit file as well as a big-endian file, but I realize that may be difficult to achieve.
Reacted by Doug KelkhoffThis looks great. I was able to replace the ReadStat source that is bundled with
havenand read in the format lengths.It would help to have the same script save a 32-bit file as well as a big-endian file, but I realize that may be difficult to achieve.
I can try to help here, but I'd need a bit of guidance on how to test it. What considerations would I need to account for to produce 32-bit and big-endian files? Is this a result of the SAS host's architecture when the sas7bdat file is saved? If so, I'm not sure I have access to alternative deployments, but I can ask around.
@dgkf I think that you'd need SAS running on a 32-bit architecture and a big-endian architecture to produce those files. So I'm willing to leave it alone for now, and simply address 32-bit or big-endian bugs as they are reported.
I'll try to comb through the project's collection of SAS7BDAT files to see if I can find such a file containing one of the date/time fields that you identified.
Note: I'm using the
havenpackage in R to read in the data. I tried to trace the formatting as far down into C as my limited C knowledge allows, but it's very possible the information loss is happening on the R side.SAS datetimes with a numeric qualifier (eg
TIME5.) are read in without the trailing numeral (TIME). Would it be possible to include the numeric part of the format string?Steps to Reproduce
Full set of formats which omit the numeric component
havenFormatDATE9DATEDDMMYY10DDMMYYDDMMYYB10DDMMYYBDDMMYYC10DDMMYYCDDMMYYD10DDMMYYDDDMMYYN6DDMMYYNDDMMYYN8DDMMYYNDDMMYYP10DDMMYYPDDMMYYS10DDMMYYSMMDDYY10MMDDYYMMDDYYB10MMDDYYBMMDDYYC10MMDDYYCMMDDYYD10MMDDYYDMMDDYYN6MMDDYYNMMDDYYN8MMDDYYNMMDDYYP10MMDDYYPMMDDYYS10MMDDYYSTIME5TIMEYYQRN7YYQRN