Repository navigation
Reading a literal (non-escaped) emoji doesn't work when reading from a file with mode 'rb' #115
Description
For a file containing the following symbol:
'🏳0🌈️'
the following behavior is observed:
>>>
>>> with open('<filename>', 'r') as f:
... print(ion.load(f, single_value=False))
...
[IonPySymbol(text='🏳0🌈️', sid=None, location=None)] # correct
>>> with open('<filename>', 'rb') as f:
... print(ion.load(f, single_value=False))
...
[IonPySymbol(text='ð\x9f\x8f³0ð\x9f\x8c\x88ï¸\x8f', sid=None, location=None)] # incorrect
Note that ion-python tests appear to be cheating: https://github.com/amzn/ion-python/blob/master/tests/test_vectors.py#L87
I believe the same symbol should be constructed whether the file is opened with mode 'r' or 'rb'.
Activity
abhinavs1920 commentedon Oct 17, 2026
Ran into this as well. The binary reader decodes symbol text byte by byte instead of as UTF-8, so non-ASCII symbols come out mangled in 'rb' mode while 'r' mode reads them fine. Making the binary path decode symbol text as UTF-8 makes both modes consistent. Happy to open a PR with a fix and regression tests if there is interest.