the code mentions compatibility with libedit, however the following happens in practice (e.g. while using irb from ruby 2.5.0):
- libedit initializes internal copy of rl_outstream
FILE* pointer during rl_initialize()
e = el_init_internal(rl_readline_name, rl_instream, rl_outstream, ...). this saves the current rl_outstream in el->el_outfile.
from that point on, libedit only operates on el_outfile in its EditLine* context.
- in this module, clear_rl_outstream() is called from readline_s_set_output() which fclose()s rl_outstream. then rl_outstream gets a fresh
FILE* assigned - however this does not affect libedit which continues using its el_outfile struct member.
- next attempt to write anything to the output file via readline API causes a segfault - at least with musl libc, which is stricter in regards to UB.
probably this was never caught because the developers used glibc where the fclose() FILE structure happened to stay in RAM at the same address. or maybe the new FILE was allocated at the exact same address. it's 100% reproducible using libedit (latest release) and musl libc, and likely when using address sanitizer even on glibc.
it's likely that rl_instream handling has the same bug, but i only investigated the crash i actually got.
arguably it's a bug of libedit to supply rl_outstream but then use it only for initialization - but it appears nobody else was closing the FILE behind libedit's back so far, else it would be fixed in latest libedit version.
(background: i was about to update ruby from 2.5.0 to a newer version in my distro sabotage linux and noticed irb crashing, so i wanted to fix it before switching to a more modern version, so someone who needs to use the old version for whatever reason can go back in commit history and produce a 100% working version).
the code mentions compatibility with libedit, however the following happens in practice (e.g. while using irb from ruby 2.5.0):
FILE*pointer during rl_initialize()e = el_init_internal(rl_readline_name, rl_instream, rl_outstream, ...). this saves the current rl_outstream inel->el_outfile.from that point on, libedit only operates on el_outfile in its
EditLine*context.FILE*assigned - however this does not affect libedit which continues using its el_outfile struct member.probably this was never caught because the developers used glibc where the fclose() FILE structure happened to stay in RAM at the same address. or maybe the new FILE was allocated at the exact same address. it's 100% reproducible using libedit (latest release) and musl libc, and likely when using address sanitizer even on glibc.
it's likely that rl_instream handling has the same bug, but i only investigated the crash i actually got.
arguably it's a bug of libedit to supply rl_outstream but then use it only for initialization - but it appears nobody else was closing the FILE behind libedit's back so far, else it would be fixed in latest libedit version.
(background: i was about to update ruby from 2.5.0 to a newer version in my distro sabotage linux and noticed irb crashing, so i wanted to fix it before switching to a more modern version, so someone who needs to use the old version for whatever reason can go back in commit history and produce a 100% working version).