Skip to content

TableDiff: Unicode characters are silently dropped from diff #136

Description

@podarcis

Problem description

The TableDiff first converts some of the string's characters into named entities using htmlentities(). The remaining characters are tried to be converted using iconv() (might behave differently depending on underlying system) from UTF-8 into ISO-8859-1.

In 0.1.16 this failed with Notice: iconv(): Detected an illegal character in input string if characters remained that have no equivalent in ISO-8859-1 and neither modifiers //TRANSLIT or //IGNORE have been used.

In 0.1.17 there was a change (see #134 and its corresponding PR) that added the iconv modifier //IGNORE. While the conversion now succeeds, it silently drops any characters that have no representation in ISO-8859-1.

In our application there is an editorial workflow that needs to rely on the diff output being complete, so silently dropping characters isn't an viable option.

Steps to reproduce

$old = '<table><tr><td>React with :-) and "joy".</td></tr></table>';
$new = '<table><tr><td>React with 🙂 and "joy".</td></tr></table>';

$htmlDiff = new HtmlDiff($old, $new);
echo $htmlDiff->build();

In 0.1.16 it fails (or might fail), in 0.1.17 the Unicode character 🙂 is silently dropped.

Possible solution

PR #137

Activity

  1. jschroed91 commented on Sep 18, 2025

    @jschroed91
    Member

    Curious how performance is affected by this change - the first priority for this library is accuracy of the diff, and the fact that we currently dropping characters silently is not acceptable, so even if this is slower it does not matter since it will be more accurate - but I wonder if it also improves performance here or makes it worse. Was going to look at how the test suite runs before/after this change

  2. podarcis commented on Sep 19, 2025

    @podarcis
    Author

    @jschroed91 According to a quick benchmark the alternative solution seems to be 45% faster.

  3. added theissue type on Dec 27, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions