Repository navigation
On renderers that already shape (Pillow + Raqm), reshape() + get_display() corrupts the text #102
Description
Not a bug in the reshaping itself — it is correct for its intended target. This is about the
usage the README and effectively every tutorial recommend, which silently produces malformed
Arabic on any renderer that already does complex-text layout. Pillow has linked Raqm/HarfBuzz
by default for some time now, so this path is common and getting more so.
What happens
reshape() maps base letters to Unicode presentation forms (U+FE70–U+FEFF) and get_display()
reorders to visual order. HarfBuzz then applies its own shaping and the bidi algorithm to that
already-shaped, already-reordered string. The work is done twice.
from PIL import Image, ImageDraw, ImageFont, features
import arabic_reshaper
from bidi.algorithm import get_display
print(features.check("raqm")) # True
TEXT = "مرحبا بكم" # "welcome"
FONT = "/System/Library/Fonts/SFArabic.ttf"
def render(s, path):
img = Image.new("L", (700, 160), 255)
ImageDraw.Draw(img).text((30, 40), s, font=ImageFont.truetype(FONT, 64), fill=0)
img.save(path)
render(TEXT, "plain.png") # correct
render(get_display(arabic_reshaper.reshape(TEXT)), "pre.png") # corrupted| rendered | |
|---|---|
| no preprocessing | مرحبا بكم |
reshape() + get_display() |
مكب ابحرم |
The intermediate string is 8 of 9 characters in presentation forms, in visual order:
U+FEE2 U+FEDC U+FE91 U+0020 U+FE8E U+FE92 ...
Letters come out disconnected and the word order is reversed.
Scope
5 strings × 3 fonts (SF Arabic, Geeza Pro, IBM Plex Sans Arabic) = 15 renders per path, scored by
shape-invariant IoU against a reference and graded in three bands rather than pass/fail:
| path | no preprocessing | with reshape() + get_display() |
|---|---|---|
| Pillow with Raqm | 15/15 identical | 0/15 identical — 14 broken, 1 recognisable |
| Pillow without Raqm | 0/15 — 15 broken | 0/15 identical — 9 recognisable, 6 still broken |
Two things follow, and the second surprised me:
- With Raqm, the preprocessing takes a path that is 100% correct and breaks all of it.
- Without Raqm, the preprocessing is a partial rescue, not a fix — it moves 9 of 15 from
broken to merely recognisable and leaves 6 broken. A pass/fail score would have reported this
as "works", which is how the recipe keeps its reputation.
So the right question is which renderer you have, not whether Arabic "needs" reshaping.
Detection is one line:
from PIL import features
needs_reshaping = not features.check("raqm")Why it goes unnoticed
The output still looks like Arabic to anyone who does not read it, so it passes review. It matters
especially for synthetic training data: text rendered this way looks plausible at a glance, and
a model trained on it learns malformed letterforms.
Suggested fix
A note in the README that the recipe applies to renderers without complex-text shaping, with the
features.check("raqm") test. Happy to open a PR against the README if that would help.
Data
Full matrix, per-string results and the rendered images, CC BY 4.0:
https://huggingface.co/datasets/syamjithnk/arshape
Environment: Pillow 12.2.0, arabic-reshaper 3.0.0, python-bidi, macOS, Python 3.
Activity
Syamjith-NK commentedon Aug 21, 2026
Correcting myself on one line of the above, before someone else has to.
I wrote that "Pillow has linked Raqm/HarfBuzz by default for some time now, so this path is common
and getting more so." The first half is misleading and the second half is a claim I have no data
for. Both cut in the direction convenient to my argument, which is exactly when to re-check.
The wheels bundle a statically linked libraqm, but per Pillow's 8.2.0 release notes it "links
against FriBiDi at runtime". FriBiDi is not in the wheel. So the same wheel goes both ways
depending on the host:
```
$ docker run --rm python:3.13-slim sh -c 'pip install -q pillow &&
python -c "import PIL,PIL.features as f; print(PIL.version, f.check("raqm"))"'
12.3.0 False
same container, same already-installed wheel, no pip, no reinstall:
$ apt-get install -y libfribidi0
12.3.0 True
```
On macOS the wheel ships libharfbuzz and libfreetype and no fribidi; my own machine only answers
True because Homebrew installed libfribidi as a dependency of libass and mpv. Slim containers,
minimal CI images, Alpine builds and frozen app bundles will mostly answer False.
What that changes: this library is needed more often than my issue implied, not less. The
problem I reported is real and unchanged — where Raqm is active, the documented usage corrupts
the text — but the fix I suggested should not read as "you probably do not need this package". It
should read as "the reader cannot currently tell which case they are in, and the README is the
right place to tell them". I have rewritten the README patch on that basis and will open it as a
PR shortly.
Two smaller corrections in the same spirit, both from checking rather than assuming:
layout_engine=ImageFont.Layout.RAQMdocuments intent but does not guarantee shaping — withoutRaqm, Pillow falls back to basic layout with a warning and carries on. I confirmed by hashing
the renders that
Layout.BASICon a Raqm build is byte-identical to the default on a buildwithout Raqm, and that the default engine tracks
features.check("raqm")exactly. So the checkis a valid predictor of what will actually happen to a string, which is the thing I was
implicitly relying on.
While hashing those renders I found that
reshape()deletes harakat by default —reshape("مرحبا بكم")andreshape("مَرْحَبًا بِكُمْ")return the same string, 16 charactersin and 9 out. It is documented (
delete_harakat) and one keyword away from not happening, butthe module-level
reshape()that every tutorial calls does it silently, and vocalised text isusually vocalised for a reason. Happy to open that separately if it is worth its own issue
rather than a README line.
Reproduction for all of the above, CC BY 4.0:
https://huggingface.co/datasets/syamjithnk/arshape — see
RAQM_CONDITION.mdandverify_no_raqm.py.Syamjith-NK commentedon Aug 23, 2026
Small follow-up, in case it is useful to anyone landing here from the same problem.
The issue above is about
reshape()corrupting text on renderers that already shape. Aneighbouring failure kept coming up in the same pipelines — Arabic-Indic digits (
٢٠٢٦) notsurviving text-to-speech at all — so I packaged the front-end for it:
https://github.com/Syamjith-NK/arabic-tts-frontend
Zero dependencies, MIT. It converts numerals, dates, currency, times and percentages into spoken
Arabic before the text reaches a TTS engine. Measured on a 45-utterance set through Fish Audio:
Arabic-Indic digits go from 0/15 to 11/15; the scorer ships with it so the numbers can be
re-derived rather than trusted.
Different layer from this package and not a dependency of it — reshaping is a rendering concern,
this is a speech one. Mentioning it only because the two problems tend to arrive together, and
both are silent: the output looks or sounds fine and the content is simply wrong.