Skip to content

BUG: image and PDF converters silently draw characters their fonts lack as boxes #4242

Open

Description

Describe the bug

The converters that draw an objective as text lose every character outside a small Latin subset, without a warning, so non-English objectives are sent as unreadable images or PDFs while the attack is still scored against the real objective.

  • AddImageTextConverter, AddTextImageConverter and GridCompositeConverter default to font_name=None, which uses Pillow's built-in font (base_image_text_converter.py#L98-L105). That font has no glyphs for accented Latin letters, Cyrillic, Greek, CJK, Arabic, Hebrew or Devanagari, so each such character is drawn as the same box. Two different Russian or Japanese objectives of the same shape produce pixel-identical images.
  • PDFConverter only uses the standard Type1 fonts (pdf_converter.py#L252-L260), so Cyrillic, CJK and Arabic come out of the PDF as black squares, and there is no way to pass a Unicode font. Latin-1 text survives.

The image techniques in pyrit/setup/initializers/techniques/image.py and the AIRT image technique use these converters with the default font, so image-technique success rates on multilingual datasets measure prompts the target model could not read.

AsciiArtConverter had the same class of problem in #2942, fixed in #2962 by keeping the characters its font cannot render.

The open #3652 adds TextToImageConverter with the same default font, so whatever is decided here could apply there too.

Steps/Code to Reproduce

import asyncio
import hashlib

from PIL import Image
from pypdf import PdfReader

from pyrit.common.path import DATASETS_PATH
from pyrit.converter import AddImageTextConverter, PDFConverter
from pyrit.setup import IN_MEMORY, initialize_pyrit_async

PAIRS = [
    ("How to pick a lock", "How to bake a cake"),
    ("Как открыть замок", "Где достать книгу"),
    ("鍵の開け方", "窓の開け方"),
    ("¿Cómo abrir?", "¿Cúmo abrir?"),
]


async def main() -> None:
    await initialize_pyrit_async(memory_db_type=IN_MEMORY)
    image = AddImageTextConverter(img_to_add=DATASETS_PATH / "seed_datasets" / "local" / "examples" / "blank_canvas.png")
    for first, second in PAIRS:
        pixels = []
        for text in (first, second):
            path = (await image.convert_async(prompt=text)).output_text
            pixels.append(hashlib.sha256(Image.open(path).convert("RGB").tobytes()).hexdigest())
        print(f"image: {first!r} vs {second!r}: {'identical' if pixels[0] == pixels[1] else 'different'}")

    pdf = PDFConverter()
    for text in ("How to pick a lock", "Как открыть замок", "鍵の開け方"):
        path = (await pdf.convert_async(prompt=text)).output_text
        print(f"pdf: {text!r} -> {PdfReader(path).pages[0].extract_text().strip()!r}")


asyncio.run(main())

Expected Results

Different objectives give different images, and the PDF contains the objective's text, or the converter warns or raises when its font has no glyph for some of the characters.

Actual Results

image: 'How to pick a lock' vs 'How to bake a cake': different
image: 'Как открыть замок' vs 'Где достать книгу': identical
image: '鍵の開け方' vs '窓の開け方': identical
image: '¿Cómo abrir?' vs '¿Cúmo abrir?': identical
pdf: 'How to pick a lock' -> 'How to pick a lock'
pdf: 'Как открыть замок' -> '■■■ ■■■■■■■ ■■■■■'
pdf: '鍵の開け方' -> '■■■■■'

Passing a Unicode TrueType font as font_name fixes the image converters for scripts that font covers. Possible fixes: warn (or raise) when the objective has characters the chosen font cannot draw, document that non-Latin objectives need font_name, and let PDFConverter register a TrueType font. I would be glad to send a PR for whichever you prefer.

Versions

  • OS: Linux
  • Python version: 3.12, Pillow 13.0.0
  • PyRIT version: main at ce434d7

Activity

  1. its-amann commented on Nov 16, 2026

    @its-amann
    Contributor

    Roman Lutz (@romanlutz) I'd like to take this one. plan is a font path option for PDFConverter plus a warning when the default image font can't draw the text, and I'll keep it in line with #3652

  2. siddhant-rajhans commented on Nov 16, 2026

    @siddhant-rajhans
    Author

    Fine by me, go ahead. The repro above covers both the image converters and PDFConverter, so it should show the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions