fix(knowledge): chunk CSV/JSON file text instead of the content dict repr

[Lesereingrape]

Related issue

Fixes #7611

Summary

CSVKnowledgeSource.add() / aadd() and JSONKnowledgeSource.add() / aadd() chunked str(self.content) — the Python repr of the whole dict[Path, str] — instead of each file's text, so every chunk handed to the embedder carried PosixPath('...') wrappers, quotes and \\n literals, and a multi-file source had all of its files merged into one string.

Measured on 3831e8b (absolute paths shortened here):

CSVKnowledgeSource(file_paths=[d / "data.csv"]).chunks    # d/data.csv = "Name,Age\nBrandon,30\nAlice,25\n"
# ["{WindowsPath('.../data.csv'): 'Name Age\\nBrandon 30\\nAlice 25\\n'}"]
JSONKnowledgeSource(file_paths=[d / "data.json"]).chunks  # {"name": "Brandon", "age": 30}
# ["{WindowsPath('.../data.json'): 'name: Brandon\\nage: 30\\n'}"]
CSVKnowledgeSource(file_paths=[a, b]).chunks              # a="Name\nBrandon\n", b="City\nChicago\n"
# ["{WindowsPath('.../multi_a.csv'): 'Name\\nBrandon\\n', WindowsPath('.../multi_b.csv'): 'City\\nChicago\\n'}"]

After the change:

# ['Name Age\nBrandon 30\nAlice 25\n']
# ['name: Brandon\nage: 30\n']
# ['Name\nBrandon\n', 'City\nChicago\n']

BaseFileKnowledgeSource declares content: dict[Path, str] = Field(init=False, default_factory=dict) and its model_post_init always assigns self.content = self.load_content(), so for these two sources the isinstance(self.content, dict) branch is not a corner case — it is the only branch that ever runs (content cannot be passed to the constructor). That makes the else arm dead and the repr the permanent behavior.

The fix drops the repr and iterates the mapping, which is exactly what the three sibling sources already do — text_file_knowledge_source.py:26/:33, pdf_knowledge_source.py:46/:53, excel_knowledge_source.py:151/:165 — so CSV/JSON now match the per-file chunking the rest of the package has. Same four-line shape in both add() and aadd() of each class; no change to any signature, field, or stored format.

Verification

• Tests added or updated for the changed behavior

• Relevant tests and quality checks pass locally

• Four tests added to lib/crewai/tests/knowledge/test_knowledge.py: CSV text, JSON text (_json_to_text output reaches the chunks), one chunk group per file for a two-file source, and the aadd() path asserting storage.asave(chunks). All four fail on 3831e8b with the repr shown above and pass with the fix: 4 failed, 18 deselected on the base → 2 failed, 20 passed on this branch.

• uv run pytest lib/crewai/tests/knowledge -q → 56 passed, 2 failed. The two failures are test_docling_source and test_multiple_docling_sources, which need the optional docling extra this environment does not have; they fail identically on 3831e8b with this diff reverted (CI installs --all-extras, where they pass).

• Blast radius: grep -rn "CSVKnowledgeSource\|JSONKnowledgeSource" lib/ returns only knowledge.py, source_helper.py (both just register the classes by extension) and this test file, and no existing test called add()/aadd() on either source — which is how the corrupted chunks stayed invisible.

• uv run ruff check and uv run ruff format --check → clean on both changed sources. uv run mypy → no error in either changed file (the 10 reported errors are the pre-existing no-any-unimported findings in crew_docling_source.py, from that same missing optional extra).

Additional context

Retrieval quality is the reason to take this: the corrupted string is what gets embedded, so path noise and escaped newlines were steering similarity search for every CSV/JSON source, and absolute file paths leaked into the chunks that end up in the model's context. For files longer than chunk_size the window also slid over repr offsets, so chunk boundaries fell mid-wording of the repr rather than in the text.

JSONKnowledgeSource._json_to_text() is the clearest tell — it builds readable key: value lines with indentation, and the old add() threw that formatting away one line later.

Follow-up this PR deliberately does not attempt: CrewDoclingSource and ExcelKnowledgeSource do their own chunking, and BaseFileKnowledgeSource could hoist the per-file chunk loop so a new source cannot forget it again. Both are refactors rather than a bug fix.

Authored with an AI coding assistant. .github/CONTRIBUTING.md requires the llm-generated label for agent-authored contributions and external contributors cannot apply it here — maintainers, please add it.

---

Note

Medium Risk
Changes what gets embedded for CSV/JSON knowledge sources, so existing vector indexes built before re-ingest may still hold bad chunks until sources are re-added; behavior is a correctness fix with limited blast radius to those two source types.

Overview
Fixes a bug where CSV and JSON knowledge sources embedded str(self.content) (the dict repr with path keys and escaped newlines) instead of each file’s parsed text.

add() and aadd() on both sources now iterate self.content.values(), chunk each file’s text separately, and extend chunks—matching PDF and text file sources. Multi-file sources produce distinct chunk groups per file rather than one merged repr string.

Four unit tests assert chunk contents for CSV/JSON (including async aadd() and two-file CSV).

^{Reviewed by Cursor Bugbot for commit e50d40f. Bugbot is set up for automated code reviews on this repo. Configure here.}

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Advanced

Run ID: 1944d838-14e8-42ea-b14a-0dd3ff63d6c9

---

📥 Commits

Reviewing files that changed from the base of the PR and between e232c46 and 6db13d1.

---

📒 Files selected for processing (2)

• lib/crewai/src/crewai/knowledge/source/csv_knowledge_source.py
• lib/crewai/src/crewai/knowledge/source/json_knowledge_source.py

---

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 8 remain after this review.

---

---

📝 Walkthrough

Walkthrough

CSV and JSON knowledge sources now chunk each file's text separately instead of chunking the string representation of the full content dictionary. Tests cover file text, multiple files, and synchronous and asynchronous storage.

Changes

Knowledge Source Chunking

Layer / File(s)	Summary
Per-file chunking lib/crewai/src/crewai/knowledge/source/csv_knowledge_source.py, lib/crewai/src/crewai/knowledge/source/json_knowledge_source.py	add() and aadd() iterate over content values and append chunks for each file. Existing save calls remain unchanged.
Chunking validation lib/crewai/tests/knowledge/test_knowledge.py	Tests verify converted file text, separate chunks for multiple CSV files, and synchronous and asynchronous storage calls.

---

Priority: ➖ Normal

Severity of issue fixed: Medium

Merge Risk

Merge Risk: ⚪ Minimal · up to 6db13

The per-file chunking change is mergeable after normal checks; the reported parked workflows still need approval to run.

Security Architecture Review

Security architecture risk: 🔵 Low · up to 96974

CSV and JSON files will be indexed as file text rather than as representations containing file paths. No new file-access route or security-control bypass was established. Existing indexed content may still need cleanup to reflect the new format.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

• inferred — The affected downstream scope is knowledge stored from selected CSV or JSON files and the agent prompts that retrieve it. The repository evidence does not establish who can choose or modify those files in a deployment.

Trust Boundaries and Controls

• observed — File-path processing and validation still precede content loading. The changed methods consume that loaded mapping rather than opening additional paths or bypassing the existing ingestion sequence.

Hardening Proposals

• proposed — If deployment expectations require only file-text chunks to remain searchable, provide an explicit collection cleanup or reindex procedure when adopting the new format.

---

---

---

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: correcting CSV and JSON chunking to use file text instead of the content dictionary representation.
Description check	✅ Passed	The description includes all required sections, links issue #7611, explains the change, documents verification results and known optional-dependency failures, and provides additional context.
Linked Issues check	✅ Passed	The PR satisfies the coding requirements in #7611. CSVKnowledgeSource and JSONKnowledgeSource now chunk each file text independently in both add() and aadd(). The tests cover readable CSV and …
Out of Scope Changes check	✅ Passed	The changes stay within #7611. The PR modifies CSV and JSON chunking and adds focused tests for the required behavior. No unrelated change is identified.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 12 functions across 3 files.

---

---

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

---

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

---

_{Comment @coderabbitai help to get the list of available commands.}

[Lesereingrape]

Two contributor-side items I cannot close myself — both need a maintainer with write access.

1. CI has not been allowed to run on this PR. On the current head, seven workflow runs are
parked in the action_required state (Run Tests, Lint, Type Checks, CodeQL, Vuln Scan,
plus the PR title/size jobs) — that is how this repo gates pull requests opened from a fork, and it
is why the only check visible here is require-issue. Nothing is failing; the required checks have
simply not been permitted to start. Could someone approve the pending runs?

In the meantime, the equivalent evidence I can produce locally: the exact pytest and ruff commands
with their output are recorded in the PR body above, run in this repo's locked venv. One caveat so
nothing is oversold: in my venv the knowledge tests also show test_docling_source and
test_multiple_docling_sources failing, and those two reproduce identically on a clean main
checkout — they need the optional docling extra, which is not installed here, so they are
unrelated to this diff.

2. The llm-generated label. .github/CONTRIBUTING.md:5-7 requires it on any PR authored by an
AI agent, and this PR is one — the body already says so. As the author of a fork PR I have no
permission to add labels to this repository (the API returns 403 for me), so this is the one
requirement of that policy I cannot satisfy myself.

Three sibling PRs from the same account were prepared the same way and need both items: #7610, #7612,
#7615, #7617. Each links its own open issue (#7609, #7611, #7614, #7616), each carries RED -> GREEN
measurements in its body, and each touches only one function plus its test file.

One deliberate non-request: I have not pushed a merge of main into this branch. The change is small
and self-contained, and since merges here are squashes a refreshed branch would only replace the
pending runs with a new set that still needs approving.