data-transform-cli

Python CLI for YAML ↔ JSON ↔ CSV ↔ NDJSON transforms. Built for shell pipelines and local scripts. Streaming-friendly where practical (stdin/stdout; full documents buffered for parse safety).

Requires Python 3.11+.

Install

pip install -e .
# or with test extras:
pip install -e ".[dev]"

After install, the data-transform console script is on your PATH.

Commands

Command	Description
`to-json`	Convert input to JSON
`to-yaml`	Convert input to YAML
`to-csv`	Convert input to CSV (array of objects or rows)
`to-ndjson`	Convert input to NDJSON (one JSON value per line)
`pretty`	Pretty-print as JSON
`minify`	Emit compact JSON
`select`	Extract dotted paths from parsed data
`pick`	Keep only listed top-level keys
`omit`	Drop listed top-level keys
`rename`	Rename top-level keys (`OLD NEW` pairs)
`flatten`	Flatten nested objects into joined keys
`unflatten`	Expand joined keys back into nested objects
`filter`	Keep array items matching KEY OP VALUE
`sort`	Sort array of objects by top-level KEY
`unique`	Deduplicate array (by KEY or whole item)
`head`	First N items of a list
`tail`	Last N items of a list
`group-by`	Group array of objects by top-level KEY
`count`	Count items, per KEY value, or per group
`stats`	count / sum / min / max / avg of a numeric FIELD
`join`	Join two arrays of objects on a top-level KEY (`--with` file)
`validate`	Check records for required paths, types, and unique keys (exit code 0 / 1 / 2)

Options (most commands)

[file] — input path; omit or use - for stdin
-o, --output <file> — write to a file instead of stdout
-f, --from <format> — force input format: json | yaml | csv | ndjson
-t, --to <format> — output format for the shaping commands (everything except to-*, pretty, and minify): json | yaml | csv | ndjson. Defaults to the --output file extension, else JSON. See Output formats
--minify — compact JSON output (to-json and the shaping commands); ignored when the output format is not JSON
--desc — reverse sort order (sort only)
--sep <text> — separator between nested keys (flatten and unflatten, default .)
-w, --with <file>, --how inner|left, --right-from <format> — right-hand file, join type, and right file format (join only)
-r, --require <path>, --type <path>=<type>, -u, --unique <path>, -s, --schema <file>, --coerce, --max-errors <n>, -q, --quiet — checks and report options (validate only). See Validate

Format is taken from --from, else the file extension (.ndjson / .jsonl → ndjson), else simple content heuristics.

Output formats

Shaping commands (select, pick, omit, rename, flatten, unflatten, filter, sort, unique, head, tail, group-by, count, stats, join) print JSON by default. Use -t / --to to write YAML, CSV, or NDJSON directly instead of piping into a to-* command.

The format comes from --to, else the --output file extension (.json, .yaml / .yml, .csv, .ndjson / .jsonl), else JSON. --to always wins, and an unknown extension stays JSON.
JSON is pretty-printed unless --minify is set. --minify does nothing for YAML, CSV, or NDJSON (NDJSON is always one compact value per line).
CSV needs a list result. A number, object, or group-by result is an error (exit code 1) that suggests --to json or --to yaml instead.
YAML output keeps key order and keeps YAML-only values such as dates, which JSON can't represent.

# Filter a CSV and write CSV back out
data-transform filter age ge 30 people.csv -t csv
# → name,age
#   Ada,36

# Format picked from the output file name (events.json is an array of objects)
data-transform flatten events.json -o events-flat.csv
data-transform pick id name users.json -o users.yaml

# Unknown extension: name the format with --to
data-transform sort id users.json -t ndjson -o users.out

# Per-team counts as YAML
data-transform count team people.json -t yaml
# → core: 1
#   web: 1

# Join JSON against CSV and emit CSV
data-transform join id --with teams.csv users.json -t csv -o report.csv

Usage examples

# YAML to JSON
data-transform to-json config.yaml

# JSON to YAML (pipe)
cat payload.json | data-transform to-yaml

# JSON array to CSV
data-transform to-csv records.json -o records.csv

# JSON array to NDJSON
data-transform to-ndjson records.json

# NDJSON to JSON array
data-transform to-json events.ndjson

# CSV to pretty JSON
data-transform pretty table.csv -f csv

# Minify JSON
data-transform minify bulky.json

# Force format when using stdin
printf "a: 1\n" | data-transform to-json -f yaml

Select, pick, omit

# Single dotted path (list index supported)
echo '{"user":{"name":"Ada"},"items":[{"id":1}]}' \
  | data-transform select user.name -f json
# → "Ada"

echo '{"items":[{"id":1},{"id":2}]}' \
  | data-transform select items.0.id -f json
# → 1

# Multiple paths → object map (missing → null)
echo '{"a":1}' | data-transform select a b -f json --minify
# → {"a":1,"b":null}

# Keep / drop top-level keys (maps over arrays of objects)
echo '[{"id":1,"name":"a","note":"x"},{"id":2,"name":"b","note":"y"}]' \
  | data-transform pick id name -f json --minify
# → [{"id":1,"name":"a"},{"id":2,"name":"b"}]

echo '{"id":1,"secret":"x","name":"a"}' \
  | data-transform omit secret -f json --minify
# → {"id":1,"name":"a"}

Rename and flatten

rename takes pairs of OLD NEW and maps over an object or each object in an array, the same way pick and omit do. Missing old keys are left alone. A later pair wins when the same old key is given twice. Renames are one pass over the original keys: a→b together with b→c leaves the value of a under b.

flatten collapses nested objects into top-level keys joined by --sep (default .). A list root is flattened item by item. Nested lists stay as values. Empty objects stay empty. A number, string, or other non-object root is returned unchanged.

# Rename top-level keys (several pairs)
echo '{"id":1,"name":"Ada","note":"x"}' \
  | data-transform rename name title note comment -f json --minify
# → {"id":1,"title":"Ada","comment":"x"}

# Same mapping over an array; missing keys are a no-op
echo '[{"name":"Ada"},{"id":2},1]' \
  | data-transform rename name title -f json --minify
# → [{"title":"Ada"},{"id":2},1]

# Flatten nested objects; lists stay as values
echo '{"user":{"name":"Ada","city":"London"},"tags":["a","b"]}' \
  | data-transform flatten -f json --minify
# → {"user.name":"Ada","user.city":"London","tags":["a","b"]}

# Custom separator
echo '{"user":{"name":"Ada"}}' \
  | data-transform flatten --sep _ -f json --minify
# → {"user_name":"Ada"}

Unflatten

unflatten is the inverse of flatten: it splits top-level keys on --sep (default .) and rebuilds nested objects. Like flatten, it maps over a list root and returns non-object roots and items unchanged. It does not rebuild lists, so a segment like 0 stays an object key. Keys are split literally, so an empty segment (a..b) becomes an empty key. If an object value and joined keys write to the same branch, they are merged. If a path runs into a non-object value, the later key wins. unflatten undoes flatten exactly as long as the original keys don't contain the separator.

echo '{"user.name":"Ada","user.addr.city":"London","tags":["a","b"]}' \
  | data-transform unflatten -f json --minify
# → {"user":{"name":"Ada","addr":{"city":"London"}},"tags":["a","b"]}

echo '[{"user_name":"Ada"},{"user_name":"Bob"}]' \
  | data-transform unflatten --sep _ -f json --minify
# → [{"user":{"name":"Ada"}},{"user":{"name":"Bob"}}]

# Round trip
data-transform flatten nested.json | data-transform unflatten

Filter, sort, unique

These commands require a list root (JSON/YAML/CSV/NDJSON array). Non-object items are dropped by filter. Missing keys sort last (first with --desc).

# Keep rows where age >= 30 (numeric when both sides look like numbers)
echo '[{"name":"Ada","age":36},{"name":"Bob","age":22}]' \
  | data-transform filter age ge 30 -f json --minify
# → [{"name":"Ada","age":36}]

# String contains / key exists (VALUE optional for exists)
echo '[{"role":"admin"},{"role":"user"}]' \
  | data-transform filter role contains adm -f json --minify
# → [{"role":"admin"}]

echo '[{"id":1,"note":"x"},{"id":2}]' \
  | data-transform filter note exists -f json --minify
# → [{"id":1,"note":"x"}]

# Sort by key (ascending / descending)
echo '[{"id":2},{"id":1},{"id":3}]' \
  | data-transform sort id -f json --minify
# → [{"id":1},{"id":2},{"id":3}]

echo '[{"id":2},{"id":1}]' \
  | data-transform sort id --desc -f json --minify
# → [{"id":2},{"id":1}]

# Deduplicate whole items, or by a key (first wins)
echo '[{"id":1},{"id":1},{"id":2}]' \
  | data-transform unique id -f json --minify
# → [{"id":1},{"id":2}]

echo '[{"a":1},{"a":1},{"b":2}]' \
  | data-transform unique -f json --minify
# → [{"a":1},{"b":2}]

Head and tail

head and tail require a list root, same as filter, sort, and unique. N is a non-negative integer. A count past the end returns the items that exist; 0 returns an empty list.

echo '[{"id":1},{"id":2},{"id":3}]' \
  | data-transform head 2 -f json --minify
# → [{"id":1},{"id":2}]

echo '[{"id":1},{"id":2},{"id":3}]' \
  | data-transform tail 1 -f json --minify
# → [{"id":3}]

echo '[1,2,3]' | data-transform head 0 -f json --minify
# → []

Group-by and count

group-by KEY requires a list root. It returns an object that maps each value of top-level KEY to the list of items that have it. Groups appear in the order they are first seen, and items keep their input order. Group labels are the value itself for strings; any other value becomes compact JSON text (1 → "1", true → "true", null → "null"). This means 1 and "1" end up in the same group, which suits CSV input where every value is a string. Non-object items and objects without KEY are dropped.

count counts records:

count on a list prints its length as a number.
count KEY on a list prints an object of per-value counts, using the same labels and dropping rules as group-by.
count on the output of group-by (an object whose values are all lists) prints the size of each group.

Any other root is an error.

echo '[{"team":"a","id":1},{"team":"b","id":2},{"team":"a","id":3},{"id":4}]' \
  | data-transform group-by team -f json --minify
# → {"a":[{"team":"a","id":1},{"team":"a","id":3}],"b":[{"team":"b","id":2}]}

echo '[{"id":1},{"id":2},{"id":3}]' | data-transform count -f json
# → 3

echo '[{"team":"a"},{"team":"b"},{"team":"a"}]' \
  | data-transform count team -f json --minify
# → {"a":2,"b":1}

# Count per group from group-by output
data-transform group-by team people.json | data-transform count --minify
# → {"a":2,"b":1}

# Works with NDJSON / CSV too
data-transform count status events.ndjson

Stats

stats FIELD summarises a numeric top-level field and prints count, sum, min, max, and avg. On a list root it prints one summary; on group-by output it prints one summary per group. Numeric strings count as numbers, so CSV input works directly. Booleans, missing keys, and non-numeric values are skipped. If nothing numeric is found the summary is count: 0, sum: 0, and null for the rest. Whole-number results print as integers.

echo '[{"ms":120},{"ms":80},{"ms":"100"},{"ms":"n/a"}]' \
  | data-transform stats ms -f json --minify
# → {"count":3,"sum":300,"min":80,"max":120,"avg":100}

# Per-group latency summary
data-transform group-by service requests.csv | data-transform stats ms

Join

join KEY --with RIGHT_FILE [file] joins two arrays of objects on a top-level KEY. The left side is [file] or stdin; the right side is always a file given with -w / --with. Each side's format is detected on its own, so you can join JSON against CSV, YAML, or NDJSON. Use -f to force the left format and --right-from to force the right one.

--how inner (default) keeps only rows that match. --how left also keeps left rows with no match, unchanged.
Output keeps left-row order. A left row that matches several right rows produces one output row per match, in right-file order.
Each output row starts with the left row's keys, then adds the right row's keys. If both have the same key (other than KEY), the right value wins. KEY appears once, with the left value.
Values match using the same labels as group-by, so 1 and "1" are equal — useful when one side is CSV. A missing or null KEY never matches.
Both roots must be lists, and every item must be an object; anything else is an error.

echo '[{"id":1,"name":"Ada"},{"id":2,"name":"Bob"},{"id":3,"name":"Cy"}]' > users.json
printf 'id,team\n1,core\n2,web\n2,ops\n' > teams.csv

data-transform join id --with teams.csv users.json --minify
# → [{"id":1,"name":"Ada","team":"core"},{"id":2,"name":"Bob","team":"web"},{"id":2,"name":"Bob","team":"ops"}]

data-transform join id -w teams.csv users.json --how left --minify
# → [... same three rows ..., {"id":3,"name":"Cy"}]

# Left side from stdin, then summarise per team
cat users.json | data-transform join id -w teams.csv -f json \
  | data-transform group-by team | data-transform count --minify
# → {"core":1,"web":1,"ops":1}

Validate

validate checks data before it moves further down a pipeline. It works on a list of objects (each item is a record) or on a single object. Paths are dotted, like select, so user.name and items.0.sku work.

-r, --require PATH — the path must be present in every record. A null value counts as present; add a type check to rule it out.
--type PATH=TYPE — the value must be one of the listed types: string, integer, number, boolean, object, array, null. Join several with |, for example email=string|null. Records without the path are skipped, so pair it with --require when the path is mandatory. Booleans are never numbers, and integer also satisfies number.
-u, --unique PATH — the value must not repeat across records. Values compare like group-by labels, so 1 and "1" count as the same. Records without the path are skipped.
-s, --schema FILE — a JSON or YAML file with any of required (list), types (path → type, or a list of types), and unique (list). Flags on the command line are added on top.
--coerce — let strings that look like integers, numbers, true / false, or an empty string satisfy integer, number, boolean, or null. Use it for CSV, where every cell is a string.
--max-errors N — list at most N errors. error_count is always the full count.
-q, --quiet — print nothing and rely on the exit code.

The report is {"valid", "records", "error_count", "errors"}, where each error has record (the list index, or null for an object root), path, and message. -t yaml writes the report as YAML; -t csv or -t ndjson writes just the errors, one row per error, which is handy as a CI artifact.

Exit codes: 0 when the data is valid, 1 when it is not, and 2 when the check could not run (unreadable input, a bad schema, an unknown type, or no checks given).

echo '[{"id":1,"email":"a@x.io"},{"id":"2"},{"id":1,"email":null}]' \
  | data-transform validate -r id -r email --type id=integer --type 'email=string|null' -u id -f json --minify
# → {"valid":false,"records":3,"error_count":3,"errors":[
#     {"record":1,"path":"email","message":"missing required value"},
#     {"record":1,"path":"id","message":"expected integer, got string"},
#     {"record":2,"path":"id","message":"duplicate value '1' (first seen in record 0)"}]}

# Data contract kept next to the pipeline
cat > users.schema.yaml <<'YAML'
required: [id, email]
types:
  id: integer
  email: [string, "null"]
unique: [id]
YAML
data-transform validate -s users.schema.yaml --coerce users.csv -q || echo "users.csv failed checks"

# Fail a CI step and keep an error table
data-transform validate -s users.schema.yaml --coerce users.csv -t csv -o validation-errors.csv

Library use

from data_transform import (
    convert,
    parse,
    serialize,
    get_path,
    pick_keys,
    omit_keys,
    rename_keys,
    flatten_object,
    unflatten_object,
    filter_rows,
    sort_rows,
    unique_rows,
    head_rows,
    tail_rows,
    group_rows,
    count_rows,
    stats_rows,
    join_rows,
    validate_rows,
)

yaml_text = convert('{"a":1}', "json", "yaml")
data = parse("id,name\n1,x\n", "csv")
json_text = serialize(data, "json", pretty=True)
name = get_path({"user": {"name": "Ada"}}, "user.name")
slim = pick_keys({"id": 1, "name": "a", "note": "x"}, ["id", "name"])
renamed = rename_keys({"id": 1, "name": "a"}, {"name": "title"})
flat = flatten_object({"user": {"name": "Ada"}, "tags": ["x"]})
nested = unflatten_object({"user.name": "Ada"})
adults = filter_rows([{"age": 36}, {"age": 22}], "age", "ge", "30")
ordered = sort_rows([{"id": 2}, {"id": 1}], "id")
deduped = unique_rows([{"id": 1}, {"id": 1}], "id")
first = head_rows([{"id": 1}, {"id": 2}, {"id": 3}], 2)
last = tail_rows([{"id": 1}, {"id": 2}, {"id": 3}], 1)
by_team = group_rows([{"team": "a"}, {"team": "b"}], "team")
total = count_rows([{"id": 1}, {"id": 2}])
per_team = count_rows([{"team": "a"}, {"team": "a"}], "team")
summary = stats_rows([{"ms": 120}, {"ms": 80}], "ms")
joined = join_rows([{"id": 1, "name": "Ada"}], [{"id": "1", "team": "core"}], "id")
report = validate_rows([{"id": 1}, {"id": "x"}], required=["id"], types={"id": ["integer"]})

NDJSON notes

NDJSON (also known as JSON Lines / .jsonl) is one JSON value per non-empty line. Parsing always yields a list. Serializing a list emits one compact JSON value per line; a non-list root becomes a single line.

CSV notes

CSV support is intentionally small and robust (RFC 4180-ish) via the Python standard library csv module: quoted fields, escaped quotes (""), and newlines inside quotes. Object arrays use a header row; all cell values are strings when parsed from CSV.

Stability

From 1.0.0 the command names, options, exit codes, and output shapes documented here follow Semantic Versioning: they only change in a breaking way in a new major version. New commands and options may arrive in minor versions. The functions exported from data_transform follow the same rule; anything starting with _ is internal.

Changelog

See CHANGELOG.md.

Development

pip install -e ".[dev]"
pytest

Tests use pytest with fixtures under tests/fixtures/ for JSON/YAML/CSV/NDJSON round-trips and CLI smoke checks.

License

MIT — see LICENSE.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
src/data_transform		src/data_transform
tests		tests
.gitignore		.gitignore
CHANGELOG.md		CHANGELOG.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md
pyproject.toml		pyproject.toml

Navigation Menu

Repository navigation

About

Repository files navigation

data-transform-cli

Install

Commands

Options (most commands)

Output formats

Usage examples

Select, pick, omit

Rename and flatten

Unflatten

Filter, sort, unique

Head and tail

Group-by and count

Stats

Join

Validate

Library use

NDJSON notes

CSV notes

Stability

Changelog

Development

License

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

data-transform-cli

Install

Commands

Options (most commands)

Output formats

Usage examples

Select, pick, omit

Rename and flatten

Unflatten

Filter, sort, unique

Head and tail

Group-by and count

Stats

Join

Validate

Library use

NDJSON notes

CSV notes

Stability

Changelog

Development

License

About

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages