From 941bba80a78974df0625731bad199cb98c49747e Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 8 Oct 2026 12:41:42 +0000 Subject: [PATCH 1/3] Table inputs: exact floats, names as written, NA texts stay text, readable errors (F2, F3, F4, F7) Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Y2SN8SuEakTdiWy3F2WUqN --- docs/AUTHORING.md | 2 +- docs/PROTOCOL.md | 1 + docs/REGRESSION_LEDGER.md | 8 +-- labconstrictor_tools/convert.py | 81 +++++++++++++++++++++++++- tests/roundtrip_known_failures.py | 67 --------------------- tests/test_roundtrip_known_failures.py | 3 +- tests/test_table_input.py | 70 ++++++++++++++++++++++ 7 files changed, 155 insertions(+), 77 deletions(-) create mode 100644 tests/test_table_input.py diff --git a/docs/AUTHORING.md b/docs/AUTHORING.md index 5127194..a5ba13a 100644 --- a/docs/AUTHORING.md +++ b/docs/AUTHORING.md @@ -37,7 +37,7 @@ app lists in `construct.yaml` under `extra_files` like the rest of `src/`). Any | `x: int / float / bool / str` | number / checkbox / text field | | `x: Literal["a", "b"]` or an `Enum` | dropdown | | `x: Image` / `Labels` | an image chosen from the host (you receive a numpy array) | -| `x: Table` | a CSV file (you receive a pandas DataFrame) | +| `x: Table` | a CSV file (you receive a pandas DataFrame; floats exact, column names as written even if duplicated or empty, empty cells NaN, texts such as `NA` or `None` stay text unless the column is numeric: see PROTOCOL.md) | | `x: Path` / `File` | a file (you receive a `Path`) | | `x: Folder` | a folder (you receive a `Path`; the worker checks that it exists; hosts show a folder chooser) | | a default value | the form's default; no default = required | diff --git a/docs/PROTOCOL.md b/docs/PROTOCOL.md index 8be37f7..9067b30 100644 --- a/docs/PROTOCOL.md +++ b/docs/PROTOCOL.md @@ -16,6 +16,7 @@ Value rules (enforced by the worker, each violation is a `FAILURE` with code `in * A default must be valid for its declared type (checked when the schema is generated, not when a host omits the value). * A `choice` declared with an `Enum` annotation reaches the tool as the Enum member. * An image/labels input with no pixels is refused (`empty_image`), from a file or from shared memory. +* Table inputs (a CSV file, UTF-8, first line is the header) reach the tool as a DataFrame exactly as written: floating point cells are read with every digit (`float_precision="round_trip"`, about twice as slow as pandas' default on very large tables); column names are kept as written, duplicates (`a,a`) and empty names included (pandas' renaming to `a.1` / `Unnamed: 0` is switched off, so a tool must address such columns by position); an empty cell is missing (NaN); the texts `NA`, `N/A`, `NULL`, `None`, `nan`, `null`, ... stay text, except in a column whose other cells are all numbers, where they are missing numbers. A table file that does not exist is refused with `file_not_found`; one that is empty or not a UTF-8 CSV with `unreadable_table`; both messages name the file. * Image outputs: integers are never changed in value (wide integers that fit 16 bits are stored as 16-bit; others must be exact as 32-bit float or the result fails with `unsupported_dtype`); floating point values are stored as float32 (about 7 significant digits); float values outside the float32 range fail. Input `type`: `string integer float boolean choice image labels table file folder`. diff --git a/docs/REGRESSION_LEDGER.md b/docs/REGRESSION_LEDGER.md index 87057bb..2e8e173 100644 --- a/docs/REGRESSION_LEDGER.md +++ b/docs/REGRESSION_LEDGER.md @@ -13,6 +13,10 @@ How to add one: [TESTING.md](TESTING.md#the-ledger). | 2026-10-08 | macOS: one request line that was not valid UTF-8 killed the worker's reader thread; the task in flight and every later task never ended | stdin decodes strictly there, so the `UnicodeDecodeError` ended the reader thread (the Windows reader decodes bytes itself) | [#36](https://github.com/CellMigrationLab/LabConstrictor-Tools/pull/36): `protocol._stdin_lines` reconfigures stdin with `errors="replace"`, and the line is reported and skipped | `test_roundtrip_ledger.py::NonUtf8Line.test_a_line_that_is_not_utf8_is_skipped_and_the_next_task_still_runs` | | 2026-10-08 | `labconstrictor-tools doctor --json \| head` (any command whose reader goes away) ended with a broken pipe (BrokenPipeError) traceback; on Windows the same shows up as EINVAL | stdout was flushed at interpreter exit, after the broken-pipe handler had already run, so the second failure was not handled | [#36](https://github.com/CellMigrationLab/LabConstrictor-Tools/pull/36): `__main__.main` flushes inside the handler, points stdout at the null device, and counts Windows' EINVAL as a closed reader | `test_roundtrip_ledger.py::BrokenPipeOnExit.test_the_reader_of_the_output_going_away_is_not_an_error` (and `test_silent_fallbacks.py`) | | 2026-10-08 | The test that checks the message after a cancel that had to kill the worker (`CancelThatHadToKill`) sometimes saw CANCELED instead of the forced kill (CRASHED) | A race in the test, not in the product: LAUNCH only says that the request arrived, so a Cancel sent right after it reached the stubborn tool before it had started to run, and the worker cancels a task that has not started | [#36](https://github.com/CellMigrationLab/LabConstrictor-Tools/pull/36) (the test now lets the tool start) | `test_hardening.py::CancelThatHadToKill.test_message_blames_the_tool_not_the_oom_killer`; and `test_roundtrip_failures.py::HangsAndCancel.test_cancel_before_the_task_has_started_still_cancels_it` pins the product side of the race (25 repetitions: a task cancelled before it starts is always CANCELED) | +| 2026-10-08 | F2: floating point cells of a Table input are not read exactly (0.30000000000000004 arrives as 0.3) | `pd.read_csv` without `float_precision="round_trip"` | this PR | `tests/test_table_input.py::TableInput.test_floats_are_exact`; also `test_roundtrip_matrix.py::Matrix.test_worker_protocol` | +| 2026-10-08 | F3: duplicate and empty column names of a Table input are renamed (`a.1`, `Unnamed: 0`) | `pd.read_csv` header mangling | this PR | `tests/test_table_input.py::TableInput.test_duplicate_and_empty_names_are_kept`; also `test_roundtrip_matrix.py::Matrix.test_worker_protocol` | +| 2026-10-08 | F4: a missing, empty or binary table file fails with `FileNotFoundError` / `EmptyDataError` / `UnicodeDecodeError` | no `file_not_found` / unreadable check for tables as for images | this PR | `tests/test_table_input.py::TableInput.test_unreadable_files_are_tool_errors`; also `test_roundtrip_matrix.py::Matrix.test_worker_protocol` | +| 2026-10-08 | F7: table text cells such as `NA`, `null`, `None` arrive as NaN | pandas default NA values | this PR | `tests/test_table_input.py::TableInput.test_na_words_stay_text_unless_numeric`; also `test_roundtrip_matrix.py::Matrix.test_worker_protocol` | ## Found by the round-trip matrix, fix PRs pending @@ -23,12 +27,8 @@ a finding deletes its entries there and moves its row up to the table above. | Date | Symptom | Root cause | Fix PR | Guarding test | |---|---|---|---|---| | 2026-10-08 | F1: a number parameter given text, a list, NaN or infinity fails with `[ValueError]`/`[TypeError]`/`[OverflowError]` instead of `invalid_parameter` | `convert._load_one` calls `int(value)` / `float(value)` before it checks anything | pending | `test_roundtrip_matrix.py::Matrix.test_worker_protocol` (`echo_int/word` and 12 more), `test_roundtrip_properties.py::Properties.test_only_toolerror_integer`, `test_roundtrip_properties.py::Properties.test_only_toolerror_float` | -| 2026-10-08 | F2: floating point cells of a Table input are not read exactly (0.30000000000000004 arrives as 0.3) | `pd.read_csv` without `float_precision="round_trip"` | pending | `test_roundtrip_matrix.py::Matrix.test_worker_protocol` (`echo_table/float-digits`, `random-floats`) | -| 2026-10-08 | F3: duplicate and empty column names of a Table input are renamed (`a.1`, `Unnamed: 0`) | `pd.read_csv` header mangling | pending | `test_roundtrip_matrix.py::Matrix.test_worker_protocol` (`echo_table/duplicate-column-names`) | -| 2026-10-08 | F4: a missing, empty or binary table file fails with `FileNotFoundError` / `EmptyDataError` / `UnicodeDecodeError` | no `file_not_found` / unreadable check for tables as for images | pending | `test_roundtrip_matrix.py::Matrix.test_worker_protocol` (`echo_table/missing-file`, `no-header-no-rows`, `binary-garbage`) | | 2026-10-08 | F5: a tool that returns the wrong kind of value for an output gets a raw Python exception code | `build_results` and its writers trust the shape and type of what a tool returns | pending | `test_roundtrip_failures.py::WhatAToolReturns.test_the_wrong_type_is_reported_with_a_tool_error_code` | | 2026-10-08 | F6: an Affine output with NaN or infinity is delivered as null | `_as_3x3` does not check finiteness (points and shapes do) | pending | `test_roundtrip_matrix.py::Matrix.test_worker_protocol` (`echo_affine/invalid/nan-cell`) | -| 2026-10-08 | F7: table text cells such as `NA`, `null`, `None` arrive as NaN | pandas default NA values | pending | `test_roundtrip_matrix.py::Matrix.test_worker_protocol` (`echo_table/text-that-pandas-calls-missing`) | | 2026-10-08 | F8, F9: the notebook form clamps numbers beyond 10**9 and cannot leave an optional parameter unset (the tool receives 0, empty text, False, the first choice) | `BoundedIntText` / `Text` / `Dropdown` controls always hold a value | pending | `test_roundtrip_matrix.py::Matrix.test_notebook_form` (`echo_optional_*/omitted`, `echo_int/ok/2147483648`) | | 2026-10-08 | F10: a copied Windows command line changes a text with backslashes before a quote | `command.quote(windows=True)` does not double backslashes in front of a quote | pending | `test_roundtrip_command.py::QuotingHelper.test_windows_quoting_keeps_every_text_one_argument`, `test_roundtrip_command.py::QuotingHelper.test_windows_quoting_survives_the_real_process_start` (the Windows jobs of CI show it on a real process start) | | 2026-10-08 | F11, F12: a malformed shared-memory descriptor, or a non-text file/folder value, raises KeyError / FileNotFoundError / TypeError | no validation before `SharedMemory(...)` and `Path(...)` | pending | `test_roundtrip_properties.py::Properties.test_only_toolerror_image_shared_memory`, `test_roundtrip_properties.py::Properties.test_only_toolerror_file_value` | diff --git a/labconstrictor_tools/convert.py b/labconstrictor_tools/convert.py index f7becd0..c4ca12e 100644 --- a/labconstrictor_tools/convert.py +++ b/labconstrictor_tools/convert.py @@ -62,9 +62,7 @@ def _load_one(param: ParamSchema, value: Any) -> Any: ) return _check_dimensions(param, array) if kind == "table": - import pandas as pd - - return pd.read_csv(value) + return _read_table(param, value) if kind == "file": return Path(value) if kind == "folder": @@ -118,6 +116,83 @@ def _check_dimensions(param: ParamSchema, array: Any) -> Any: return array +# Texts pandas reads as "missing" by default. In a table input they stay text, except in a column whose other cells are numbers +# (there they are missing numbers); an empty cell is always missing. Documented in docs/PROTOCOL.md ("Table inputs"). +PANDAS_NA_WORDS = [ + "#N/A", + "#N/A N/A", + "#NA", + "-1.#IND", + "-1.#QNAN", + "-NaN", + "-nan", + "1.#IND", + "1.#QNAN", + "", + "N/A", + "NA", + "NULL", + "NaN", + "None", + "n/a", + "nan", + "null", +] +EMPTY_CELL = "" + + +def _read_table(param: ParamSchema, value: Any) -> Any: + """Read a CSV file into a DataFrame exactly as the host wrote it: floats with all their digits (`round_trip`), column names + as written (duplicates and empty names included: columns are read by position and named afterwards), empty cells missing, + texts like "NA" kept as text unless the column is numeric. A missing or unreadable file is a ToolError, not a pandas error. + """ + import pandas as pd + + path = Path(value) + label = param["label"] + if not path.is_file(): + raise ToolError("file_not_found", "'%s': table file not found: %s" % (label, path)) + try: + header = pd.read_csv(path, header=None, nrows=1, dtype=str, keep_default_na=False) + names = [str(name) for name in header.iloc[0]] + columns = list(range(len(names))) + options: dict[str, Any] = {"header": 0, "names": columns, "float_precision": "round_trip"} + frame = pd.read_csv(path, keep_default_na=False, na_values=[EMPTY_CELL], **options) + words = [ + column + for column in columns + if pd.api.types.is_string_dtype(frame[column]) and frame[column].isin(PANDAS_NA_WORDS).any() + ] + if words: # a column of numbers with some "NA" cells is numeric, with those cells missing + numeric = pd.read_csv( + path, + usecols=words, + keep_default_na=False, + na_values=PANDAS_NA_WORDS + [EMPTY_CELL], + **options, + ) + for column in words: + if ( + pd.api.types.is_numeric_dtype(numeric[column]) and numeric[column].notna().any() + ): # all words: text + frame[column] = numeric[column] + frame.columns = names + return frame + except ( + pd.errors.ParserError, + pd.errors.EmptyDataError, + UnicodeDecodeError, + ValueError, + OSError, + ) as error: + logging.getLogger("labconstrictor.convert").error("cannot read table %s", path, exc_info=True) + raise ToolError( + "unreadable_table", + "cannot read %s as a CSV table: %s; save the table as a UTF-8 CSV file with a header row" + % (path.name, error), + ) from error + + def _read_image(value: Any) -> Any: if isinstance(value, dict) and value.get("appose_type") == "ndarray": return _ndarray_from_shared_memory(value) diff --git a/tests/roundtrip_known_failures.py b/tests/roundtrip_known_failures.py index 18faf42..3505f2d 100644 --- a/tests/roundtrip_known_failures.py +++ b/tests/roundtrip_known_failures.py @@ -39,28 +39,6 @@ class Finding: "as docs/PROTOCOL.md promises for every value-rule violation", "FAILURE with the Python exception class as code, e.g. [ValueError] invalid literal for int() with base 10: 'abc'", ), - "F2": Finding( - "A table input does not read floating point cells exactly", - 'a CSV file "f\\n0.30000000000000004\\n" given to a Table parameter: convert.load_inputs(schema, {"t": path})["t"]["f"][0]' - " is 0.3 (and 6.4575963826141304e+16 is read as 6.45759638261413e+16)", - "the cell equals float('0.30000000000000004'): pd.read_csv(path, float_precision='round_trip') in convert._load_one", - "pandas' default C parser rounds the last digit; the tool computes with a different number than the host wrote, and a " - "TableOut echo then writes the rounded value back", - ), - "F3": Finding( - "Duplicate and empty column names of a table input are renamed silently", - 'a CSV whose header is "a,a" (or ",b"): the tool receives the columns ["a", "a.1"] (["Unnamed: 0", "b"])', - "the tool sees the names the host wrote, or the call fails with a message that names the problem", - "pd.read_csv mangles the header; nothing tells the person", - ), - "F4": Finding( - "A table that cannot be read raises a raw pandas/OS exception instead of a ToolError", - "a missing file, an empty file or binary garbage given to a Table parameter: FileNotFoundError / EmptyDataError / " - "UnicodeDecodeError", - "ToolError('file_not_found', ...) for a missing file (like image: file_not_found) and ToolError('unreadable_table', ...) " - "for content that is not a CSV, so the host shows a readable message", - "FAILURE with code FileNotFoundError / EmptyDataError / UnicodeDecodeError", - ), "F5": Finding( "A tool that returns the wrong kind of value for an output gets a raw Python exception, not a ToolError('bad_return')", "a tool returns [[1, 0, 0], [0, 1], [0, 0, 1]] (ragged) or [['a', 0, 0], [0, 1, 0], [0, 0, 1]] as its Affine output; or " @@ -77,13 +55,6 @@ class Finding: "('affines are finite 3x3')", "COMPLETE with matrix_yx containing null; a host that applies it moves the image to nowhere", ), - "F7": Finding( - "Text cells that pandas treats as missing (NA, null, None, N/A, nan, ...) arrive as NaN", - 'a CSV column holding the texts "NA", "null", "None": the tool receives NaN for every one of them', - "a text cell stays text (pd.read_csv(..., keep_default_na=False) and an empty cell as missing), or the behaviour is " - "documented in docs/PROTOCOL.md", - "silent loss of the cell's content (a gene called NA, a status 'None')", - ), "F8": Finding( "The notebook form clamps integers and floats beyond 10**9 without telling the person", "labconstrictor_tools.notebook.ToolForm(echo_int).controls['value'].value = 2**31; form.run() reports 1000000000", @@ -221,36 +192,6 @@ def _add(finding: str, *keys: str) -> None: "property:only_toolerror_float", "property:only_toolerror_bounded", ) -_add( - "F2", - "cli:echo_table/float-digits", - "cli:echo_table/random-floats", - "notebook:echo_table/float-digits", - "notebook:echo_table/random-floats", - "snippet:echo_table/float-digits", - "snippet:echo_table/random-floats", - "terminal:echo_table/float-digits", - "terminal:echo_table/random-floats", - "worker:echo_table/float-digits", - "worker:echo_table/random-floats", - "property:table_floats_survive_the_csv_exactly", -) -_add( - "F3", - "worker:echo_table/duplicate-column-names", - "worker:echo_table/empty-column-name", -) -_add( - "F4", - "cli:echo_table/missing-file", - "cli:echo_table/no-header-no-rows", - "terminal:echo_table/missing-file", - "terminal:echo_table/no-header-no-rows", - "worker:echo_table/binary-garbage", - "worker:echo_table/missing-file", - "worker:echo_table/no-header-no-rows", - "property:only_toolerror_table_file", -) _add( "F5", "property:only_toolerror_result_affine", @@ -272,14 +213,6 @@ def _add(finding: str, *keys: str) -> None: "worker:echo_affine/invalid/inf-cell", "worker:echo_affine/invalid/nan-cell", ) -_add( - "F7", - "cli:echo_table/text-that-pandas-calls-missing", - "notebook:echo_table/text-that-pandas-calls-missing", - "snippet:echo_table/text-that-pandas-calls-missing", - "terminal:echo_table/text-that-pandas-calls-missing", - "worker:echo_table/text-that-pandas-calls-missing", -) _add( "F8", "notebook:echo_float/ok/0x1.c6bf526340000p+49", diff --git a/tests/test_roundtrip_known_failures.py b/tests/test_roundtrip_known_failures.py index ee6d260..5ad61d4 100644 --- a/tests/test_roundtrip_known_failures.py +++ b/tests/test_roundtrip_known_failures.py @@ -171,8 +171,7 @@ def test_windows_process_entries_still_fail(self): def test_the_skip_is_exactly_the_list(self): self.assertFalse(is_known("worker:echo_int/ok/0")) self.assertTrue(is_known("worker:echo_int/word")) - if os.name in ("posix", "nt"): - self.assertTrue(is_known("cli:echo_table/missing-file")) + self.assertFalse(is_known("cli:echo_table/missing-file")) # F4 is fixed: its ids are no longer skipped def tearDownModule(): diff --git a/tests/test_table_input.py b/tests/test_table_input.py new file mode 100644 index 0000000..dbcb206 --- /dev/null +++ b/tests/test_table_input.py @@ -0,0 +1,70 @@ +"""Table inputs (convert.load_inputs with a CSV file): floats exact, names as written, missing files refused, NA texts stay text.""" + +import tempfile +import unittest +from pathlib import Path + +import _paths # noqa: F401 (must come first) + +from labconstrictor_tools import convert +from labconstrictor_tools.types import ToolError + +SCHEMA = { + "id": "t", + "label": "t", + "outputs": [], + "inputs": [{"name": "t", "label": "Table", "type": "table", "required": True}], +} + + +class TableInput(unittest.TestCase): + def read(self, content: bytes | str, name: str = "t.csv"): + with tempfile.TemporaryDirectory() as folder: + path = Path(folder) / name + path.write_bytes(content if isinstance(content, bytes) else content.encode()) + return convert.load_inputs(SCHEMA, {"t": str(path)})["t"] + + def test_floats_are_exact(self): + frame = self.read("f\n0.30000000000000004\n6.4575963826141304e+16\n") + self.assertEqual(frame["f"].tolist(), [float("0.30000000000000004"), float("6.4575963826141304e+16")]) + + def test_duplicate_and_empty_names_are_kept(self): + self.assertEqual(list(self.read("a,a\n1,2\n").columns), ["a", "a"]) + self.assertEqual(list(self.read(",b\n1,2\n").columns), ["", "b"]) + frame = self.read("a,a,b\n1,x,3\n") + self.assertEqual(frame.iloc[0].tolist(), [1, "x", 3]) + self.assertEqual(frame.dtypes.iloc[0].kind, "i") + + def test_unreadable_files_are_tool_errors(self): + with tempfile.TemporaryDirectory() as folder: + with self.assertRaises(ToolError) as caught: + convert.load_inputs(SCHEMA, {"t": str(Path(folder) / "nothing.csv")}) + self.assertEqual(caught.exception.code, "file_not_found") + self.assertIn("nothing.csv", str(caught.exception)) + for content in (b"", bytes(range(256)) * 4): + with self.assertRaises(ToolError) as caught: + self.read(content, "bad.csv") + self.assertEqual(caught.exception.code, "unreadable_table") + self.assertIn("bad.csv", str(caught.exception)) + + def test_na_words_stay_text_unless_numeric(self): + words = ["NA", "null", "None", "N/A", "nan", "NaN", "n/a"] + frame = self.read("t,n,e\n" + "".join("%s,%d,\n" % (w, i) for i, w in enumerate(words)) + "x,7,\n") + self.assertEqual(frame["t"].tolist(), words + ["x"]) + self.assertEqual(frame["n"].tolist(), list(range(7)) + [7]) + self.assertTrue(frame["e"].isna().all()) + + def test_na_word_in_a_numeric_column_is_a_missing_number(self): + frame = self.read("n,s\n1.5,a\nNA,\nNone,b\n3,\n") + self.assertEqual(frame["n"].isna().tolist(), [False, True, True, False]) + self.assertEqual(frame["n"].dropna().tolist(), [1.5, 3.0]) + self.assertTrue(frame["s"].isna().tolist()[1]) # an empty cell is missing + + def test_header_only_table_has_its_columns(self): + frame = self.read("a,b\n") + self.assertEqual(list(frame.columns), ["a", "b"]) + self.assertEqual(len(frame), 0) + + +if __name__ == "__main__": + unittest.main() From 85aff9f72e7859b35ad5503fd22fa54d1973c49b Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 8 Oct 2026 13:43:30 +0000 Subject: [PATCH 2/3] Table inputs: NaN spellings are missing numbers (all-NaN column stays float) No existing test edited in this commit. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Y2SN8SuEakTdiWy3F2WUqN --- docs/PROTOCOL.md | 2 +- labconstrictor_tools/convert.py | 19 ++++++++++++++++--- tests/test_roundtrip_known_failures.py | 4 +++- tests/test_table_input.py | 21 +++++++++++++++++++++ 4 files changed, 41 insertions(+), 5 deletions(-) diff --git a/docs/PROTOCOL.md b/docs/PROTOCOL.md index 9b4da27..3a2a723 100644 --- a/docs/PROTOCOL.md +++ b/docs/PROTOCOL.md @@ -16,7 +16,7 @@ Value rules (enforced by the worker, each violation is a `FAILURE` with code `in * A default must be valid for its declared type (checked when the schema is generated, not when a host omits the value). * A `choice` declared with an `Enum` annotation reaches the tool as the Enum member. * An image/labels input with no pixels is refused (`empty_image`), from a file or from shared memory. -* Table inputs (a CSV file, UTF-8, first line is the header) reach the tool as a DataFrame exactly as written: floating point cells are read with every digit (`float_precision="round_trip"`, about twice as slow as pandas' default on very large tables); column names are kept as written, duplicates (`a,a`) and empty names included (pandas' renaming to `a.1` / `Unnamed: 0` is switched off, so a tool must address such columns by position); an empty cell is missing (NaN); the texts `NA`, `N/A`, `NULL`, `None`, `nan`, `null`, ... stay text, except in a column whose other cells are all numbers, where they are missing numbers. A table file that does not exist is refused with `file_not_found`; one that is empty or not a UTF-8 CSV with `unreadable_table`; both messages name the file. +* Table inputs (a CSV file, UTF-8, first line is the header) reach the tool as a DataFrame exactly as written: floating point cells are read with every digit (`float_precision="round_trip"`, about twice as slow as pandas' default on very large tables); column names are kept as written, duplicates (`a,a`) and empty names included (pandas' renaming to `a.1` / `Unnamed: 0` is switched off, so a tool must address such columns by position); an empty cell is missing (NaN); the spellings `NaN`, `nan`, `NAN` mean a missing number (a column of numbers, `NaN` and empty cells, or of only `NaN` and empty cells, is a float column with NaN), and the other words pandas calls missing (`NA`, `N/A`, `NULL`, `None`, `null`, ...) stay text, except in a column whose other cells are all numbers, where they are missing numbers; a text column that happens to contain `nan` keeps it as text. A table file that does not exist is refused with `file_not_found`; one that is empty or not a UTF-8 CSV with `unreadable_table`; both messages name the file. * Image outputs: integers are never changed in value (wide integers that fit 16 bits are stored as 16-bit; others must be exact as 32-bit float or the result fails with `unsupported_dtype`); floating point values are stored as float32 (about 7 significant digits); float values outside the float32 range fail. Input `type`: `string integer float boolean choice image labels table file folder`. diff --git a/labconstrictor_tools/convert.py b/labconstrictor_tools/convert.py index 2295fac..d9906fa 100644 --- a/labconstrictor_tools/convert.py +++ b/labconstrictor_tools/convert.py @@ -135,12 +135,16 @@ def _check_dimensions(param: ParamSchema, array: Any) -> Any: "N/A", "NA", "NULL", + "NAN", "NaN", "None", "n/a", "nan", "null", ] +NAN_SPELLINGS = frozenset( + {"NaN", "nan", "NAN"} +) # what Fiji, QuPath, numpy and pandas write for an unavailable number EMPTY_CELL = "" @@ -161,11 +165,20 @@ def _read_table(param: ParamSchema, value: Any) -> Any: columns = list(range(len(names))) options: dict[str, Any] = {"header": 0, "names": columns, "float_precision": "round_trip"} frame = pd.read_csv(path, keep_default_na=False, na_values=[EMPTY_CELL], **options) + + def is_text(column: int) -> bool: + return bool(pd.api.types.is_string_dtype(frame[column])) + words = [ - column - for column in columns - if pd.api.types.is_string_dtype(frame[column]) and frame[column].isin(PANDAS_NA_WORDS).any() + column for column in columns if is_text(column) and frame[column].isin(PANDAS_NA_WORDS).any() ] + for column in words: + cells = frame[column].dropna() + if cells.isin( + NAN_SPELLINGS + ).all(): # only NaN spellings (and empty cells): an unavailable measurement + frame[column] = pd.Series(float("nan"), index=frame.index, dtype=float) + words = [column for column in words if is_text(column)] if words: # a column of numbers with some "NA" cells is numeric, with those cells missing numeric = pd.read_csv( path, diff --git a/tests/test_roundtrip_known_failures.py b/tests/test_roundtrip_known_failures.py index 5ad61d4..e410712 100644 --- a/tests/test_roundtrip_known_failures.py +++ b/tests/test_roundtrip_known_failures.py @@ -171,7 +171,9 @@ def test_windows_process_entries_still_fail(self): def test_the_skip_is_exactly_the_list(self): self.assertFalse(is_known("worker:echo_int/ok/0")) self.assertTrue(is_known("worker:echo_int/word")) - self.assertFalse(is_known("cli:echo_table/missing-file")) # F4 is fixed: its ids are no longer skipped + self.assertFalse( + is_known("cli:echo_table/missing-file") + ) # F4 is fixed: its ids are no longer skipped def tearDownModule(): diff --git a/tests/test_table_input.py b/tests/test_table_input.py index dbcb206..bb39989 100644 --- a/tests/test_table_input.py +++ b/tests/test_table_input.py @@ -60,6 +60,27 @@ def test_na_word_in_a_numeric_column_is_a_missing_number(self): self.assertEqual(frame["n"].dropna().tolist(), [1.5, 3.0]) self.assertTrue(frame["s"].isna().tolist()[1]) # an empty cell is missing + def test_nan_spellings_are_missing_numbers(self): + for spelling in ("NaN", "nan", "NAN"): + frame = self.read("a,b\n%s,1\n%s,2\n" % (spelling, spelling)) + self.assertEqual(frame["a"].dtype.kind, "f", spelling) + self.assertTrue(frame["a"].isna().all()) + frame = self.read("a,b\nNaN,1\n,2\nnan,3\n") + self.assertEqual(frame["a"].dtype.kind, "f") + self.assertTrue(frame["a"].isna().all()) + frame = self.read("a\nNaN\n2.5\nnan\n") + self.assertEqual(frame["a"].dtype.kind, "f") + self.assertEqual(frame["a"].isna().tolist(), [True, False, True]) + + def test_nan_word_in_a_text_column_stays_text(self): + frame = self.read("a\nnan\nNA\nhello\nNaN\n") + self.assertEqual(frame["a"].tolist(), ["nan", "NA", "hello", "NaN"]) + + def test_infinity_columns_are_floats(self): + frame = self.read("a\ninf\n-inf\nInfinity\n1\n") + self.assertEqual(frame["a"].dtype.kind, "f") + self.assertEqual(frame["a"].tolist(), [float("inf"), float("-inf"), float("inf"), 1.0]) + def test_header_only_table_has_its_columns(self): frame = self.read("a,b\n") self.assertEqual(list(frame.columns), ["a", "b"]) From 369f4a8653ebb2e576db3bdfeab7897fa37f348e Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 8 Oct 2026 14:07:07 +0000 Subject: [PATCH 3/3] Table inputs: detect text columns by dtype so pandas 2.x object columns with missing cells are handled like pandas 3 No existing test edited. Co-Authored-By: Claude Sonnet 5.5 Claude-Session: https://claude.ai/code/session_01Y2SN8SuEakTdiWy3F2WUqN --- labconstrictor_tools/convert.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/labconstrictor_tools/convert.py b/labconstrictor_tools/convert.py index d9906fa..34eb489 100644 --- a/labconstrictor_tools/convert.py +++ b/labconstrictor_tools/convert.py @@ -167,7 +167,9 @@ def _read_table(param: ParamSchema, value: Any) -> Any: frame = pd.read_csv(path, keep_default_na=False, na_values=[EMPTY_CELL], **options) def is_text(column: int) -> bool: - return bool(pd.api.types.is_string_dtype(frame[column])) + return bool( + pd.api.types.is_string_dtype(frame[column].dtype) + ) # dtype, not cells: object columns hold NaN too words = [ column for column in columns if is_text(column) and frame[column].isin(PANDAS_NA_WORDS).any()