Fix: table inputs read floats exactly, keep names as written, NA words stay text, readable errors (F2 F3 F4 F7)

docs/AUTHORING.md

@@ -37,7 +37,7 @@ app lists in `construct.yaml` under `extra_files` like the rest of `src/`). Any
 | `x: int / float / bool / str` | number / checkbox / text field |
 | `x: Literal["a", "b"]` or an `Enum` | dropdown |
 | `x: Image` / `Labels` | an image chosen from the host (you receive a numpy array) |
-| `x: Table` | a CSV file (you receive a pandas DataFrame) |
+| `x: Table` | a CSV file (you receive a pandas DataFrame; floats exact, column names as written even if duplicated or empty, empty cells NaN, texts such as `NA` or `None` stay text unless the column is numeric: see PROTOCOL.md) |
 | `x: Path` / `File` | a file (you receive a `Path`) |
 | `x: Folder` | a folder (you receive a `Path`; the worker checks that it exists; hosts show a folder chooser) |
 | a default value | the form's default; no default = required |

docs/PROTOCOL.md

@@ -16,6 +16,7 @@ Value rules (enforced by the worker, each violation is a `FAILURE` with code `in
 * A default must be valid for its declared type (checked when the schema is generated, not when a host omits the value).
 * A `choice` declared with an `Enum` annotation reaches the tool as the Enum member.
 * An image/labels input with no pixels is refused (`empty_image`), from a file or from shared memory.
+* Table inputs (a CSV file, UTF-8, first line is the header) reach the tool as a DataFrame exactly as written: floating point cells are read with every digit (`float_precision="round_trip"`, about twice as slow as pandas' default on very large tables); column names are kept as written, duplicates (`a,a`) and empty names included (pandas' renaming to `a.1` / `Unnamed: 0` is switched off, so a tool must address such columns by position); an empty cell is missing (NaN); the spellings `NaN`, `nan`, `NAN` mean a missing number (a column of numbers, `NaN` and empty cells, or of only `NaN` and empty cells, is a float column with NaN), and the other words pandas calls missing (`NA`, `N/A`, `NULL`, `None`, `null`, ...) stay text, except in a column whose other cells are all numbers, where they are missing numbers; a text column that happens to contain `nan` keeps it as text. A table file that does not exist is refused with `file_not_found`; one that is empty or not a UTF-8 CSV with `unreadable_table`; both messages name the file.
 * Image outputs: integers are never changed in value (wide integers that fit 16 bits are stored as 16-bit; others must be exact as 32-bit float or the result fails with `unsupported_dtype`); floating point values are stored as float32 (about 7 significant digits); float values outside the float32 range fail.
 
 Input `type`: `string integer float boolean choice image labels table file folder`.

docs/REGRESSION_LEDGER.md

@@ -13,6 +13,10 @@ How to add one: [TESTING.md](TESTING.md#the-ledger).
 | 2026-10-08 | macOS: one request line that was not valid UTF-8 killed the worker's reader thread; the task in flight and every later task never ended | stdin decodes strictly there, so the `UnicodeDecodeError` ended the reader thread (the Windows reader decodes bytes itself) | [#36](https://github.com/CellMigrationLab/LabConstrictor-Tools/pull/36): `protocol._stdin_lines` reconfigures stdin with `errors="replace"`, and the line is reported and skipped | `test_roundtrip_ledger.py::NonUtf8Line.test_a_line_that_is_not_utf8_is_skipped_and_the_next_task_still_runs` |
 | 2026-10-08 | `labconstrictor-tools doctor --json \| head` (any command whose reader goes away) ended with a broken pipe (BrokenPipeError) traceback; on Windows the same shows up as EINVAL | stdout was flushed at interpreter exit, after the broken-pipe handler had already run, so the second failure was not handled | [#36](https://github.com/CellMigrationLab/LabConstrictor-Tools/pull/36): `__main__.main` flushes inside the handler, points stdout at the null device, and counts Windows' EINVAL as a closed reader | `test_roundtrip_ledger.py::BrokenPipeOnExit.test_the_reader_of_the_output_going_away_is_not_an_error` (and `test_silent_fallbacks.py`) |
 | 2026-10-08 | The test that checks the message after a cancel that had to kill the worker (`CancelThatHadToKill`) sometimes saw CANCELED instead of the forced kill (CRASHED) | A race in the test, not in the product: LAUNCH only says that the request arrived, so a Cancel sent right after it reached the stubborn tool before it had started to run, and the worker cancels a task that has not started | [#36](https://github.com/CellMigrationLab/LabConstrictor-Tools/pull/36) (the test now lets the tool start) | `test_hardening.py::CancelThatHadToKill.test_message_blames_the_tool_not_the_oom_killer`; and `test_roundtrip_failures.py::HangsAndCancel.test_cancel_before_the_task_has_started_still_cancels_it` pins the product side of the race (25 repetitions: a task cancelled before it starts is always CANCELED) |
+| 2026-10-08 | F2: floating point cells of a Table input are not read exactly (0.30000000000000004 arrives as 0.3) | `pd.read_csv` without `float_precision="round_trip"` | this PR | `tests/test_table_input.py::TableInput.test_floats_are_exact`; also `test_roundtrip_matrix.py::Matrix.test_worker_protocol` |
+| 2026-10-08 | F3: duplicate and empty column names of a Table input are renamed (`a.1`, `Unnamed: 0`) | `pd.read_csv` header mangling | this PR | `tests/test_table_input.py::TableInput.test_duplicate_and_empty_names_are_kept`; also `test_roundtrip_matrix.py::Matrix.test_worker_protocol` |
+| 2026-10-08 | F4: a missing, empty or binary table file fails with `FileNotFoundError` / `EmptyDataError` / `UnicodeDecodeError` | no `file_not_found` / unreadable check for tables as for images | this PR | `tests/test_table_input.py::TableInput.test_unreadable_files_are_tool_errors`; also `test_roundtrip_matrix.py::Matrix.test_worker_protocol` |
+| 2026-10-08 | F7: table text cells such as `NA`, `null`, `None` arrive as NaN | pandas default NA values | this PR | `tests/test_table_input.py::TableInput.test_na_words_stay_text_unless_numeric`; also `test_roundtrip_matrix.py::Matrix.test_worker_protocol` |
 
 ## Found by the round-trip matrix, fix PRs pending
 
@@ -23,12 +27,8 @@ a finding deletes its entries there and moves its row up to the table above.
 | Date | Symptom | Root cause | Fix PR | Guarding test |
 |---|---|---|---|---|
 | 2026-10-08 | F1: a number parameter given text, a list, NaN or infinity fails with `[ValueError]`/`[TypeError]`/`[OverflowError]` instead of `invalid_parameter` | `convert._load_one` calls `int(value)` / `float(value)` before it checks anything | pending | `test_roundtrip_matrix.py::Matrix.test_worker_protocol` (`echo_int/word` and 12 more), `test_roundtrip_properties.py::Properties.test_only_toolerror_integer`, `test_roundtrip_properties.py::Properties.test_only_toolerror_float` |
-| 2026-10-08 | F2: floating point cells of a Table input are not read exactly (0.30000000000000004 arrives as 0.3) | `pd.read_csv` without `float_precision="round_trip"` | pending | `test_roundtrip_matrix.py::Matrix.test_worker_protocol` (`echo_table/float-digits`, `random-floats`) |
-| 2026-10-08 | F3: duplicate and empty column names of a Table input are renamed (`a.1`, `Unnamed: 0`) | `pd.read_csv` header mangling | pending | `test_roundtrip_matrix.py::Matrix.test_worker_protocol` (`echo_table/duplicate-column-names`) |
-| 2026-10-08 | F4: a missing, empty or binary table file fails with `FileNotFoundError` / `EmptyDataError` / `UnicodeDecodeError` | no `file_not_found` / unreadable check for tables as for images | pending | `test_roundtrip_matrix.py::Matrix.test_worker_protocol` (`echo_table/missing-file`, `no-header-no-rows`, `binary-garbage`) |
 | 2026-10-08 | F5: a tool that returns the wrong kind of value for an output gets a raw Python exception code | `build_results` and its writers trust the shape and type of what a tool returns | pending | `test_roundtrip_failures.py::WhatAToolReturns.test_the_wrong_type_is_reported_with_a_tool_error_code` |
 | 2026-10-08 | F6: an Affine output with NaN or infinity is delivered as null | `_as_3x3` does not check finiteness (points and shapes do) | pending | `test_roundtrip_matrix.py::Matrix.test_worker_protocol` (`echo_affine/invalid/nan-cell`) |
-| 2026-10-08 | F7: table text cells such as `NA`, `null`, `None` arrive as NaN | pandas default NA values | pending | `test_roundtrip_matrix.py::Matrix.test_worker_protocol` (`echo_table/text-that-pandas-calls-missing`) |
 | 2026-10-08 | F8, F9: the notebook form clamps numbers beyond 10**9 and cannot leave an optional parameter unset (the tool receives 0, empty text, False, the first choice) | `BoundedIntText` / `Text` / `Dropdown` controls always hold a value | pending | `test_roundtrip_matrix.py::Matrix.test_notebook_form` (`echo_optional_*/omitted`, `echo_int/ok/2147483648`) |
 | 2026-10-08 | F10: a copied Windows command line changes a text with backslashes before a quote | `command.quote(windows=True)` does not double backslashes in front of a quote | pending | `test_roundtrip_command.py::QuotingHelper.test_windows_quoting_keeps_every_text_one_argument`, `test_roundtrip_command.py::QuotingHelper.test_windows_quoting_survives_the_real_process_start` (the Windows jobs of CI show it on a real process start) |
 | 2026-10-08 | F11, F12: a malformed shared-memory descriptor, or a non-text file/folder value, raises KeyError / FileNotFoundError / TypeError | no validation before `SharedMemory(...)` and `Path(...)` | pending | `test_roundtrip_properties.py::Properties.test_only_toolerror_image_shared_memory`, `test_roundtrip_properties.py::Properties.test_only_toolerror_file_value` |

labconstrictor_tools/convert.py

@@ -65,9 +65,7 @@ def _load_one(param: ParamSchema, value: Any) -> Any:
             )
         return _check_dimensions(param, array)
     if kind == "table":
-        import pandas as pd
-
-        return pd.read_csv(value)
+        return _read_table(param, value)
     if kind == "file":
         return Path(value)
     if kind == "folder":
@@ -121,6 +119,98 @@ def _check_dimensions(param: ParamSchema, array: Any) -> Any:
     return array
 
 
+# Texts pandas reads as "missing" by default. In a table input they stay text, except in a column whose other cells are numbers
+# (there they are missing numbers); an empty cell is always missing. Documented in docs/PROTOCOL.md ("Table inputs").
+PANDAS_NA_WORDS = [
+    "#N/A",
+    "#N/A N/A",
+    "#NA",
+    "-1.#IND",
+    "-1.#QNAN",
+    "-NaN",
+    "-nan",
+    "1.#IND",
+    "1.#QNAN",
+    "<NA>",
+    "N/A",
+    "NA",
+    "NULL",
+    "NAN",
+    "NaN",
+    "None",
+    "n/a",
+    "nan",
+    "null",
+]
+NAN_SPELLINGS = frozenset(
+    {"NaN", "nan", "NAN"}
+)  # what Fiji, QuPath, numpy and pandas write for an unavailable number
+EMPTY_CELL = ""
+
+
+def _read_table(param: ParamSchema, value: Any) -> Any:
+    """Read a CSV file into a DataFrame exactly as the host wrote it: floats with all their digits (`round_trip`), column names
+    as written (duplicates and empty names included: columns are read by position and named afterwards), empty cells missing,
+    texts like "NA" kept as text unless the column is numeric. A missing or unreadable file is a ToolError, not a pandas error.
+    """
+    import pandas as pd
+
+    path = Path(value)
+    label = param["label"]
+    if not path.is_file():
+        raise ToolError("file_not_found", "'%s': table file not found: %s" % (label, path))
+    try:
+        header = pd.read_csv(path, header=None, nrows=1, dtype=str, keep_default_na=False)
+        names = [str(name) for name in header.iloc[0]]
+        columns = list(range(len(names)))
+        options: dict[str, Any] = {"header": 0, "names": columns, "float_precision": "round_trip"}
+        frame = pd.read_csv(path, keep_default_na=False, na_values=[EMPTY_CELL], **options)
+
+        def is_text(column: int) -> bool:
+            return bool(
+                pd.api.types.is_string_dtype(frame[column].dtype)
+            )  # dtype, not cells: object columns hold NaN too
+
+        words = [
+            column for column in columns if is_text(column) and frame[column].isin(PANDAS_NA_WORDS).any()
+        ]
+        for column in words:
+            cells = frame[column].dropna()
+            if cells.isin(
+                NAN_SPELLINGS
+            ).all():  # only NaN spellings (and empty cells): an unavailable measurement
+                frame[column] = pd.Series(float("nan"), index=frame.index, dtype=float)
+        words = [column for column in words if is_text(column)]
+        if words:  # a column of numbers with some "NA" cells is numeric, with those cells missing
+            numeric = pd.read_csv(
+                path,
+                usecols=words,
+                keep_default_na=False,
+                na_values=PANDAS_NA_WORDS + [EMPTY_CELL],
+                **options,
+            )
+            for column in words:
+                if (
+                    pd.api.types.is_numeric_dtype(numeric[column]) and numeric[column].notna().any()
+                ):  # all words: text
+                    frame[column] = numeric[column]
+        frame.columns = names
+        return frame
+    except (
+        pd.errors.ParserError,
+        pd.errors.EmptyDataError,
+        UnicodeDecodeError,
+        ValueError,
+        OSError,
+    ) as error:
+        logging.getLogger("labconstrictor.convert").error("cannot read table %s", path, exc_info=True)
+        raise ToolError(
+            "unreadable_table",
+            "cannot read %s as a CSV table: %s; save the table as a UTF-8 CSV file with a header row"
+            % (path.name, error),
+        ) from error
+
+
 def _read_image(value: Any) -> Any:
     if isinstance(value, dict) and value.get("appose_type") == "ndarray":
         return _ndarray_from_shared_memory(value)

tests/roundtrip_known_failures.py

@@ -39,28 +39,6 @@ class Finding:
         "as docs/PROTOCOL.md promises for every value-rule violation",
         "FAILURE with the Python exception class as code, e.g. [ValueError] invalid literal for int() with base 10: 'abc'",
     ),
-    "F2": Finding(
-        "A table input does not read floating point cells exactly",
-        'a CSV file "f\\n0.30000000000000004\\n" given to a Table parameter: convert.load_inputs(schema, {"t": path})["t"]["f"][0]'
-        " is 0.3 (and 6.4575963826141304e+16 is read as 6.45759638261413e+16)",
-        "the cell equals float('0.30000000000000004'): pd.read_csv(path, float_precision='round_trip') in convert._load_one",
-        "pandas' default C parser rounds the last digit; the tool computes with a different number than the host wrote, and a "
-        "TableOut echo then writes the rounded value back",
-    ),
-    "F3": Finding(
-        "Duplicate and empty column names of a table input are renamed silently",
-        'a CSV whose header is "a,a" (or ",b"): the tool receives the columns ["a", "a.1"] (["Unnamed: 0", "b"])',
-        "the tool sees the names the host wrote, or the call fails with a message that names the problem",
-        "pd.read_csv mangles the header; nothing tells the person",
-    ),
-    "F4": Finding(
-        "A table that cannot be read raises a raw pandas/OS exception instead of a ToolError",
-        "a missing file, an empty file or binary garbage given to a Table parameter: FileNotFoundError / EmptyDataError / "
-        "UnicodeDecodeError",
-        "ToolError('file_not_found', ...) for a missing file (like image: file_not_found) and ToolError('unreadable_table', ...) "
-        "for content that is not a CSV, so the host shows a readable message",
-        "FAILURE with code FileNotFoundError / EmptyDataError / UnicodeDecodeError",
-    ),
     "F5": Finding(
         "A tool that returns the wrong kind of value for an output gets a raw Python exception, not a ToolError('bad_return')",
         "a tool returns [[1, 0, 0], [0, 1], [0, 0, 1]] (ragged) or [['a', 0, 0], [0, 1, 0], [0, 0, 1]] as its Affine output; or "
@@ -77,13 +55,6 @@ class Finding:
         "('affines are finite 3x3')",
         "COMPLETE with matrix_yx containing null; a host that applies it moves the image to nowhere",
     ),
-    "F7": Finding(
-        "Text cells that pandas treats as missing (NA, null, None, N/A, nan, ...) arrive as NaN",
-        'a CSV column holding the texts "NA", "null", "None": the tool receives NaN for every one of them',
-        "a text cell stays text (pd.read_csv(..., keep_default_na=False) and an empty cell as missing), or the behaviour is "
-        "documented in docs/PROTOCOL.md",
-        "silent loss of the cell's content (a gene called NA, a status 'None')",
-    ),
     "F8": Finding(
         "The notebook form clamps integers and floats beyond 10**9 without telling the person",
         "labconstrictor_tools.notebook.ToolForm(echo_int).controls['value'].value = 2**31; form.run() reports 1000000000",
@@ -180,36 +151,6 @@ def _add(finding: str, *keys: str) -> None:
     "property:only_toolerror_float",
     "property:only_toolerror_bounded",
 )
-_add(
-    "F2",
-    "cli:echo_table/float-digits",
-    "cli:echo_table/random-floats",
-    "notebook:echo_table/float-digits",
-    "notebook:echo_table/random-floats",
-    "snippet:echo_table/float-digits",
-    "snippet:echo_table/random-floats",
-    "terminal:echo_table/float-digits",
-    "terminal:echo_table/random-floats",
-    "worker:echo_table/float-digits",
-    "worker:echo_table/random-floats",
-    "property:table_floats_survive_the_csv_exactly",
-)
-_add(
-    "F3",
-    "worker:echo_table/duplicate-column-names",
-    "worker:echo_table/empty-column-name",
-)
-_add(
-    "F4",
-    "cli:echo_table/missing-file",
-    "cli:echo_table/no-header-no-rows",
-    "terminal:echo_table/missing-file",
-    "terminal:echo_table/no-header-no-rows",
-    "worker:echo_table/binary-garbage",
-    "worker:echo_table/missing-file",
-    "worker:echo_table/no-header-no-rows",
-    "property:only_toolerror_table_file",
-)
 _add(
     "F5",
     "property:only_toolerror_result_affine",
@@ -231,14 +172,6 @@ def _add(finding: str, *keys: str) -> None:
     "worker:echo_affine/invalid/inf-cell",
     "worker:echo_affine/invalid/nan-cell",
 )
-_add(
-    "F7",
-    "cli:echo_table/text-that-pandas-calls-missing",
-    "notebook:echo_table/text-that-pandas-calls-missing",
-    "snippet:echo_table/text-that-pandas-calls-missing",
-    "terminal:echo_table/text-that-pandas-calls-missing",
-    "worker:echo_table/text-that-pandas-calls-missing",
-)
 _add(
     "F8",
     "notebook:echo_float/ok/0x1.c6bf526340000p+49",

tests/test_roundtrip_known_failures.py

@@ -171,8 +171,9 @@ def test_windows_process_entries_still_fail(self):
     def test_the_skip_is_exactly_the_list(self):
         self.assertFalse(is_known("worker:echo_int/ok/0"))
         self.assertTrue(is_known("worker:echo_int/word"))
-        if os.name in ("posix", "nt"):
-            self.assertTrue(is_known("cli:echo_table/missing-file"))
+        self.assertFalse(
+            is_known("cli:echo_table/missing-file")
+        )  # F4 is fixed: its ids are no longer skipped
 
 
 def tearDownModule():