Read result tables and points with one RFC 4180 CSV reader; check outputs and choices of cached schemas

README.md

@@ -31,12 +31,12 @@ Needs **labconstrictor-tools 0.3.0a1 or newer** in the app's environment (the ru
 ## What it does
 * Pick an app and a tool. Numbers (range, unit), choices, checkboxes, text, files and folders become fields; parameters of a group sit under a heading, advanced ones in a collapsed section, optional ones without a default have a "set" box (unticked = the tool gets nothing), and `enabled_when` greys fields out.
 * **Image inputs**: the image open in QuPath, any image of the open project, or a file. QuPath's image is exported losslessly as TIFF (the whole image, or the part chosen under **Image area**, see below). The pixel size of the chosen image or typed file fills the tool's pixel-size field (see **Pixel size** below).
-* Results open in a window: values, tables (as a table: the first 100 000 rows, then "table 'name' (N rows, first M shown; the full table is in P)", the Napari wording; cells with commas, quotes and line breaks are read as CSV says), images (a preview, the path and an "Open in QuPath" button), alignment matrices, files, messages (also shown under the status line) and points.
+* Results open in a window: values, tables (as a table: the first 100 000 rows, then "table 'name' (N rows, first M shown; the full table is in P)", the Napari wording; cells with commas, quotes and line breaks are read as CSV says, by one reader (`LcCsv`) for tables and points: a record is one row or one point however many lines it spans, and a malformed file, a points file without `y` and `x` columns or with a coordinate that is not a finite number is said in one sentence naming the record, never shown or placed partly), images (a preview, the path and an "Open in QuPath" button), alignment matrices, files, messages (also shown under the status line) and points.
 * **Interaction hints** of the manifest: a `choices_from` parameter is a dropdown filled by another tool of the app (a text field when that tool cannot answer), `clear_after_run` parameters are reset after a successful run, a `Collapsed` group is a folded section, and an output with `Replace()` reuses the results window of the tool instead of opening a new one.
 * **Channel selector**: an image input declared with `PickChannel()` gets a Channel chooser (the channel names of the chosen image, or of a file); the tool receives only that channel.
 * **Points** found in the image that is open in QuPath become a point annotation named `<app>:<output>` (replaced by the next run when the output declares `Replace()`); points of any other image are shown as a table.
 * "No match" is a message, not an error. Progress bar, Cancel (a tool that ignores it is killed after 3 s), worker kept between runs (or not), details of the last run, no worker left behind when the window closes.
-* **Trust checks** on every registry entry (same rules as the Python registry and the Fiji script): the entry names its five text fields; the app name is a plain name; the interpreter lies inside the install prefix (which is not a filesystem root); the entry and its schema file are owned by you (in a shared folder also by root) and not writable by group or others; the schema sits next to the entry; the interpreter, its folder and the prefix are not writable by everybody (a sticky folder like `/tmp` excepted); tool ids and parameter names are plain names, parameter names Python identifiers (a parameter name becomes a file name). An entry that fails is listed under "skipped" with its reason and never started. On Windows the owner and permission checks do not apply.
+* **Trust checks** on every registry entry (same rules as the Python registry and the Fiji script): the entry names its five text fields; the app name is a plain name; the interpreter lies inside the install prefix (which is not a filesystem root); the entry and its schema file are owned by you (in a shared folder also by root) and not writable by group or others; the schema sits next to the entry; the interpreter, its folder and the prefix are not writable by everybody (a sticky folder like `/tmp` excepted); tool ids and parameter names are plain names, parameter names Python identifiers (a parameter name becomes a file name); a `choice` parameter lists its choices and every declared output has a plain name and a text type (the cached schema is checked as far as this host reads it, the Fiji script's rules plus the output shape). An entry that fails is listed under "skipped" with its reason and never started. On Windows the owner and permission checks do not apply.
 * **Results and run records, and their clean-up** (SPEC_WRONG_SCIENCE section 3). Each run makes a results folder `~/.labconstrictor/results/<time>_<6 hex>_<app>_<tool>` (names made safe, like the command line) and a run record folder `~/.labconstrictor/runs/<time>_<app>_<tool>`. **This host never deletes anything there.** It writes the two marker files of the spec: `.lc-owner.json` first, when the folder is made (a QuPath that dies mid-run leaves an owned, unfinished folder that nobody removes), and `.lc-done.json` last, atomically, when the run ends: `ok` (completed), `cancelled` (the tool honoured Cancel, or the worker had to be killed after Cancel was pressed, or the window was closed during the run), `failed` (the tool failed, or the worker died with no Cancel pressed). The status comes from the Tools helper `labconstrictor_tools.runs.done_status`. Then it starts `labconstrictor-tools prune` (`<the app's Python> -m labconstrictor_tools prune`) in a subprocess and does not wait for it: the newest 20 finished results and the newest 50 run records are kept, and Tools removes the older ones, **only** folders that carry a valid owner marker and a done marker. Folders from older versions (no marker), foreign folders, folders of runs that are still going or that crashed with QuPath (no done marker), links and files are never removed and do not use up the quota. A prune that cannot start, or that fails, is written to the log (`prune could not be started ...` / `prune ended with exit code N`) and never fails or delays the run; the folders then stay until a `prune` works. The older folders written by earlier versions of this script have no marker, so they are never cleaned up automatically (Tools has a follow-up item for an explicit `clean --legacy`).
 * **Run record** `runs/<...>/run.json`: built by the Tools helper `labconstrictor_tools.runs.build_record` and written with `record_json` (this script keeps no copy of the layout), plus `"host": "qupath"`, the `run_id` and `started_utc` of the run, and the `_calibration` input. A crash after Cancel keeps its crash text there (status `CRASHED`). If the record cannot be built (the app's Python cannot start, the helper fails) the status line says `(the run record could not be written (...); its folders stay as they are and are never removed)`. Every log line is also appended to the shared `~/.labconstrictor/logs/labconstrictor.log` (Tools line format, `qupath:` prefix, 1 MB rotation with 5 backups), so `labconstrictor-tools logs` and `support-bundle` see QuPath runs.
 * **Same rules as the other hosts** (the host-parity inventory of labconstrictor-tools): a form shows parameters in the Napari/Fiji order (a group gathered where it first appears; a Collapsed group is folded where it first appears; advanced last); `clear_after_run` resets every type of field; `enabled_when` follows text, number, choice, yes/no and "set" drivers (`""` counts as unset, numbers compare as numbers); a whole-number box holds ±(2^53 - 1) at most, declared limits or not (the largest whole number a double holds exactly, Tools `notebook.MAX_FORM_INTEGER`), a decimal box without declared limits ±1e15, and **a typed number beyond a limit is refused, never put at the limit** (see **Number limits** below); a choice of integers is sent as an integer; a missing input says `'L' is required`; NaN is refused with `'L' must be a finite number`; a folder input must be a folder; a worker that is idle for 10 minutes is closed; a worker ended after Cancel reads `cancelled (worker stopped)`; a worker that died is explained with the Tools sentences (a missing package first, then the exit code; exit code 3 is called an import failure only when the worker's last words contain `cannot import tool module`, Tools PR #124: the Windows C runtime also ends a process with 3 for `abort()` and a raised SIGSEGV); a tool with a parameter type this host does not know is refused alone (`this tool cannot be shown: ...`); two tools with one label are listed as `label (id)` and the apps are sorted by name; `Copy as command` writes Python text the way `repr` does (a newline cannot break the snippet) and says `there is no clipboard to copy to` when there is none; message results show `- ` items as bullets; a result file that is missing, or a result type this version does not know, is said in one sentence.
@@ -159,7 +159,7 @@ The `gui_test_parity_*` bodies need the registered `interactions` app (its Pytho
 
 `BODY=gui_test_parity_messages_body.groovy tests/run_gui_test.sh` opens a 3-plane TIFF (written with tifffile from `/home/tester/v-napari`) and checks that a refusal is the sentence alone, that a z-stack is refused for a `YX` tool with the Tools sentence and the way out, that a series, z, t and channel counts are named, and when nothing is refused (18 checks).
 
-`BODY=gui_test_parity_results_body.groovy tests/run_gui_test.sh` builds result tables of 150 000, 100 000, 2 001 and a few rows, with quoted commas, doubled quotes and multi-line cells, and checks the rows shown and the sentence (12 checks).
+`BODY=gui_test_parity_results_body.groovy tests/run_gui_test.sh` builds result tables of 150 000, 100 000, 2 001 and a few rows, with quoted commas, doubled quotes and multi-line cells, and checks the rows shown and the sentence, then the CSV reader on points files and malformed files (issue 17) (12 + 14 checks).
 
 `BODY=gui_test_calibration_body.groovy tests/run_gui_test.sh` runs CAL-01..CAL-20 and CAL-T1..CAL-T9 of `conformance.json` through the form (a file input, project images and the image open in QuPath, one real worker run each for a required and an optional field) and checks the request and the `_calibration` input, not only the field; the images are written by `tests/calibration_fixtures.py` (tifffile) with the Python of the registered `frame` app.
 

src/main/resources/org/cellmigrationlab/labconstrictor/qupath/LabConstrictorTools.groovy

@@ -792,6 +792,11 @@ class LcRegistry {
             for (p in tool.inputs) {
                 if (!(p instanceof Map) || !identifier(p.name) || !(p.label instanceof String))
                     return "tool '" + tool.id + "' has a parameter without an identifier name and a text label"
+                if (p.type == "choice" && !(p.choices instanceof List && p.choices)) return "tool '" + tool.id + "': choice parameter '" + p.name + "' has no choices"      // as the Fiji script
+            }
+            for (o in tool.outputs) {      // the results window reads `name`, `type` and `replace` of each declared output
+                if (!(o instanceof Map) || !plainName(o.name) || !(o.type instanceof String))
+                    return "tool '" + tool.id + "' has an output without a plain name and a text type"
             }
         }
         return null
@@ -2515,7 +2520,13 @@ class LcDialog {
                 messageLabel.visible = true; messageLabel.managed = true
                 break
             case "points":
-                box.children.addAll(boldLabel((r.name ?: "points") + " (points)"), new Label(placePoints(appName, r)), tableView(r.path as String, (r.name ?: "points") as String))
+                String placed
+                try { placed = placePoints(appName, r) }
+                catch (LcCsvError problem) {
+                    LcLog.warn("points '" + r.name + "' not placed", problem)
+                    placed = "Not placed on an image: " + problem.message
+                }
+                box.children.addAll(boldLabel((r.name ?: "points") + " (points)"), wrapped(placed), tableView(r.path as String, (r.name ?: "points") as String))
                 break
             case "shapes":
                 box.children.addAll(boldLabel((r.name ?: "shapes") + " (outlines)"), wrapped(placeShapes(appName, r)))
@@ -2663,16 +2674,10 @@ class LcDialog {
             LcLog.once("points-not-placed:" + appName + ":" + r.name, "points '" + r.name + "' not placed: they were not found in the image open in QuPath (table only)")
             return notPlacedSentence(r, "points")
         }
-        def lines = new File(r.path as String).readLines("UTF-8")
-        def header = splitCsv(lines[0])
-        def yi = header.indexOf("y"), xi = header.indexOf("x")
         def area = areaFor(r)
-        def ys = [], xs = []
-        lines.drop(1).each { line ->          // pixel centres, moved by the Image area's offset
-            def cells = splitCsv(line)
-            ys << LcArea.slideY(area, (cells[yi] as double) + 0.5d)
-            xs << LcArea.slideX(area, (cells[xi] as double) + 0.5d)
-        }
+        Map found = LcCsv.points(r.path as String)          // an LcCsvError (malformed file, missing column, bad coordinate) places nothing; addResult says why
+        def ys = found.ys.collect { LcArea.slideY(area, (it as double) + 0.5d) }      // pixel centres, moved by the Image area's offset
+        def xs = found.xs.collect { LcArea.slideX(area, (it as double) + 0.5d) }
         def name = appName + ":" + r.name
         def hierarchy = data.hierarchy
         def previous = hierarchy.annotationObjects.findAll { it.name == name }
@@ -2777,8 +2782,13 @@ class LcDialog {
 
     /** The first LcConst.TABLE_ROWS_SHOWN rows of a CSV file as a table; under it "table 'name' (N rows, first M shown)" (the Napari sentence) and the path. */
     static Node tableView(String csvPath, String name) {
-        def csv = readCsv(csvPath, LcConst.TABLE_ROWS_SHOWN + 1)      // the header and the rows that are shown
-        def rows = csv.rows
+        Map csv
+        try { csv = LcCsv.read(csvPath, LcConst.TABLE_ROWS_SHOWN + 1) }      // the header and the rows that are shown
+        catch (LcCsvError problem) {
+            LcLog.warn("table '" + name + "' cannot be shown", problem)
+            return wrapped("table '" + name + "' cannot be shown: " + problem.message)      // the sentence names the record; the file is left as it is
+        }
+        List<List<String>> rows = csv.rows as List<List<String>>
         if (rows.isEmpty()) return new Label("(empty table)")
         def table = new TableView<List<String>>()
         rows[0].eachWithIndex { String h, int i ->
@@ -2789,54 +2799,8 @@ class LcDialog {
         rows.drop(1).each { table.items.add(it) }
         table.prefHeight = LcConst.TABLE_HEIGHT
         int shown = rows.size() - 1
-        def counted = shown + csv.hidden
-        return new VBox(4, table, new Label("table '" + name + "' (" + (csv.hidden ? counted + " rows, first " + shown + " shown; the full table is in " + csvPath : counted + " rows") + ")  " + csvPath))
-    }
-
-    /** A CSV file as records: [rows: the first `keep` records (cells as text), hidden: how many more there are]. A quoted cell may hold commas, doubled quotes and line breaks. */
-    static Map readCsv(String csvPath, int keep) {
-        def rows = [], row = [], cell = new StringBuilder()
-        int hidden = 0
-        boolean quoted = false, any = false
-        def endRecord = {
-            row << cell.toString(); cell.setLength(0)
-            if (rows.size() < keep) rows << row else hidden++
-            row = []; any = false
-        }
-        new File(csvPath).withReader("UTF-8") { reader ->
-            int previous = -1
-            int c = reader.read()
-            for (; c != -1; c = reader.read()) {
-                any = true
-                if (quoted) {
-                    if (c == 34) { quoted = false } else cell.append((char) c)
-                } else if (c == 34 && previous == 34) {         // a doubled quote inside a quoted cell: one quote
-                    cell.append('"'); quoted = true
-                } else if (c == 34) { quoted = true }
-                else if (c == 44) { row << cell.toString(); cell.setLength(0) }
-                else if (c == 10) { endRecord() }
-                else if (c != 13) cell.append((char) c)
-                previous = c
-            }
-        }
-        if (any) endRecord()
-        return [rows: rows, hidden: hidden]
-    }
-
-    /** One CSV line into cells: commas separate, double quotes group, a doubled quote is one quote (no multi-line cells). */
-    static List<String> splitCsv(String line) {
-        def out = [], cur = new StringBuilder()
-        boolean quoted = false
-        for (int i = 0; i < line.length(); i++) {
-            char c = line.charAt(i)
-            if (quoted) {
-                if (c == ('"' as char)) { if (i + 1 < line.length() && line.charAt(i + 1) == ('"' as char)) { cur.append('"'); i++ } else quoted = false } else cur.append(c)
-            } else if (c == ('"' as char)) quoted = true
-            else if (c == (',' as char)) { out << cur.toString(); cur.setLength(0) }
-            else cur.append(c)
-        }
-        out << cur.toString()
-        return out
+        long counted = (csv.count as long) - 1        // the records of the file without the header
+        return new VBox(4, table, new Label("table '" + name + "' (" + (counted > shown ? counted + " rows, first " + shown + " shown; the full table is in " + csvPath : counted + " rows") + ")  " + csvPath))
     }
 }