Skip to content

perf(results): decide column kinds from formatted samples, type-test the rest (#1548) - #1665

Merged
ZhuchkaTriplesix merged 1 commit into
devfrom
issue/1548-dataset-type-detection
Oct 17, 2026
Merged

ZhuchkaTriplesix merged 1 commit into
devfrom
issue/1548-dataset-type-detection

Conversation

@ZhuchkaTriplesix

Copy link
Copy Markdown
Member

Closes #1548.

Before (measured)

CompactResultDataset.fromRawRows decides each column's kind (int64 / float64 / string / generic). With a cell formatter (every SQL driver passes one), it called the formatter and toString() again on every non-null cell of every column that stayed uniform. A normal table, where most columns are uniform, therefore formatted every cell once just to pick the kind.

The detection loop copied as is into a standalone Dart AOT script, PostgreSQL-like formatter (is chain + toString), 6 int + 4 double + 10 string columns:

rows before after
5,000 (the default result cap, every query run) 3.5 ms 0.5 ms
50,000 46 ms 20 ms

Change

  • The first 60 non-null cells of a column go through the formatter, as before; the rest are only type-tested (is int / is num / is String).
  • This relies on the formatters printing a column's ints, numbers and strings by the column's type, not by the value. I checked all three:
    • PostgreSQL: bytea / undecoded types;
    • MySQL: binary and tinyint(1) booleans;
    • SQLite: blobs.
      So the samples show whether the column prints as its values do. A type that is not uniform is still caught on every row by the type test.
  • The loop stops as soon as all three flags are false (the column can only be generic). Before, it waited for 60 samples; the result is the same.

Packing (the other ~39 ms of the 50k case in the issue) is unchanged.

Checked

Same kinds from the old and the new loop on the 50,000-row table plus:

  • a column of ints broken by a string at row 40,000 → generic;
  • an all-NULL column → string;
  • a bytea-typed column of strings → generic.

Tests (compact_result_dataset_test.dart)

  • Kinds with a formatter, with an upper bound on formatter calls during detection.
  • A late value of another type still makes the column generic and keeps its text.

Not run with flutter test locally; CI decides.

…the rest (#1548)

CompactResultDataset.fromRawRows formatted every non-null cell with the
driver's cell formatter (and stringified it again) only to decide whether
a column is int, float, string or generic, and a uniform column never
left that loop early.

The formatters print a column's ints, numbers and strings by the
column's type (bytea, binary, boolean...), not by the value, so the first
60 non-null cells of a column now go through the formatter and the rest
are type-tested; the loop also stops as soon as the column can only be
generic.

Standalone Dart AOT copy of the detection loop, 6 int + 4 double + 10
string columns: 50,000 rows 46 ms -> 20 ms; 5,000 rows (the default cap)
3.5 ms -> 0.5 ms per query run. Same kinds, including a column broken by
a late value of another type and a bytea-typed string column.
@github-actions github-actions Bot added enhancement New feature or request performance Theme parser epic label: performance core Core library logic and services backend Backend database driver execution and queries P3 Low priority / Polish & Enhancements labels Oct 17, 2026
@github-actions github-actions Bot added this to the 0.5.0 milestone Oct 17, 2026
@ZhuchkaTriplesix
ZhuchkaTriplesix merged commit 86ea84d into dev Oct 17, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend Backend database driver execution and queries core Core library logic and services enhancement New feature or request P3 Low priority / Polish & Enhancements performance Theme parser epic label: performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant