[dimitri-yatsenko]

Fixes #1660.

Background

Every DataJoint query has a primary key that identifies its rows. The SQL therefore needs deduplication in only two places:

• dj.U(...) & expr, when U's attributes do not include expr's primary key.
• The union of key sets that may overlap.

Everywhere else, DISTINCT (or UNION without ALL) only adds work for the database. Where DISTINCT is needed, some operators dropped it when the expression was used inside another query.

Redundant deduplication removed

• len() counts rows with count(*) unless the query deduplicates. It used to count distinct primary keys, which for multi-column keys meant count(*) FROM (SELECT DISTINCT pk ...) on every call.
• Aggregations no longer add DISTINCT when the heading equals the primary key, because GROUP BY on that key already returns one row per key. DISTINCT is now emitted only for dj.U(...) & aggregation, driven by _distinct instead of the heading == PK heuristic, which had been doing that job implicitly.
• The outer union (operands with secondary attributes) uses UNION ALL. Its branches, arg1 LEFT JOIN arg2 and arg2 - arg1, are disjoint.
• dj.U(...) & expr skips DISTINCT when U's attributes include expr's primary key.

Deduplication kept through composition

• Joins (Joining with a dj.U restriction returns duplicate rows and inflates aggr counts: the DISTINCT is dropped #1660). join (and so *, extend, aggr) wraps a dj.U(...) & expr operand in a subquery, so it contributes one row per key. It used to be joined as the underlying table, which gave repeated rows and inflated aggr counts. len() hid this by counting distinct keys. Without this fix, the len() change above would have exposed the wrong count in len() too.
• dj.U(...).aggr deduplicates a dj.U(...) & expr group before GROUP BY.
• dj.U(...) & union sets DISTINCT after proj(). proj() wraps a union in a subquery, and the flag used to be set on the inner copy, where Union.make_sql ignores it, so duplicates came through.

parents_with_children = dj.U("parent_id") & Child
len((Parent * parents_with_children).to_dicts())
# before: one row per child (7); after: one row per parent (4)

dj.U("kind").aggr(dj.U("parent_id", "kind") & Child, n="count(*)")
# before: n counts children; after: n counts parents per kind

Why it matters for performance

Measured outside the test suite on MySQL 8.0/8.4 and PostgreSQL 15/18:

• PostgreSQL. A deduplicating UNION hashes every column, blobs included, and the planner cannot drop unused columns under it. With UNION ALL, the planner flattens the union, drops unused columns, and removes the redundant left join: 2.5 ms instead of 167 ms on a blob table.
• Both backends. DISTINCT prevents MySQL from merging a derived table into the outer query and PostgreSQL from dropping unused columns. Removing the redundant ones avoids both problems.

Tests

tests/integration/test_distinct.py has 10 tests, each run on MySQL and PostgreSQL. They cover:

• DISTINCT and UNION ALL appear in the generated SQL only where needed.
• len() issues no DISTINCT for non-deduplicating queries.
• len() equals the number of fetched rows across restriction, anti-restriction, join, extend, proj, dj.Top, union and dj.U.
• dj.U operands inside joins, extend, aggr and dj.U(...).aggr.
• dj.U(...) & aggregation and dj.U(...) & union.

On master, 16 of the 20 runs fail. With this change all pass. The full suite passes locally on both backends; the only failures are the diagram-rendering tests, which need graphviz's dot program and fail identically on master.

Overlap with other v2.3.7 PRs

These PRs edit the same lines in expression.py. Whichever merges second needs small conflict fixes:

• fix(query): count and restrict expressions with an empty primary key #1605: adds an empty-key early return in __len__ and rewrites Aggregation.make_sql. Keep its GROUP BY/HAVING restructuring and use distinct = self._distinct.
• fix: dj.Top is honored when a limited expression is used in another query #1657: wraps _top operands in join, U.__and__ and U.aggr, next to the _distinct checks added here. It also adds a second UNION for a limited outer union, which should become UNION ALL as well.
• fix: aggr, extend and union keep unmatched rows when the right operand is restricted #1655: touches the join wrap rules.

Behavior change

• Joins with a dj.U(...) & expr operand return one row per key instead of repeated rows.
• aggr over such a group counts each key once.
• dj.U(...) & union returns distinct rows.

All three match the documented semantics.