Make key_source restrict-only: add key_scope, deprecate parent-redefining overrides #1523

Summary

Make key_source a restrict-only surface. Today key_source can be overridden freely, including in ways that redefine the parent join — the source of subtle, hard-to-diagnose bugs. In 2.4, overriding key_source raises an error, and a new property key_scope becomes the only supported way to narrow the source, by restriction.

Background

key_source defaults to the join of the tables referenced by the foreign keys in a computed/imported table's primary key (Autopopulate → Key Source Calculation). populate() iterates it, passing each key to make(key).

The default is almost always correct. The only legitimate reason to change it is to restrict it — compute a subset of the default source (e.g. a completion barrier that fires only when every member of a batch has results). Restricting keeps the parents and the grain intact; each make(key) still receives a fully-formed primary key.

Overriding key_source to redefine the parent join — dropping a parent, adding one, or rebuilding the key from an aggregate — is a different and dangerous operation: it can change or drop primary-key attributes, so make(key) receives a key missing part of its identity or carrying attributes the table's primary key doesn't declare. populate's pending-work bookkeeping (key_source minus rows already present) then breaks too: a key coarser than the primary key is marked done as soon as one row for it exists, so later members never compute. The downstream effects — partial inserts, mismatched keys, silent miscomputation — are hard to reason about, and the framework was not designed for it.

Proposal

1. Add key_scope — a declared restriction condition applied to the default key_source. It holds the restriction itself, not a query:

   • a string: key_scope = 'quality = "good"'
   • a mapping: key_scope = {"quality": "good"}
   • several conditions: key_scope = dj.AndList('quality = "good"', 'session_date > "2026-01-01"')
   • a restriction by another table (a dependency visible in the source but not necessarily a formal FK edge).

   The framework computes the source as default_key_source & key_scope. The parents and grain are never changed, so every make(key) is fully keyed. (The name replaces the earlier working name key_source_restriction: shorter, and it doesn't point users back at the internal they should stop touching.)

2. Make key_source restrict-only. Treat key_source as an internal of the auto-populate machinery. In 2.4, a subclass that overrides key_source raises a DataJointError directing it to key_scope.

3. Batching is a master–part concern, not a key_source concern. A run- or cohort-level batch should be a master keyed on the batch with per-member results on a part — consistent with the DataJoint 2 rule that an auto-populated table introduces no new dimension (only its parts may). This removes the historical "batch by overriding key_source" motivation.

Rollout

• 2.4: add key_scope; overriding key_source raises an error.
• Docs: remove any guidance that recommends overriding key_source; document key_scope as the supported way to narrow a source, and "batch via master–part."

Docs task (coordinated)

datajoint-docs: pull any recommendation to override key_source, and document key_scope + the "batch via master–part" guidance (Autopopulate spec §2 and the how-to pages).

cc @MilagrosMarin @ttngu207