[FabianHofmann]

Closes #1049. Part of #1019 (step 5, the last step).

Note

The following content was generated by AI.

Changes proposed in this Pull Request

Both entry points are marked as evolving: they emit linopy.EvolvingAPIWarning once per session (message prefix long-form:), as their signature and input format may still change in minor releases. The once-per-key helper moved from piecewise.py to constants.py and is shared.

linopy.from_long(frame, coords, default=None) -> DataArray. Converts a long table (polars or pandas frame) to a DataArray, eagerly at the call site. Columns named after a dimension of coords are the key columns; exactly one other column must remain and is the value column, the result is named after it. coords is required and defines the cube, e.g. coords=var.coords: cells not listed raise unless default= fills them, dims of coords without a key column are left to linopy's usual broadcasting. The result is an ordinary DataArray, so it works everywhere a constant does (bounds, coefficients, arithmetic, rhs, penalty) and passes the strict 1.0 alignment unchanged; nothing is added to ConstantLike or as_dataarray.

p_nom = linopy.from_long(generators[["gen", "p_nom"]], coords=[gens])
load = linopy.from_long(demand, coords=[buses, snapshots])
p_max_pu = linopy.from_long(availability, coords=p.coords, default=1.0)

Strict schema, fail fast: no key column, zero or several value columns (so a misspelled key column raises), null keys, non-numeric or null values, duplicate keys, keys outside the coords, coords with duplicate labels, a key dtype family (numeric / datetime / date / duration / string, pandas categoricals by their categories) or time zone differing from the coord, lossy key casts, and pandas frames with a non-default index raise. pl.LazyFrame, Series and MultiIndex forms are not accepted.

Pivot: polars keys are mapped to positions by a left join against each coord (arithmetic for a RangeIndex), combined into a flat index in polars, checked for duplicates and gaps with n_unique, then scattered into one np.full buffer; a full, sorted table is reshaped without the buffer. pandas input uses Index.get_indexer (no pyarrow dependency). The key-to-position step is one private helper shared with Relation.

linopy.Relation(frame, member, group, groups=None) for relation group-by. expr.groupby(Relation(...)).sum() sums over a mapping with repeated member keys, e.g. (link, port) → bus. member names one or several key columns, each a dimension of the expression; the frame has exactly the member and group columns. The sum is a composition of two existing operations: a vectorised isel with the member positions of the relation rows, sorted by group, then the row-dim groupby(pd.Series).sum(). No store changes are needed for it, csr.py and store.py are back to master apart from a sorted fast path in coo_to_csr. groups= lists the labels the result spans: groups with no member get a zero cell, rows with a group outside groups raise. Relation.group_of(coords) is the adjoint, the group of each member cell as a DataArray, so price.sel(bus=bus_of.group_of(gens)) spreads nodal prices onto generators; gaps and duplicate members raise. A bare frame passed to groupby raises and points to Relation. Quadratic expressions raise NotImplementedError.

Vectorised selection keeps the sparse store: isel, sel, __getitem__ with DataArray indexers over a new indexed dimension, and where, no longer fall back to dense on Model(sparse=True). The composition costs more than a dedicated scatter on CSR (20k links × 500 timesteps, 2 rows per link: 0.89 s / 1142 MB against 0.48 s / 819 MB), accepted for the ~60 deleted store lines and the general sparse selection it buys.

A tutorial notebook, examples/long-form-inputs.ipynb, builds a small dispatch model from long-form data with both tools.

Design notes

A first draft used a lazy linopy.Table constant pivoted once target coords were known; it leaked into ~40 ConstantLike call sites and converted polars through Python lists and a cube-sized MultiIndex. It was replaced by the eager function.

An explicit value argument was dropped in favour of inferring the single non-key column. Inferring the coords from the data was rejected: absent rows would shrink the cube, label order would depend on the data, default would lose its meaning and dtype checks their reference (xr.DataArray.from_series already covers that). Wide frames returning a Dataset were rejected: a misspelled numeric key column would silently become a value column, and ConstantLike has no Dataset.

Benchmark: long table → 10k × 2920 cube

rows (fill)	keys	lazy Table (first draft)	from_long, polars	from_long, pandas
1.93M (7 %)	int	4.1 s	0.35 s	0.35 s
2.78M (10 %)	str + datetime	6.9 s	0.39 s	0.57 s
29.2M (100 %)	str + datetime	18.5 s	0.87 s	2.40 s

Review and tests

Branch review (four reviewers plus a coordinator) on the first draft found one must-fix (quadratic relation group-by crashed), fixed. Not addressed: sparse quadratic group-by with a plain Series already fails on master; a cube too large for np.prod gives an unclear IndexError.

Full suite: 5048 passed, 84 skipped. mypy ., ruff check ., ruff format --check . and the doctests pass.

Checklist

• AI-generated content is marked (see AGENTS.md).
• Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
• Unit tests for new features were added (if applicable).
• A note for the release notes doc/release_notes.rst of the upcoming release is included.
• I consent to the release of this PR's code under the MIT license.

[codspeed]

Merging this PR will not alter performance

✅ 181 untouched benchmarks
⏩ 181 skipped benchmarks¹

---

_{Comparing store/5-long-form (74f6929) with master (51c4977)}

Footnotes

1. 181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩