Skip to content

refactor: per-case store fallbacks on a copy, merge_stores (#1019 step 2b) - #1052

Merged
FabianHofmann merged 5 commits into
masterfrom
store/2b-fallbacks
Oct 23, 2026
Merged

FabianHofmann merged 5 commits into
masterfrom
store/2b-fallbacks

Conversation

@FabianHofmann

@FabianHofmann FabianHofmann commented Oct 22, 2026 •

Copy link
Copy Markdown
Collaborator

Note

The following content was generated by AI.

Stack 3/3, #1050 and #1051 merged; now based on master. Part 2 of 2 for #1020.

Closes #1020.

Changes proposed in this Pull Request

Step 2b of the store refactoring plan in #1019: a fallback never converts the receiver, and the frontend calls the store.

  • LinearExpression._fallback(reason) converts a copy with a named reason in op(case) form, for example reindex(method=...) or where(non-scalar other). The receiver keeps its store. .data returns a converted copy under the reason data(read).
  • _densify, _densify_all, _try_csr_merge, _aligned and _concatenated are deleted. linopy.store gains merge_stores, align_stores and concat_stores, which return a store or a reason. Any sparse operand pulls a merge sparse; a reason falls the whole merge back to dense on copies.
  • Dense operations call DenseStore; the inline dense bodies in the expression classes are gone. DenseStore and CSRStore gain combine, absorbed and copy.
  • A named sparse expression keeps its name and its sparse backing through Model.copy, netcdf writing and vars, coeffs or const assignment, via a new _attrs slot.
  • A product with a constant expression stays sparse; only a true quadratic product converts, under mul(quadratic product).
  • LinearExpression.where and QuadraticExpression.where take cond, other and drop only, matching xarray. The extra keyword arguments are gone.
  • Sparse sum with an unknown dimension raises KeyError without densifying first.
  • Conformance suite: every fallback row asserts the receiver stays sparse; a scan over the source collects every op(case) reason and asserts each has a row; new rows for products with constants, where with new dimensions, the override merge, getitem and division over a new dimension, and assignment(coeffs).

Deferred to step 3 (#1045): sel, isel, __getitem__, shift, roll, where, drop_vars and dense reindex still use the dense xarray code, because the grid rejects 2-D auxiliary coordinates and the grid path changes a coordinate dtype. A named sparse expression still reads back dense from netcdf until step 4c (#1048).

Deferred cleanups from review: _off_grid duplicates the DenseStore.grid check; the dense constant-combine exists in DenseStore.combine and in the base class for quadratic expressions; _printout_cell could call store.cell; the dense branches of flat, to_polars and linear_terms repeat DenseStore.terms but differ on absent cells; Variable.where still takes extra keyword arguments.

Checklist

  • AI-generated content is marked (see AGENTS.md).
  • Code changes are sufficiently documented; i.e. new functions contain docstrings and further explanations may be given in doc.
  • Unit tests for new features were added (if applicable).
  • A note for the release notes doc/release_notes.rst of the upcoming release is included.
  • I consent to the release of this PR's code under the MIT license.

…ensify_all (#1020, part 2)

A fallback converts a copy with a named reason and never the receiver.
Dense operations call DenseStore; merge/align/concat go through
linopy.store. Named sparse expressions stay sparse.
@codspeed

codspeed Bot commented Oct 23, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 49.11%

⚡ 1 improved benchmark
✅ 180 untouched benchmarks
⏩ 181 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ test_op[expr_mul_scalar] 281.3 KB 188.6 KB +49.11%

Tip

Curious why performance improved? Comment @codspeedbot explain why performance improved on this PR, or directly use the CodSpeed MCP with your agent.


Comparing store/2b-fallbacks (0c95576) with master (95f6bfa)2

Open in CodSpeed

Footnotes

  1. 181 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on master (5bb19bb) during the generation of this report, so 95f6bfa was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

Reassign the term-dim const and the factor-dim coeffs and const to their
reduced form so the un-reduced concat results do not stay alive during
absorb_absence; restores the peak memory of the dense merge to master's
(CodSpeed var_mul_var, expr_mul_var, expr_add_masked).
@FabianHofmann
FabianHofmann marked this pull request as ready for review October 23, 2026 07:26
@FabianHofmann
FabianHofmann merged commit 9a67427 into master Oct 23, 2026
22 checks passed
@FabianHofmann
FabianHofmann deleted the store/2b-fallbacks branch October 23, 2026 09:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Store protocol for LinearExpression: DenseStore and CSRStore behind one interface, fallback in the base class

1 participant