Repository navigation
Split the simulation runtime and its output builder tests under the file-length limit - #942
Merged
Merged
Conversation
2 tasks done
anth-volk
force-pushed
the
file-length-split-simulation-runtime
branch
4 times, most recently
from
October 27, 2026 16:52
75099c3 to
248b72f
Compare
simulation_runtime.py (1,111 lines) and tests/test_simulation_output_builder.py (3,048 lines) were over the file-length limits. The code moves unchanged. Three parts of simulation_runtime.py move out: - gcp_credentials.py: writing the worker's Google Cloud credentials; its messages still log under the simulation_runtime logger, now named explicitly - request_policy.py: the request's year and its policy periods - region_resolution.py: resolving a request's region, and checking a local-area breakdown request simulation_runtime.py keeps dataset selection and loading, simulation building, local-area breakdowns and the run itself (651 lines). Tests in several files replace those functions by name on simulation_runtime, and the code that calls them has to look them up there for the replacements to apply, so they stay together. Private names that another module now uses are public, with a docstring where they had none: _parse_year, _normalise_policy, _compile_interval_policy, _runtime_policy (as wrapper_policy, since _build_simulation has a local variable named runtime_policy), _resolve_region, _resolve_dataset_reference and _require_local_area_breakdown. simulation_runtime.py re-exports them, and the modules that imported the private names from it (segmented_national, spm, both precompute planners, Stage 12 aggregation and simulation, and worker validation) import the public names from it instead. The tests move to seven files and a module of shared fakes, all named tests/test_simulation_output_*.py, the files the weighted-frame rule (sim.microdata.builder-tests-use-weighted-frames) checks, so the entity frames they build stay checked. The shared fakes are now public. All 1,895 executor tests keep their names. Patches follow the code: the region group and SPM boundary tests patch resolve_dataset_reference in region_resolution, where regions are resolved; the identity fixture replaces get_country_release_bundle in both simulation_runtime and region_resolution, which both read it; and tests that patched or read a renamed helper use its new name, or the module a private helper now lives in. The file-length baseline drops both files (count 4 to 2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
anth-volk
force-pushed
the
file-length-split-simulation-runtime
branch
from
October 27, 2026 17:00
248b72f to
382e5dc
Compare
anth-volk
marked this pull request as ready for review
October 27, 2026 17:10
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #941
Same-Repository Draft PR Check
PolicyEngine/policyengine-sim-api, not a personal fork.Summary
Seventh of nine stacked PRs; it builds on #938. It splits the executor's
simulation_runtime.pyand its output builder tests without changing behaviour, and removes both from the file-length baseline (count 4 to 2). Scripts compared every top-level statement of each old file with the new files: all appear exactly once and unchanged apart from the renames below and one logger line, in their original order.simulation_runtime.py(1,111 lines)gcp_credentials.pyrequest_policy.pyregion_resolution.pyRegionResolution, resolving a request's region or region group, and checking a local-area breakdown requestsimulation_runtime.pysimulation_runtime.pystays at 651 lines, above the 400 the split aims for. Tests across the executor (the runtime, precompute, Stage 12, job-progress and segmented-national tests) replace_country_module,_load_dataset,_build_simulation,_resolve_dataset_selection,resolve_data_folder,SimulationOutputBuilderand others by name onsimulation_runtime. A replacement only reaches code that looks the name up in that module, and_load_datasetand_build_simulationthemselves call_country_moduleandresolve_data_folder, so moving any of these apart would silently stop some replacements applying. They stay together; the code moved out reads none of them.gcp_credentials.pynames thepolicyengine_simulation_executor.simulation_runtimelogger explicitly, so the credential messages keep their logger; this is the one statement that isn't verbatim.POLICYENGINE_SKIP_COUNTRY_IMPORTSis still set before any policyengine import.Renames
Private names another module now uses are public, each with a docstring where it had none:
_parse_yeartoparse_year,_normalise_policytonormalise_policy,_compile_interval_policytocompile_interval_policy,_runtime_policytowrapper_policy(its docstring says it converts a request's policy into the form the wrapper receives;_build_simulationalready has a local variable namedruntime_policy),_resolve_regiontoresolve_region,_resolve_dataset_referencetoresolve_dataset_reference, and_require_local_area_breakdowntorequire_local_area_breakdown.simulation_runtime.pyre-exports them. The modules that imported the private names from it now import the public names from it, so tests that replace them onsimulation_runtimestill reach those modules:src/modal/segmented_national.py,spm.py,precompute_planning.py,stage12_cache/precompute_planning.py,stage12_runtime/aggregation.py,stage12_runtime/simulation.pyandstage12_worker_validation.py.Tests:
tests/test_simulation_output_builder.py(3,048 lines)test_simulation_output_fakes.py(shared fakes, no tests)test_simulation_output_builder.pytest_simulation_output_impacts.pytest_simulation_output_runtime_runs.pytest_simulation_output_dataset_loading.pytest_simulation_output_dataset_selection.pytest_simulation_output_uk_local_areas.pytest_simulation_output_region_resolution.pytest_simulation_output_*.py, the patternsim.microdata.builder-tests-use-weighted-frameschecks, so the entity frames these tests and fakes build are still checked; the fakes module's docstring says so.test_simulation_output_uk_local_areas.pykeeps the routed-UK tests together with the fakes only they use._FakeOutputDataset,_FakeSimulation,_TrackedSpan,_TrackingRuntime,_with_observability_timings,_macro_baseline_reform,_congressional_district_output,_constituency_impact_record,_local_authority_impact_record,_stub_policyengine_output_callsand_build_schema_outputdrop their underscore, and_simulation_output_builderismake_output_builder.tests/test_region_group_resolution.py) and the SPM boundary helper (tests/test_spm_runtime_native.py) replaceresolve_dataset_referenceinregion_resolution, where regions are resolved;tests/test_precompute_planning.pyreplacesget_country_release_bundlein bothsimulation_runtimeandregion_resolution, which both read it, as the one replacement did before;simulation_runtime, under the new name where the helper was renamed (resolve_regionin seven tests and helpers);_normalize_credentials_blob(gcp_credentials),_parse_policy_period(request_policy),_normalise_region_codeand_resolve_region_group(region_resolution).Environment variables
No change:
setup_gcp_credentialsreads the sameGOOGLE_APPLICATION_CREDENTIALS_JSONandGOOGLE_APPLICATION_CREDENTIALSvariables in its new module, and no workflow, image or deployment file changes.Testing
python scripts/run_quality_guards.pyandpython -m scripts.rules guard --base-ref origin/file-length-split-precompute: passed;python -m scripts.rules renderleaves no diff.Only the lowest PR in the stack runs full CI, so this PR's
CI completecheck fails by design until the PRs below it merge.🤖 Generated with Claude Code