Repository navigation
Conversation
`simulate.py --check` compares the AMICI simulations at the nominal parameters against `v1/simulations.tsv` and exits with status 1 if they differ (default rtol 1e-3, atol 1e-6, configurable with --rtol/--atol). Rows are matched by observable, conditions and time point (to 6 significant digits), replicates by their order; observable IDs of a flattened problem in the table are mapped back to the original ones. `--output` writes the simulations as PEtab simulation table. The Simulate workflow runs the check for every problem, also on pull requests that touch the problems, and fails on any mismatch, except for the problems in KNOWN_FAILURES, whose tables do not match yet. A known failure that passes fails the job too, so that the list is kept current. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`simulate.py --check-tolerances` simulates a second time with rtol and atol divided by 100 and fails if the simulations (relative to the largest absolute simulation of the observable) or the log-likelihood change by more than 1e-5, 100 times below the tolerance of the comparison to the simulation tables. Failed simulations are reported instead of raising. The solver settings (rtol, atol, max_steps) are read from `solver_settings.yaml`: the AMICI defaults, and stricter settings for the problems that do not pass the check with them (Borghans, Crauste, Isensee, Lang, Raimundez, Zheng); Chen gets a larger atol, with which the check at stricter tolerances does not fail. With these settings, the simulations of Crauste, Lang and Raimundez agree with their simulation tables; they are removed from KNOWN_FAILURES. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI only simulates the problems with the solver settings of `solver_settings.yaml` and compares the simulations to the simulation tables; `--check-tolerances` remains a local tool to choose the settings. Froehlich_CellSystems2018 needs rtol 1e-10 (change 2.5e-4 at the default, 2.5e-6 at rtol 1e-10). Simulating all of its 9169 conditions at that tolerance takes about 20 minutes on 4 cores, so a new optional setting, `conditions`, restricts the simulation to evenly spaced conditions (1000 for Froehlich, about 2 minutes) and the comparison to their rows of the simulation table. `--all-conditions` simulates all. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
COMPARISON_RTOL (1e-3) and COMPARISON_ATOL (1e-6) replace the --rtol and --atol options of simulate.py. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
to --check-simulations and --check-simulation-tolerances. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ables The simulation tables had rows for data points that are not in the measurement tables; the rows that correspond to measurements agree with the simulations: - Blasi_CellSystems2016: 6 surplus steady-state replicates each of observable_k5k12, observable_k8k12 and observable_k8k16, and 18 rows of observable_k5k16, which has no measurements (36 rows). - Zhao_QuantBiol2020: time points of observable_confirmed_infected after the measured time points of each condition (87 rows). Both are removed from KNOWN_FAILURES. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The simulations of these four problems were not the model at the nominal parameters of the parameter table (Fujita_SciSignal2010, Sneyd_PNAS2002, Perelson_Science1996) or were the measurements (Zheng_PNAS2012, #181). They are simulated again at the nominal parameters, row by row in the order of the measurements; the simulations of Fujita_SciSignal2010 get the current condition ids of the measurements (condition_step_*), the table still named the former ones (model1_data*).
The table differed from the model at the nominal parameters in 5 of 120 simulations by up to 0.28 % (AKT_PP of model1_data4 at t = 1950: 17.7718 instead of 17.7228). The simulations are the ones of AMICI 1.2.0 (simulate_petab, absolute and relative tolerance 1e-16 and 1e-12), which sbmlsim reproduces to 4e-6; their log-likelihood is -130591.753.
Chen_MSB2009, Fujita_SciSignal2010, Perelson_Science1996, Sneyd_PNAS2002 and Zheng_PNAS2012 reproduce their regenerated simulation tables. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The measurement tables with the measurements replaced by the AMICI simulations at the nominal parameters (simulate_petab, rtol 1e-12, atol 1e-16), which change by less than 1e-8 (relative) from rtol 1e-10. Log-likelihoods: Okuonghae_ChaosSolitonsFractals2020 -470490.606, Oliveira_NatCommun2021 -9623396.411. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The table had the column names simulationCondition and preequilibrationCondition instead of simulationConditionId and preequilibrationConditionId, observable IDs without the prefix observable_ of the observable table, empty noiseParameters, and lacked the columns observableTransformation, noiseDistribution and datasetId of the measurement table. Its values agree with the new ones to 5e-6 (relative). The new table is the measurement table with the measurements replaced by the AMICI simulations at the nominal parameters (simulate_petab, rtol 1e-12, atol 1e-16), which change by about 1e-9 (relative) from rtol 1e-10; log-likelihood -346.090. Raia_CancerResearch2011 is removed from KNOWN_FAILURES. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Released AMICI instead of a development version. The development version, like AMICI 0.34.2 and 1.0.0, ignores the condition table value of a parameter with an initial assignment where the parameter is used in another initial assignment: the virus dose u_2 of Liu_IFACPapersOnLine2025 (initial assignment u_2 = 1e9, condition ctrl u_2 = 0) then also enters the initial virus V = u_2 of the control condition. With AMICI 1.2.0, Liu_IFACPapersOnLine2025 reproduces its simulation table, with the log-likelihood of the paper (-723.357), and is removed from KNOWN_FAILURES, which is now empty. simulate.py imports simulate_petab, rdatas_to_simulation_df, LLH and RDATAS from their location in AMICI 1.2.0. All problems reproduce their simulation tables and pass the tolerance check with AMICI 1.2.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
simulate.py --check-simulations fails for a problem without a simulation table instead of skipping it, so that new problems need one; all problems have one. CONTRIBUTING.md describes the simulation table, how to generate it (simulate.py --output) and how it is checked. --output always simulates all conditions, also for problems with a subset of the conditions in solver_settings.yaml. Problems without a PEtab v1 encoding are skipped (simulate.py does not support PEtab v2 yet), unknown problem IDs are an error. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Select the rows of a subset of conditions with one helper, rows_in_conditions, for both the simulated problem and the simulation table; it normalizes missing condition IDs to "" as petab.v1.get_simulation_conditions does, so rows without a preequilibration condition are no longer dropped when other rows have one. - Filter the simulation table to the subset only with --check-simulations. - Check that the simulation table has the columns time and simulation. - Treat an empty problem entry in solver_settings.yaml as no overrides. - Run the workflow also for pull requests that change solver_settings.yaml or the benchmark_models_petab package. - Accept any whitespace, also newlines, between the IDs of KNOWN_FAILURES. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- KNOWN_FAILURES in the Simulate workflow: it is empty, as all problems reproduce their simulation tables; a mismatch now always fails the job. - The rtol and atol parameters of compare_simulations, which were always the hardcoded COMPARISON_RTOL and COMPARISON_ATOL. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 200 evenly spaced conditions cover all 143 cell lines and all drugs, like 1000; they take about 22 instead of 120 seconds on 4 cores. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add a new CI task to reproduce simulation tables.
simulate.pyis adapted to perform this check.Solver settings were chosen in a problem-specific manner and are stored in a YAML.
Only 1000/9169 Froehlich conditions are checked in the CI, to speed it up.
The following changes were made to pre-existing simulation tables.
This requires AMICI >=1.2.0 -- earlier versions simulated Liu_IFACPapersOnLine2025 incorrectly.
No PEtab v2 tables are checked yet.
Supersedes #343. Closes #181.
🤖 Generated with Claude Code