Skip to content

Add Launa CSM-only evaluation results - #232

Open
zomux wants to merge 7 commits into
ServiceNow:mainfrom
zomux:codex/launa-csm-results
Open

zomux wants to merge 7 commits into
ServiceNow:mainfrom
zomux:codex/launa-csm-results

Conversation

@zomux

@zomux zomux commented Sep 21, 2026 •

Copy link
Copy Markdown

Summary

Adds Launa to the clean English CSM results as a 2-part system. ITSM, HR, pooled scores, and perturbation results remain unavailable.

This is a draft submission for review of partial-domain coverage, incomplete-trial handling, and evaluation-version compatibility.

The only changed file is website/src/data/leaderboardStats.json. Evaluation result is documented below; supporting evidence is retained locally and can be provided on request.

Project website: Launa

Results

Metric EVA-A EVA-X Scenarios
pass@1 48.70% 55.03% 50
Mean 64.19% 75.36% 50
pass@5 80.43% 95.65% 46
Estimated pass^5 29.09% 18.14% 46

There are 250 recorded trials across 50 scenarios. 244 trials passed simulation validation; six invalid trials affect four scenarios. Pass@1 and means first average valid trials within each scenario, then weight scenarios equally. Five-trial metrics use the 46 complete scenarios. Valid model failures are retained.

The original saved report averages valid trials directly (49.59% EVA-A / 55.33% EVA-X pass@1). This PR uses scenario weighting so point estimates and confidence intervals have the same sampling unit. All intervals are recomputed using EVA's bootstrap implementation: 2,000 scenario resamples, 95% percentile intervals, deterministic run-derived seed. Estimated pass^5 follows the website's per-scenario success-probability definition.

Maintainer guidance requested

Is a CSM-only entry acceptable with other domains and pooled results unavailable?

Thank you for maintaining EVA!

@zomux
zomux marked this pull request as ready for review September 21, 2026 23:15
@fanny-riols

Copy link
Copy Markdown
Collaborator

Hi @zomux , thank you for your interest! We would actually prefer having submissions on the 3 data domains. Also, after some analysis, we concluded that we could use 3 trials only, so no need to run 5 trials. Finally, once you have all results (on the 3 domains), could you send us your output artifacts (all the output files created) so that we could do some sanity checks on your configs, etc., please?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants