Repository navigation
Conversation
zomux
marked this pull request as ready for review
September 21, 2026 23:15
Collaborator
|
Hi @zomux , thank you for your interest! We would actually prefer having submissions on the 3 data domains. Also, after some analysis, we concluded that we could use 3 trials only, so no need to run 5 trials. Finally, once you have all results (on the 3 domains), could you send us your output artifacts (all the output files created) so that we could do some sanity checks on your configs, etc., please? |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds Launa to the clean English CSM results as a
2-partsystem. ITSM, HR, pooled scores, and perturbation results remain unavailable.This is a draft submission for review of partial-domain coverage, incomplete-trial handling, and evaluation-version compatibility.
The only changed file is
website/src/data/leaderboardStats.json. Evaluation result is documented below; supporting evidence is retained locally and can be provided on request.Project website: Launa
Results
There are 250 recorded trials across 50 scenarios. 244 trials passed simulation validation; six invalid trials affect four scenarios. Pass@1 and means first average valid trials within each scenario, then weight scenarios equally. Five-trial metrics use the 46 complete scenarios. Valid model failures are retained.
The original saved report averages valid trials directly (49.59% EVA-A / 55.33% EVA-X pass@1). This PR uses scenario weighting so point estimates and confidence intervals have the same sampling unit. All intervals are recomputed using EVA's bootstrap implementation: 2,000 scenario resamples, 95% percentile intervals, deterministic run-derived seed. Estimated pass^5 follows the website's per-scenario success-probability definition.
Maintainer guidance requested
Is a CSM-only entry acceptable with other domains and pooled results unavailable?
Thank you for maintaining EVA!