Zafir Stojanovski
17a8431013
rename to easy and hard ( #445 )
2025-06-02 10:34:05 +02:00
Zafir Stojanovski
6614338ecc
add numbers to performance heatmap ( #442 )
2025-05-30 18:39:13 +02:00
Zafir Stojanovski
b843f33b1d
fix(eval): comparison plot ( #441 )
...
* heatmap
* filter comparison plots
* latex style
* curriculum heatmap
* pre-commit
* update figsize
* large y-ticks
* larger font
* thinner
* include 50
2025-05-29 12:31:07 +02:00
Zafir Stojanovski
93e731c29c
heatmap ( #438 )
2025-05-19 09:07:45 +01:00
Zafir Stojanovski
5961a10145
comparison plot ( #436 )
2025-05-18 22:57:49 +01:00
Zafir Stojanovski
4cab1c3e6d
comparison plots ( #431 )
2025-05-12 15:41:58 +02:00
Zafir Stojanovski
98e976642d
gemini flash and o3 mini configs ( #425 )
2025-04-20 21:24:52 +02:00
Adefioye
169d8c3aec
Evals: Sonnet-3.7-eval partial results ( #423 )
...
* Add results for sonnet and config
* Make some cleanup
2025-04-18 10:31:40 +02:00
Zafir Stojanovski
290bfc4fdd
(evals): Medium configs ( #415 )
...
* updated medium configs
* fix problematic curriculum values / small issues causing exceptions to be raised
* optimus alpha config
* all configs so far
* fix tests
2025-04-14 08:25:31 +02:00
Andreas Köpf
cd1a9ea58b
add llama-4-maverick eval yaml config ( #413 )
2025-04-06 21:10:00 +01:00
Zafir Stojanovski
dced3bfc45
fix(curriculum): Make boundaries in curriculum more sensible ( #407 )
...
* init
* fix tests
* unify codeio
* filtered for libraries not present in reasoning-gym
* fix more bounds
* puzzle24
* knight swap curriculum
* fix number sorting
* fix attributes
* add validation of config in creation of dataset
* dry run for instantiating and validating the datasets
* remove unused imports
* fix curriculum tests to reference newly updated attribute names
2025-04-04 20:24:14 +02:00
Zafir Stojanovski
8c45571a48
visualize heatmap sorted by overall performance ( #397 )
2025-04-01 00:08:39 +02:00
Andreas Köpf
4e7d9296ee
feat: Add comprehensive visualization script for evaluation results ( #376 )
2025-03-16 12:26:27 +01:00
Andreas Köpf
d6f399b8e4
Add eval configs, small fixes to eval script & rush-hour score_answer
2025-03-16 09:18:05 +01:00
Andreas Koepf
27cdd25548
add gemma-3-27b & qwq-32b configs
2025-03-15 20:47:51 +01:00
Andreas Koepf (aider)
f373881c06
feat: Add fallback to first non-None model answer when best_answer is unset
2025-03-15 16:52:50 +01:00
Andreas Köpf
bfa5f8078b
Eval N completions per prompt ( #374 )
...
* feat: Add support for generating multiple completions per prompt
* feat: Track best and mean scores for multiple completions per prompt
* feat: Add checkpoint and resume functionality to evaluation script
2025-03-15 16:39:36 +01:00
Andreas Köpf
c8e77d21a7
fix: Improve error logging and preserve full model response in eval process ( #337 )
2025-03-12 00:01:49 +01:00
Andreas Koepf
770255b608
fix pre-commit
2025-03-11 08:18:55 +01:00
joesharratt1229
105374183f
Algebra/curr ( #320 )
...
* add polynomial equation curriculum
* added simple integration
* addded metadata to config
2025-03-11 00:17:07 +01:00
Andreas Koepf (aider)
d0b49cfffd
feat: Add --category option to evaluate datasets from a specific category
2025-03-11 00:00:38 +01:00
Andreas Koepf
4109b5b72c
update eval yaml config files
2025-03-10 00:48:32 +01:00
vncntt
3672b231f1
should exit if API key isn't defined ( #259 )
...
* should exit if open-router and no api key
2025-03-04 09:45:36 +01:00
joesharratt1229
6770ee3eef
updated for config by dataset ( #257 )
...
* updated for config by dataset
* updated read me
2025-03-03 21:58:32 +01:00
Andreas Köpf
a66a7e7965
Revert "log error message on bad api response ( #243 )" ( #249 )
...
This reverts commit 8e2089b6c0
2025-03-01 23:56:42 +01:00
Andreas Köpf
4ad9d22fa3
Add base_url and api_key command line args for eval.py script ( #244 )
...
* feat: Add base URL command line parameter to eval.py script
* feat: Add API key parameter and CLI option to AsyncModelEvaluator
2025-02-28 18:32:58 +01:00
Rich Jones
8e2089b6c0
log error message on bad api response ( #243 )
2025-02-28 15:32:27 +01:00
Andreas Köpf
b4207162ff
Eval sampling settings for generation (temperature, top-p, max_tokens) ( #242 )
...
* feat: Add sampling parameters to eval configuration and API call
* feat: Add support for system_prompt_id and optional system_prompt configuration
2025-02-28 11:48:37 +01:00
Andreas Koepf (aider)
24a4b7a4c8
feat: Add system prompt to dataset results and summary output
2025-02-28 00:26:06 +01:00
Andreas Köpf
5b8d1b5175
Generate eval config tool ( #240 )
...
* feat: Add generate_config.py script to create eval configurations
2025-02-27 21:40:53 +01:00
Andreas Köpf
850c1cf6f4
Eval script consolidation ( #238 )
...
The script now supports:
- YAML and JSON configurations
- Dataset-specific parameters
- Overriding configuration via command line
- Detailed logging and error handling
2025-02-27 17:39:14 +01:00
Andreas Koepf
477e1f85cc
verify that OPENROUTER_API_KEY env var is set
2025-02-26 22:15:30 +01:00
Andreas Koepf
acb2d7eb53
add llama-3.3-70b-instruct eval yaml files
2025-02-26 20:54:07 +01:00
Andreas Köpf
5b89a3a2d0
Merge pull request #217 from open-thought/feat/o3-mini-eun
...
added o3 mini yaml rconfiguration
2025-02-26 09:38:11 +01:00
vncntt
29179f783e
fix sonnet eval_dir ( #216 )
...
* fix eval_dir
* add logging
2025-02-26 09:37:09 +01:00
joesharratt1229
7d7e44d1af
added o3 mini yaml
2025-02-26 08:09:12 +00:00
Andreas Koepf
6d5168d1e5
add llama-3.3-70b-instruct algebra, algorithmic eval configs
2025-02-25 23:43:29 +01:00
Andreas Koepf
791f16ec0f
use results folder name for eval results
2025-02-25 19:41:21 +01:00
joesharratt1229
ffe60ef112
finalised readme
2025-02-25 18:14:39 +00:00
joesharratt1229
56cc111ab3
Merge remote-tracking branch 'origin/consolidate_eval_script' into fix/eval
2025-02-25 18:10:07 +00:00
joesharratt1229
9ac6ea4eb2
changed structure
2025-02-25 16:32:42 +00:00
joesharratt1229
52c3c430b9
updated config and read me
2025-02-25 16:25:16 +00:00
joesharratt1229
7b39f4a3c7
updated read me
2025-02-25 15:51:31 +00:00
joesharratt1229
046c46c0bb
updated read me
2025-02-25 15:46:43 +00:00
Andreas Koepf
878f9bbc76
move r1 configs into r1 yaml/r1 subfolder
2025-02-25 16:24:30 +01:00
Andreas Koepf
e7ae82a831
consolidate eval scripts to have single eval.py
2025-02-25 16:13:22 +01:00
Andreas Koepf
8291956554
add aiohttp & tenacity deps to requirements-eval.txt
2025-02-25 15:50:11 +01:00
Andreas Koepf (aider)
e48c1f82cd
docs: Update installation instructions in eval README
2025-02-25 15:37:09 +01:00
Andreas Koepf (aider)
a1b0a0414e
docs: Add dependency installation step to eval README setup instructions
2025-02-25 15:19:38 +01:00
Andreas Koepf
574edb5c5b
remove eval results from main repo
2025-02-25 11:02:02 +01:00
