open-thought/reasoning-gym
1
0
Fork 0
mirror of https://github.com/open-thought/reasoning-gym.git synced 2026-04-19 12:58:07 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Generate eval config tool (#240)

Browse source 
* feat: Add generate_config.py script to create eval  configurations
This commit is contained in:
Andreas Köpf 2025-02-27 21:40:53 +01:00 • committed by GitHub
parent 850c1cf6f4
commit 5b8d1b5175
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
9 changed files with 858 additions and 338 deletions
Download patch file Download diff file Expand all files Collapse all files

16
eval/README.md
View file

@ -97,6 +97,22 @@ categories:
- dataset: "word_sorting"
```
### Generating Configurations
You can generate a configuration file with all registered datasets using the `generate_config.py` script:
```bash
python generate_config.py --output my_config.yaml --model "anthropic/claude-3.5-sonnet" --provider "Anthropic" --size 50 --seed 42
```
Options:
- `--output`: Output YAML file path (default: all_datasets.yaml)
- `--model`: Model name (default: openai/gpt-4)
- `--provider`: Provider name (default: None)
- `--size`: Default dataset size (default: 100)
- `--seed`: Default dataset seed (default: 42)
- `--include-params`: Include all configuration parameters (default: False)
### Running Evaluations
To run evaluations: