open-thought/reasoning-gym
1
0
Fork 0
mirror of https://github.com/open-thought/reasoning-gym.git synced 2026-04-19 12:58:07 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions

Add base_url and api_key command line args for eval.py script (#244)

Browse source 
* feat: Add base URL command line parameter to eval.py script
* feat: Add API key parameter and CLI option to AsyncModelEvaluator
This commit is contained in:
Andreas Köpf 2025-02-28 18:32:58 +01:00 • committed by GitHub
parent 8e2089b6c0
commit 4ad9d22fa3
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
2 changed files with 53 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

23
eval/README.md
View file

@ -28,10 +28,19 @@ pip install -e ..
pip install -r requirements-eval.txt
```
3. Set your OpenRouter API key as an environment variable:
```bash
export OPENROUTER_API_KEY=your-api-key
```
3. Set your API key (if required by the API):
For OpenRouter, you can set it as an environment variable:
```bash
export OPENROUTER_API_KEY=your-api-key
```
Or provide it directly when running the script:
```bash
python eval.py --config your_config.yaml --api-key your-api-key
```
Note: API key is optional for some APIs (e.g., local deployments).
4. Prepare your evaluation configuration in YAML or JSON format (see example in `example_config.yaml`):
@ -132,6 +141,12 @@ For example:
python eval.py --config example_config.yaml --full-results
```
You can specify a different API base URL if needed:
```bash
python eval.py --config example_config.yaml --base-url "https://api.together.xyz/v1" --api-key "your-together-api-key"
```
The results will be stored in a directory named after the model and timestamp, containing:
- `summary.json` - Summary of all results