Commit graph

12 commits

Author SHA1 Message Date
Zafir Stojanovski
49b1dbbcce
Fix misleading instruction in shortest_path asking for "length" instead of path (#523)
The prompt asked to "find the length of the shortest path" but the expected
answer is a sequence of directions. This caused models to answer with a number
instead of directions, degrading evaluation results.

Closes #522

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-25 13:02:23 +01:00
Zafir Stojanovski
ce0a6c4878
fix(envs): Add source dataset and index to metadata (#388)
* add source dataset and index to metadata

* fix typo

* fix coach class and its test
2025-03-20 11:12:14 +00:00
Oliver Stanley
7475a20700
include ranges rather than sampled values in difficulty metadata dicts (#387)
* update difficulty metadata for logic datasets

* update difficulty metadata for graph datasets

* update difficulty metadata for geometry datasets

* update difficulty metadata for games datasets

* update difficulty metadata for cognition datasets

* update difficulty metadata for arithmetic datasets

* update difficulty metadata for arc datasets

* update difficulty metadata for algorithmic datasets

* update difficulty metadata for algebra datasets

* use tuples

* update tests

* update tests
2025-03-20 10:27:03 +01:00
Andreas Köpf
d2c895f1d3
Refactor Curriculum Attributes (#335)
* remove min_value from AttributeDefinition
* remove type from AttributeDefinition
* Add CurriculumContext
* add ensure_interval option for RangeAttributes
* docs: Add legend explaining curriculum indicators in dataset gallery
* update GALLERY.md
2025-03-16 15:40:28 +01:00
Zafir Stojanovski
b915565c0d
add difficulty where possible (#274) 2025-03-07 19:01:26 +01:00
Andreas Köpf
c69bc5d4e6
Basic curriculum (#198)
* feat: Add optional curriculum support to dataset registration and creation
* docs: Add docstrings to create_curriculum() and register_dataset()
* feat: Add curriculum configuration classes for CurriculumExperiment
* feat: Add weight parameter to CurriculumAttributeConfig and use in DatasetSpec
* refactor: Simplify CurriculumAttributeConfig with "*" attribute level support
* test: Add unit tests for CurriculumExperiment class
* feat: Add from_yaml() method to CurriculumExperimentConfig with unit test
2025-03-07 11:22:12 +01:00
Zafir Stojanovski
f426db90ec
shortest path curriculum (#271) 2025-03-05 22:46:10 +01:00
Andreas Köpf
5d7fbac0ad
Minor question template & score_answer improvements (#261)
* math prompt improvements
* ignore brackets in complex_arithmetic results
* improve additional instruction in prompt of polynomial_equations
* more strict tests for score_answer in polynomial_equations
* simplify special reward handling
* fix test_intermediate_integration
* fix sokoban dataset
* add common dataset score_answer consistency test
2025-03-04 21:55:09 +01:00
Zafir Stojanovski
01e1c8f9af
fix: Unify Prompts (#254)
* remove cot
* fix prompt template
* fix pool matrix
* spiral matrix fixed
2025-03-03 21:55:53 +01:00
Andreas Koepf
45e452bff6 reduce size of default shortest_path maze grid 2025-02-23 22:27:17 +01:00
Zafir Stojanovski
c5f37d5e9f predict actual path 2025-02-23 18:24:23 +01:00
Zafir Stojanovski
97b3097984 shortest path 2025-02-23 11:25:00 +01:00