reasoning-gym/tests
Andreas Köpf 52b44c47d5 reasoning-gym-server & cli tool (#154)
* feat: Add initial server structure with configuration, registry, and middleware

* feat: Add chain_sum dataset to experiment registry test

* fix: Update test_registry to use DatasetSpec for composite config validation

* refactor: Update Pydantic config to use json_schema_extra and ConfigDict

* feat: Add Pydantic models for API request/response data

* feat: Implement basic experiment management endpoints with tests

* feat: Implement composite configuration endpoints for experiments

* fix: Add missing DatasetConfigUpdate import in server.py

* refactor: Update dataset config update method to properly merge config updates

* fix: Correctly retrieve current dataset config in composite endpoint

* feat: Add basic CLI structure with experiments and config commands

* feat: Add initial CLI tool with basic experiment management commands

* refactor: Reorganize CLI package structure and fix import paths

* refactor: Implement initial CLI commands for experiment management

* feat: Implement HTTP client for Reasoning Gym server in RGC CLI tool

* fix: Move print statements inside try block to resolve SyntaxError

* fix: Resolve SyntaxError in edit_config function by adding missing except block

* feat: Add default app instance in server module for easier uvicorn startup

* docs: Add README.md with server and RGC tool documentation

* remove unused files

* refactor: Remove unsupported type annotation in registry.py

* refactor: Move ExperimentRegistry to coaching module and add Experiment class

* fix: Add missing CompositeDataset import in test_registry.py

* refactor: Implement lazy ASGI app creation for server initialization

* feat: Add health check command to RGC CLI for server connection

* feat: Add version tracking support to CompositeDataset

* feat: Add DatasetVersionManager for tracking dataset versions

* feat: Add entry_id metadata and score_answer_with_id method to CompositeDataset

* feat: Add entry_id metadata combining version and index

* fix: Resolve undefined variable by storing version_id before use

* test: Add comprehensive unit tests for score_answer_with_id() function

* test: Add comprehensive version tracking test for dataset config updates

* feat: Validate dataset weights are positive in CompositeDataset initialization

* feat: Add weight update and normalization methods to CompositeDataset

* refactor: Centralize weight normalization in CompositeDataset and allow zero-weight datasets

* feat: Add negative weight validation to CompositeDataset constructor

* feat: Add duplicate dataset name check in CompositeDataset and update test

* refactor: Move duplicate dataset name check inside dataset iteration loop

* refactor: Update CompositeDataset weight management to use config as source of truth

* refactor: Move duplicate dataset name check to CompositeConfig.validate()

* test: Update composite dataset weight test assertions and validation

* feat: Add methods to add and remove datasets in CompositeDataset

* refactor: Remove weight normalization and use unnormalized weights directly

* refactor: Remove redundant total weight check in update_dataset_weights

* feat: Add batch generation and scoring endpoints to server

* fix: Import BatchEntry in server.py to resolve undefined name error

* refactor: Update ReasoningGymDataset to use server for batch generation and scoring

* fix: Add missing List and Dict type imports

* feat: Add get_batch() and score_outputs() methods to RGClient

* test: Add unit tests for generate_batch and score_outputs endpoints

* refactor: Add DatasetVersionManager to Experiment class and CompositeDataset constructor

* feat: Add validation for base_index and batch_size in generate_batch endpoint

* refactor: Remove unused BatchRequest type from imports

* refactor: Convert models to use Pydantic exclusively

* test: Update scoring endpoint tests to use correct request model format

* refactor: Rename ScoreItem to AnswerItem and update related code

* feat: Update scoring endpoint to return ordered ScoringResponse with scores and entry_ids

* fix: Add missing ScoringResponse import in server.py

* move verl ppo sample with server into own file

* refactor: Use Pydantic models for get_batch() and score_outputs() in RGClient

* refactor: Update client methods to use Pydantic models for type safety

* refactor: Use Pydantic models for experiment and dataset config operations

* refactor: Clean up duplicate methods and improve error handling in main.py

* first bits of rg server use for verl

* refactor: Optimize scoring with single HTTP request in _score_output

* fix: Correct experiment creation with ExperimentCreate object

* grpo tests with server
2025-02-19 22:41:33 +01:00
..
__init__.py build: Initialize reasoning_gym package structure with packaging and development setup 2025-01-23 10:50:54 +01:00
test_ab.py fix: Add validation for size parameter in ABConfig 2025-02-11 23:39:57 +01:00
test_advanced_geometry.py lint 2025-02-01 17:01:11 +01:00
test_aiw.py post merge lint 2025-02-02 10:04:18 +01:00
test_arc_1d.py adapt unit tests to partial match changes 2025-02-14 21:30:50 +01:00
test_arc_1d_tasks.py move arc_1d into from cognition into arc folder 2025-02-08 19:37:26 +01:00
test_arc_agi.py add configuration option for ArcAgiDataset 2025-02-16 12:49:21 +01:00
test_base_conversion.py lint 2025-01-31 12:16:08 +01:00
test_basic_arithmetic.py minor formatting changes 2025-02-17 18:20:18 +01:00
test_bf.py lint 2025-01-30 22:55:04 +01:00
test_binary_matrix.py pre-commit 2025-02-15 16:15:09 +01:00
test_caesar_cipher.py formatting 2025-01-25 18:51:28 +01:00
test_calendar_arithmetic.py post merge formatting 2025-02-02 15:24:39 +01:00
test_chain_sum.py add ProductsDataset (multiplication tasks) 2025-02-13 17:59:02 +01:00
test_circuit_logic.py feat: Add scoring method & unit tests for circuit logic dataset 2025-02-16 22:48:51 +01:00
test_coaching.py add ProductsDataset (multiplication tasks) 2025-02-13 17:59:02 +01:00
test_color_cube_rotation.py add reasoning_gym.create_dataset({name}, ...) global factory function 2025-01-25 00:58:34 +01:00
test_complex_arithmetic.py add ArcAgiDataset class, fix score_entry() metadata params 2025-02-08 23:18:18 +01:00
test_composite.py reasoning-gym-server & cli tool (#154) 2025-02-19 22:41:33 +01:00
test_count_bits.py count bits (#101) 2025-02-10 22:12:50 +01:00
test_count_primes.py lint 2025-02-11 14:44:46 +01:00
test_countdown.py add ArcAgiDataset class, fix score_entry() metadata params 2025-02-08 23:18:18 +01:00
test_course_schedule.py course schedule 2025-02-04 23:50:24 +01:00
test_cryptarithm.py formatting 2025-02-16 16:30:28 +01:00
test_dataset.py ignore single whitespace at beginning and end of answer, use reward = len(oracle_answer) / len(answer) 2025-02-14 15:40:12 +01:00
test_dice.py lint again 2025-02-11 13:00:12 +01:00
test_family_relationships.py feat: Add mother-in-law and father-in-law relationship detection 2025-01-27 21:24:35 +01:00
test_figlet_fonts.py test: Add deterministic test for FigletFontDataset generation 2025-01-30 00:59:57 +01:00
test_fraction_simplification.py formatting 2025-01-24 10:34:07 +01:00
test_futoshiki.py more tolerant parsing of futoshiki answers 2025-02-16 14:23:40 +01:00
test_game_of_life.py feat: Add comprehensive unit tests for Game of Life dataset 2025-02-14 07:28:41 +01:00
test_gcd.py formatting 2025-01-24 10:34:07 +01:00
test_graph_color.py add graph coloring 2025-02-13 01:28:09 +01:00
test_group_anagrams.py test malformed json answer 2025-02-06 10:13:02 +01:00
test_gsm_symbolic.py gsm_symbolic generator changes 2025-02-05 20:58:01 +01:00
test_intermediate_integration.py add ArcAgiDataset class, fix score_entry() metadata params 2025-02-08 23:18:18 +01:00
test_isomorphic_strings.py isomorphic strings 2025-02-07 18:23:34 +01:00
test_knight_swap.py add seed for knight swap 2025-02-14 22:08:46 +01:00
test_largest_island.py added largest island code 2025-02-03 22:46:06 +01:00
test_lcm.py formatting 2025-01-24 10:34:07 +01:00
test_leg_counting.py formatting 2025-01-24 10:34:07 +01:00
test_letter_counting.py formatting 2025-01-24 10:34:07 +01:00
test_letter_jumble.py fix letter jumble 2025-02-14 12:21:56 +01:00
test_manipulate_matrix.py add more config params 2025-02-10 22:30:36 +01:00
test_maze.py add reasoning_gym.create_dataset({name}, ...) global factory function 2025-01-25 00:58:34 +01:00
test_mini_sudoku.py formatting 2025-01-24 10:34:07 +01:00
test_n_queens.py fix score function and add test 2025-02-15 16:09:08 +01:00
test_number_filtering.py formatting 2025-01-24 10:34:07 +01:00
test_number_format.py number format 2025-02-19 11:41:06 +01:00
test_number_sequences.py formatting, cleanup 2025-01-24 17:12:42 +01:00
test_number_sorting.py formatting 2025-01-24 10:34:07 +01:00
test_palindrome.py add ArcAgiDataset class, fix score_entry() metadata params 2025-02-08 23:18:18 +01:00
test_palindrome_partitioning.py palindrome partitioning 2025-02-06 21:42:39 +01:00
test_polynomial_equations.py Fix formatting of added instructions 2025-02-13 21:26:58 -06:00
test_polynomial_multiplication.py Refactor PolynomialMultiplicationDataset and fix issues with score_answer 2025-02-17 17:04:48 +01:00
test_pool_matrix.py min matrix size, fixed question template 2025-02-12 11:20:55 +01:00
test_power_function.py Power Function (#102) 2025-02-10 22:04:58 +01:00
test_prime_factorization.py normalize answer and partial reward 2025-02-09 11:13:23 +01:00
test_products.py adapt unit tests to partial match changes 2025-02-14 21:30:50 +01:00
test_propositional_logic.py feat: Add propositional logic dataset generator with comprehensive logical reasoning tasks 2025-01-23 14:38:56 +01:00
test_quantum_lock.py add algorithmic verification hint in README, lint 2025-01-30 10:14:54 +01:00
test_ransom_note.py lint 2025-02-14 12:17:50 +01:00
test_rearc.py add ArcAgiDataset class, fix score_entry() metadata params 2025-02-08 23:18:18 +01:00
test_rectangle_count.py add rectangle count dataset 2025-02-11 13:56:27 +01:00
test_rotate_matrix.py generlize to k rotations 2025-02-08 15:14:04 +01:00
test_rubiks_cube.py fix chain_sum unit test 2025-01-30 10:57:55 +01:00
test_rush_hour.py fix handling of walls, add unit test 2025-02-14 23:29:17 +01:00
test_self_reference.py add self-reference puzzles 2025-02-07 15:09:42 +01:00
test_sentence_reordering.py formatting 2025-02-16 16:18:39 +01:00
test_simple_equations.py fix operators configuration 2025-01-24 19:44:46 +01:00
test_simple_geometry.py lint 2025-02-01 17:01:11 +01:00
test_simple_integration.py add ArcAgiDataset class, fix score_entry() metadata params 2025-02-08 23:18:18 +01:00
test_sokoban.py Sokoban without pygame (#77) 2025-02-07 11:57:53 +01:00
test_spell_backward.py rename word_reversal.py -> word_sequence_reversal.py 2025-01-26 11:57:50 +01:00
test_spiral_matrix.py update prompt and score answer 2025-02-16 15:18:45 +01:00
test_string_insertion.py fix score function 2025-02-15 17:40:37 +01:00
test_string_manipulation.py Fix more conflict 2025-02-13 21:24:05 -06:00
test_string_splitting.py remove template from test 2025-02-14 21:58:45 +01:00
test_string_synthesis.py string synthesis 2025-02-13 16:33:28 +01:00
test_sudoku.py formatting 2025-01-24 10:34:07 +01:00
test_syllogisms.py lint 2025-02-08 17:22:55 +01:00
test_time_intervals.py Add time interval dataset class 2025-02-01 02:10:48 +01:00
test_tower_of_hanoi.py add ArcAgiDataset class, fix score_entry() metadata params 2025-02-08 23:18:18 +01:00
test_tsumego.py chore: run pre-commit 2025-02-08 08:33:21 +00:00
test_word_ladder.py Add score_answer method to word_ladder (#93) 2025-02-10 15:15:23 +01:00
test_word_sequence_reversal.py rename word_reversal.py -> word_sequence_reversal.py 2025-01-26 11:57:50 +01:00
test_word_sorting.py fix template 2025-02-16 19:51:24 +01:00
test_zebra.py test: Add deterministic test for ZebraDataset generation 2025-02-03 22:59:23 +01:00