Commit graph

34 commits

Author SHA1 Message Date
joesharratt1229
1c98584f28
Feat/unsloth example (#482)
* cleaned up examples

* updated failing hooks

* updated readme

* corrected linting checks
2025-06-28 17:04:38 +01:00
Zafir Stojanovski
56ce2e79a7
tutorial(training): Add a minimal example with trl (#473)
* v0

* 2 gpu setup

* improve parsing from yaml

* update yaml dataset example

* remove restriction on flash attn

* more comments

* first version of the readme

* pin torch

* simplify requirements

* just flash attn

* use set env instead

* simpler set env

* readme

* add wandb project to setup

* update template

* update model id

* post init to capture the config and weight

* extract metadata

* update config

* update dataset config

* move env for wandb project

* pre-commit

* remove qwen-math from training

* more instructions

* unused import

* remove trl old

* warmup ratio

* warmup ratio

* change model id

* change model_id

* add info about CUDA_VISIBLE_DEVICES
2025-06-21 00:01:31 +02:00
Oliver Stanley
49f3821098
add minimal verifiers example (#472) 2025-06-20 16:31:02 +01:00
joesharratt1229
51c2afc1fc
Fix/verl example (#465)
* updated verl ex

* updated script

* removed curriculum verl and updated

* updatied linting errors

* renamed

* updated config
2025-06-09 09:53:43 +01:00
joesharratt1229
9234aa77bf
Feat/open instruct example (#381)
* added open-instruct

* fixed hooks

* GRPO

---------

Co-authored-by: Andreas Koepf <andreas.koepf@provisio.com>
2025-03-17 23:20:11 +01:00
Oliver Stanley
1c6f2d01ee
use StatefulDataLoader in veRL examples (#378) 2025-03-17 07:28:10 +01:00
Oliver Stanley
bd13b1b92a
Fix chain sum veRL example for latest veRL (#371)
* fixes for latest verl

* add balance_batch cofg

* 1 -> 2 gpu

* tweaks

* also add raw ids to server script
2025-03-14 20:15:54 +01:00
Andreas Köpf
c69bc5d4e6
Basic curriculum (#198)
* feat: Add optional curriculum support to dataset registration and creation
* docs: Add docstrings to create_curriculum() and register_dataset()
* feat: Add curriculum configuration classes for CurriculumExperiment
* feat: Add weight parameter to CurriculumAttributeConfig and use in DatasetSpec
* refactor: Simplify CurriculumAttributeConfig with "*" attribute level support
* test: Add unit tests for CurriculumExperiment class
* feat: Add from_yaml() method to CurriculumExperimentConfig with unit test
2025-03-07 11:22:12 +01:00
Andreas Koepf
eeb9fa31d5 more native type hints 2025-02-21 21:23:14 +01:00
Andreas Köpf
b59ccdefa2
Merge pull request #178 from olliestanley/feature/unsloth-train
Add minimal working GRPO training example with Unsloth
2025-02-21 15:37:24 +01:00
Andreas Koepf
3e7ff3b084 use native types List->list, Dict->dict, Set->set, Tuple->tuple 2025-02-21 15:15:38 +01:00
Oliver
f16dd9a7d4 Better progress tracking 2025-02-20 23:32:54 +00:00
Oliver
90547f30c7 Set log level 2025-02-20 22:36:07 +00:00
Oliver
e50444b924 Add requirements for Unsloth example 2025-02-20 22:29:13 +00:00
Oliver
29d116b034 Call for_inference before eval 2025-02-20 22:27:33 +00:00
Oliver
a01110aa7c Add eval 2025-02-20 22:18:52 +00:00
Oliver
8bd18f5387 Add minimal unsloth GRPO example 2025-02-20 22:03:40 +00:00
Andreas Köpf
e2702092f4
reasoning-gym-server & cli tool (#154)
* feat: Add initial server structure with configuration, registry, and middleware

* feat: Add chain_sum dataset to experiment registry test

* fix: Update test_registry to use DatasetSpec for composite config validation

* refactor: Update Pydantic config to use json_schema_extra and ConfigDict

* feat: Add Pydantic models for API request/response data

* feat: Implement basic experiment management endpoints with tests

* feat: Implement composite configuration endpoints for experiments

* fix: Add missing DatasetConfigUpdate import in server.py

* refactor: Update dataset config update method to properly merge config updates

* fix: Correctly retrieve current dataset config in composite endpoint

* feat: Add basic CLI structure with experiments and config commands

* feat: Add initial CLI tool with basic experiment management commands

* refactor: Reorganize CLI package structure and fix import paths

* refactor: Implement initial CLI commands for experiment management

* feat: Implement HTTP client for Reasoning Gym server in RGC CLI tool

* fix: Move print statements inside try block to resolve SyntaxError

* fix: Resolve SyntaxError in edit_config function by adding missing except block

* feat: Add default app instance in server module for easier uvicorn startup

* docs: Add README.md with server and RGC tool documentation

* remove unused files

* refactor: Remove unsupported type annotation in registry.py

* refactor: Move ExperimentRegistry to coaching module and add Experiment class

* fix: Add missing CompositeDataset import in test_registry.py

* refactor: Implement lazy ASGI app creation for server initialization

* feat: Add health check command to RGC CLI for server connection

* feat: Add version tracking support to CompositeDataset

* feat: Add DatasetVersionManager for tracking dataset versions

* feat: Add entry_id metadata and score_answer_with_id method to CompositeDataset

* feat: Add entry_id metadata combining version and index

* fix: Resolve undefined variable by storing version_id before use

* test: Add comprehensive unit tests for score_answer_with_id() function

* test: Add comprehensive version tracking test for dataset config updates

* feat: Validate dataset weights are positive in CompositeDataset initialization

* feat: Add weight update and normalization methods to CompositeDataset

* refactor: Centralize weight normalization in CompositeDataset and allow zero-weight datasets

* feat: Add negative weight validation to CompositeDataset constructor

* feat: Add duplicate dataset name check in CompositeDataset and update test

* refactor: Move duplicate dataset name check inside dataset iteration loop

* refactor: Update CompositeDataset weight management to use config as source of truth

* refactor: Move duplicate dataset name check to CompositeConfig.validate()

* test: Update composite dataset weight test assertions and validation

* feat: Add methods to add and remove datasets in CompositeDataset

* refactor: Remove weight normalization and use unnormalized weights directly

* refactor: Remove redundant total weight check in update_dataset_weights

* feat: Add batch generation and scoring endpoints to server

* fix: Import BatchEntry in server.py to resolve undefined name error

* refactor: Update ReasoningGymDataset to use server for batch generation and scoring

* fix: Add missing List and Dict type imports

* feat: Add get_batch() and score_outputs() methods to RGClient

* test: Add unit tests for generate_batch and score_outputs endpoints

* refactor: Add DatasetVersionManager to Experiment class and CompositeDataset constructor

* feat: Add validation for base_index and batch_size in generate_batch endpoint

* refactor: Remove unused BatchRequest type from imports

* refactor: Convert models to use Pydantic exclusively

* test: Update scoring endpoint tests to use correct request model format

* refactor: Rename ScoreItem to AnswerItem and update related code

* feat: Update scoring endpoint to return ordered ScoringResponse with scores and entry_ids

* fix: Add missing ScoringResponse import in server.py

* move verl ppo sample with server into own file

* refactor: Use Pydantic models for get_batch() and score_outputs() in RGClient

* refactor: Update client methods to use Pydantic models for type safety

* refactor: Use Pydantic models for experiment and dataset config operations

* refactor: Clean up duplicate methods and improve error handling in main.py

* first bits of rg server use for verl

* refactor: Optimize scoring with single HTTP request in _score_output

* fix: Correct experiment creation with ExperimentCreate object

* grpo tests with server
2025-02-19 22:41:33 +01:00
Andreas Koepf
8e25a4585f add grpo launch script 2025-02-17 20:52:03 +00:00
Andreas Koepf
2ae21c6548 update config to latest veRL version 2025-02-17 18:43:51 +00:00
Andreas Köpf
7b72c3470b
docs: Update TRL README with GRPO example details and usage instructions (#76) 2025-02-07 07:56:22 +01:00
joesharratt1229
a8e11e71be
Test training with trl (#70)
* first trl grpo implementation
* added config yaml file
* added read me and dependencies
* updated reward format func
2025-02-07 07:42:32 +01:00
Cavit Erginsoy
aff0fecef4 lint 2025-02-03 11:35:30 +00:00
Cavit Erginsoy
9b1068ea39 Merge remote-tracking branch 'upstream/main' 2025-02-03 07:44:32 +00:00
Cavit Erginsoy
7b61fc5043 Completed: full example suite 2025-02-03 07:21:12 +00:00
Andreas Koepf
8202f234be reduce veRL example size 2025-02-01 23:56:11 +00:00
Andreas Koepf
3f24df31dc add deps for veRL experiment in README 2025-02-01 21:27:33 +00:00
Andreas Koepf
e671b97ab4 first bits of veRL example 2025-02-01 21:20:36 +00:00
Andreas Koepf
5ae329becd lint 2025-01-30 23:14:32 +01:00
Cavit Erginsoy
df3c4580ee INIT 2025-01-30 21:32:46 +00:00
Andreas Koepf
fc775eda7e lint, seed & size for figlet 2025-01-30 00:58:34 +01:00
Andreas Koepf
f7313d409c use more realistic hparams for OpenRLHF example 2025-01-28 22:20:22 +00:00
Andreas Koepf
c196d622e0 extract answer from last answer tag 2025-01-28 16:37:19 +00:00
Andreas Koepf
cc0312e446 add first example with OpenRLHF 2025-01-28 14:40:06 +00:00