141 commits

Author	SHA1	Message	Date
balyan.sid@gmail.com	57fa229846	remove unused managed_server wrapper + tese	2026-01-14 17:09:57 +05:30
balyan.sid@gmail.com	6a27e88023	use managed server	2026-01-14 17:09:01 +05:30
balyan.sid@gmail.com	cf636595d2	rework server and eval for rl rollout. add in asyncmanagedserver for ... verifiers	2026-01-12 10:34:05 +05:30
Dakota	8ec5066998	add eval runner	2025-12-19 19:56:59 -06:00
Ksenchi	6c843d6677	Update MANAGED_SERVER.md	2025-11-12 07:22:40 +01:00
Ksenchi	edf26d68e1	Update MANAGED_SERVER.md	2025-11-12 07:22:06 +01:00
Dhyaneesh	39d5fb4452	feat: dump evaluate subcommand config to YAML in env save dir ... Automatically save the final merged evaluate configuration to evaluate_config.yaml in the data_dir_to_save_evals directory. This includes env config, OpenAI/server configs, and server manager settings, enabling reproducibility and easier debugging of evaluation runs. The config is saved after all merging (CLI args > YAML > defaults) to capture the exact configuration used for the evaluation.	2025-11-08 23:46:13 +05:30
dmahan93	b4080a4f37	Merge pull request #273 from NousResearch/add-vllm-manager-fn ... add managed vllm server	2025-11-07 14:22:07 -08:00
Dakota	e6ac3abdcb	add managed vllm server	2025-11-07 13:06:49 -06:00
dmahan93	c96b8a1255	Merge pull request #267 from bobtajson/main ... fix: correct typo and improve code quality	2025-11-06 18:36:37 -08:00
dmahan93	b1e164eef5	Merge pull request #264 from NousResearch/add-logprob-server-manager-fn ... add sglang specific token level logprob handling and server manager/b…	2025-10-29 13:53:39 -07:00
Dakota	578175a709	fix pre-commit	2025-10-29 14:47:50 -05:00
Dakota	5d6d6bb0dc	add docs :)	2025-10-29 11:26:43 -05:00
Dakota	c3a118f50d	fix tests	2025-10-29 10:55:10 -05:00
Dakota	d5400460e8	made masked logprobs coherently decided on	2025-10-29 10:52:38 -05:00
Dakota	e57c396f86	ran pre-commit	2025-10-29 10:45:27 -05:00
Dakota	17bb7bdf15	revert base.py	2025-10-29 10:11:05 -05:00
dmahan93	c483840f59	set prompt logprobs to a masked value	2025-10-26 11:58:55 -07:00
dmahan93	c22f8ca81b	Merge remote-tracking branch 'origin/add-logprob-server-manager-fn' into add-logprob-server-manager-fn	2025-10-24 23:18:37 -07:00
dmahan93	5d662bf1aa	add chat example and fix bug in managed_server	2025-10-24 23:15:56 -07:00
pre-commit-ci[bot]	0d80da5146	[pre-commit.ci] auto fixes from pre-commit.com hooks ... for more information, see https://pre-commit.ci	2025-10-24 20:10:29 +00:00
dmahan93	7bf4cfbf80	add managed server to make grabbing logprobs easier w/ tokenized items	2025-10-24 13:09:46 -07:00
bobtajson	6e2d36bd2a	Update base.py	2025-10-23 10:27:23 +02:00
pre-commit-ci[bot]	312f8859e3	[pre-commit.ci] auto fixes from pre-commit.com hooks ... for more information, see https://pre-commit.ci	2025-10-16 18:47:07 +00:00
Dakota	d240dbb3b7	Merge remote-tracking branch 'origin/add-logprob-server-manager-fn' into add-logprob-server-manager-fn	2025-10-16 13:46:03 -05:00
Dakota	134cbc09d0	update openai/trl_vllm server with new fn	2025-10-16 13:45:55 -05:00
pre-commit-ci[bot]	1e6a745491	[pre-commit.ci] auto fixes from pre-commit.com hooks ... for more information, see https://pre-commit.ci	2025-10-16 17:39:04 +00:00
Dakota	c36ec29656	add sglang specific token level logprob handling and server manager/baseline logprob/token fn	2025-10-16 12:38:03 -05:00
ropresearch	e5b8fb8654	clean up	2025-10-10 11:50:39 -04:00
ropresearch	baf4b2d8a8	gzip compression for atropos api	2025-10-10 01:26:52 -04:00
ropresearch	c3fc68879c	group temps, sample temps, and logprob api params	2025-09-25 16:41:58 -04:00
shannonsands	1a808e2038	Revert "Fix multiple scored data groups (#223)" ... This reverts commit 67b3144113.	2025-08-29 17:55:45 +10:00
shannonsands	67b3144113	Fix multiple scored data groups (#223) ... * removed changes to other files * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fail on scores empty --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>	2025-08-29 15:47:32 +10:00
Dakota	11f1303da0	add error logging to collect_trajectories so they don't fail silently	2025-08-15 16:34:21 -05:00
pre-commit-ci[bot]	3d2d9e67fa	[pre-commit.ci] auto fixes from pre-commit.com hooks ... for more information, see https://pre-commit.ci	2025-07-15 11:42:46 +00:00
Alexey Gorbatovski	53984580c8	Bug fix	2025-07-15 14:37:55 +03:00
hjc-puro	72210cf4ad	rename fn	2025-07-11 04:04:55 +00:00
hjc-puro	d133ba3867	comment	2025-07-11 03:54:03 +00:00
hjc-puro	ccb8eaf230	move table to util	2025-07-11 03:52:24 +00:00
hjc-puro	5e61331360	simplify schema	2025-07-11 03:49:49 +00:00
hjc-puro	290e087fc5	remove some imports	2025-07-11 03:25:10 +00:00
hjc-puro	68da3809e2	move table to display util	2025-07-11 02:06:56 +00:00
hjc-puro	3e08c6d788	simplify schema	2025-07-11 00:52:09 +00:00
hjc-puro	6c64df0226	remove jsonlines dependency	2025-07-11 00:42:55 +00:00
hjc-puro	da0d64ae89	linting errors	2025-07-11 00:29:57 +00:00
hjc-puro	e601251893	gsm8k eval example	2025-07-11 00:22:36 +00:00
hjc-puro	eb926dc58b	working evals	2025-07-10 01:45:21 +00:00
hjc-puro	f4de3ad6f5	add printing	2025-07-09 23:35:26 +00:00
hjc-puro	a11af27298	add eval saving cli args	2025-07-09 03:12:13 +00:00
hjc-puro	5519f190d2	add evaluate subcommand to cli	2025-07-07 17:39:33 -04:00