mirror of
https://github.com/InternLM/InternBootcamp.git
synced 2026-04-19 12:58:04 +00:00
Merge branch 'add--custom-bootcamp-create-doc' into 'main'
docs(README): update quick start guide and add custom Bootcamp creation instructions See merge request lilinyang/internbootcamp!11
This commit is contained in:
李林阳
commit
66796c6a1e
4 changed files with 459 additions and 28 deletions
|
|
@ -11,6 +11,7 @@ Internbootcamp is an easy-to-use and extensible library of bootcamp environments
|
|||
|
||||
## Getting Started
|
||||
|
||||
**Quickly get started with data generation, reinforcement learning training, model evaluation, and custom Bootcamp creation!**
|
||||
* [Installation](#section3)
|
||||
* [Quick Start](examples/get_started.md)
|
||||
* [Interfaces and Usages](#section4)
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@
|
|||
Internbootcamp是一个易用、可扩展的推理模型训练训练场环境。通过集成大量可验证的推理任务并为其支持无限量的**自动化问题生成**和**结果验证**，Internbootcamp旨在提供推理形式多样且难度可控的大规模数据资源，以推动大模型在广泛场景下的推理能力提升及泛化。目前，Internbootcamp已集成上千种可验证的、难度可控的推理任务，包含游戏（games）、逻辑问题（logic）、谜题（puzzles）、算法问题（algorithms）等，并在持续扩展中。
|
||||
|
||||
## Getting Started
|
||||
|
||||
**快速开始数据生成、模型训练、模型评测以及自定义Bootcamp！**
|
||||
* [安装](#section3)
|
||||
* [快速开始](examples/get_started_zh.md)
|
||||
* [接口及用法](#section4)
|
||||
|
|
|
|||
|
|
@ -1,24 +1,23 @@
|
|||
# Quick Start
|
||||
|
||||
**InternBootcamp** provides functionalities such as data generation, model training, and model evaluation. You can refer to the following guide to get started quickly.
|
||||
InternBootcamp provides functionalities such as data generation, model training, model evaluation, and custom Bootcamp creation. Please refer to the following guide for a quick start.
|
||||
|
||||
To ensure successful execution of the following steps, please make sure **InternBootcamp is installed**, and **the project root directory is set as your working directory**.
|
||||
To ensure the successful execution of subsequent operations, make sure you have installed InternBootcamp and set the project root directory as your working directory.
|
||||
|
||||
## Data Generation
|
||||
|
||||
Run [**run\_pipeline.sh**](examples/pipelines/run_pipeline.sh) to generate training and testing data based on the [default configuration](examples/pipelines/data_configs).
|
||||
For custom configurations, please refer to the [Pipeline Usage Guide](examples/pipelines/README.md) for personalized setup.
|
||||
Running [**run_pipeline.sh**](/examples/pipelines/run_pipeline.sh) will generate corresponding test and training data based on the **default configuration**`examples/pipelines/data_configs`. If you have custom configuration needs, please refer to [Pipeline Usage](pipelines/README.md) for personalized configuration.
|
||||
|
||||
```bash
|
||||
source examples/pipelines/run_pipeline.sh
|
||||
```
|
||||
|
||||
The generated data will be saved in the [bootcamp\_generator\_outputs directory](examples/bootcamp_generator_outputs). Each data batch is timestamped, and the directory structure is as follows:
|
||||
The generated data is saved in the **bootcamp_generator_outputs directory**`examples/bootcamp_generator_outputs`, with data batches named using timestamps. The specific directory structure is as follows:
|
||||
|
||||
```
|
||||
examples/
|
||||
├── ...
|
||||
└──bootcamp_generator_outputs/
|
||||
└── bootcamp_generator_outputs/
|
||||
├── ...
|
||||
└── 2025-xx-xx-xx:xx:xx/
|
||||
├── test/
|
||||
|
|
@ -31,23 +30,21 @@ examples/
|
|||
└── bootcamp_n.jsonl
|
||||
```
|
||||
|
||||
## Model Training(Reinforcement learning)
|
||||
## Model Training
|
||||
|
||||
We provide support for two training frameworks: **Xpuyu** and **Verl**.
|
||||
We provide support systems for two training frameworks (Xpuyu and Verl).
|
||||
|
||||
### Xpuyu
|
||||
|
||||
Refer to the [Xpuyu documentation](examples/xpuyu_usage/README.md) to get started with efficient training.
|
||||
Refer to the [Xpuyu documentation](/examples/xpuyu_usage/README.md) for efficient training.
|
||||
|
||||
### Verl
|
||||
|
||||
To integrate Bootcamp tasks into the Verl framework for training, you need to embed the Bootcamp reward computation method into Verl.
|
||||
See the [Verl documentation](examples/verl_usage/README.md) for detailed guidance.
|
||||
To incorporate Bootcamp tasks into training within the Verl framework, you need to embed the Bootcamp reward calculation method into the Verl framework according to the instructions. For details, see the [Verl documentation](/examples/verl_usage/README.md).
|
||||
|
||||
## Model Evaluation
|
||||
|
||||
For Bootcamp tasks, we offer a customized evaluation service.
|
||||
Once the model to be evaluated is deployed using frameworks like FastChat or Ollama, and you have the corresponding API URL and API Key, you can run the following command to evaluate your deployed model on the **InternBootcamp\_eval** dataset:
|
||||
We offer personalized evaluation services for Bootcamp tasks. After deploying the model to be tested using frameworks like FastChat or Ollama and obtaining the corresponding API URL and API Key, use the following command to quickly evaluate the performance of the deployed model on the **InternBootcamp_eval** dataset:
|
||||
|
||||
```bash
|
||||
cd InternBootcamp
|
||||
|
|
@ -66,9 +63,216 @@ python examples/unittests/run_eval.py \
|
|||
--max_retrying_delay 60
|
||||
```
|
||||
|
||||
**Note:** When `api_mode` is set to `completion`, be sure to correctly specify the corresponding template (supported: `r1`, `qwen`, `internthinker`, `chatml` (with no system prompt)).
|
||||
For more detailed instructions, refer to the [Evaluation Manual](examples/unittests/README.md).
|
||||
Note: When `api_mode` is set to `completion`, ensure that the corresponding `template` is correctly configured (supported options include `r1`, `qwen`, `internthinker`, and `chatml` (no system prompt)). For more details, refer to the [Evaluation Manual](/examples/unittests/README.md).
|
||||
|
||||
---
|
||||
## Custom Bootcamp
|
||||
|
||||
Let me know if you’d like a version with clearer formatting for publishing or documentation.
|
||||
If you need to create a custom Bootcamp task, follow these steps:
|
||||
|
||||
### 1. Create a new subdirectory under the `internbootcamp/bootcamp` directory, named after the Bootcamp task name
|
||||
|
||||
In this directory, you can define multiple versions of the task class. Below is an example using `binary_arithmetic_operations_default.py` (implementing a binary arithmetic operation Bootcamp) to demonstrate the entire process of creating a custom Bootcamp.
|
||||
|
||||
### 2. Create a Bootcamp class, inherit from the `Basebootcamp` class, and implement the `__init__` method
|
||||
|
||||
- The class name must end with `bootcamp`.
|
||||
- Avoid defining mutable variables like `list` in the `__init__` method (recommended to define them in the `case_generator` method) to prevent non-reset variables from being reused.
|
||||
- Avoid fixing random variables here (recommended to determine random variables in the `case_generator` method) to avoid generating repetitive data due to lack of randomness.
|
||||
- Input parameters for the `__init__` method are configured via JSON files located in **examples/pipelines/puzzle_configs**, ensuring the JSON configuration file name matches the Bootcamp class name, e.g., `BinaryArithmeticOperationsbootcamp_test.json`, `BinaryArithmeticOperationsbootcamp_train.json`.
|
||||
|
||||
```python
|
||||
from internbootcamp.bootcamp.base import Basebootcamp
|
||||
|
||||
class BinaryArithmeticOperationsbootcamp(Basebootcamp):
|
||||
def __init__(self, operands_range=(0, 10), precision=3, operators=['+', '-', '*', '/']):
|
||||
self.operands_range = operands_range
|
||||
self.precision = precision
|
||||
self.operators = operators
|
||||
|
||||
def case_generator(self):
|
||||
pass
|
||||
|
||||
def prompt_func(self, identity) -> dict:
|
||||
pass
|
||||
|
||||
@staticmethod
|
||||
def extract_output(output):
|
||||
pass
|
||||
|
||||
@classmethod
|
||||
def _verify_correction(cls, solution, identity: dict):
|
||||
pass
|
||||
```
|
||||
|
||||
### 3. Implement the `case_generator` method (optional if no data generation is needed)
|
||||
|
||||
`case_generator` is an instance method used to generate parameters required to construct a single question or validate a response (i.e., parameters that uniquely determine a question), returning a parameter dictionary.
|
||||
|
||||
```python
|
||||
import random
|
||||
|
||||
def case_generator(self) -> dict:
|
||||
operand1 = round(random.uniform(*self.operands_range), self.precision)
|
||||
operand2 = round(random.uniform(*self.operands_range), self.precision)
|
||||
operator = random.choice(self.operators)
|
||||
while operator == '/' and operand2 == 0:
|
||||
operand2 = round(random.uniform(*self.operands_range), self.precision)
|
||||
case = {}
|
||||
case['operand1'] = operand1
|
||||
case['operand2'] = operand2
|
||||
case['operator'] = operator
|
||||
case['precision'] = self.precision
|
||||
return case
|
||||
```
|
||||
|
||||
### 4. Implement the `prompt_func` method (optional if no data generation is needed)
|
||||
|
||||
`prompt_func` is an instance, static, or class method used to construct the problem statement for a single question, taking the parameter dictionary returned by `case_generator` as input `identity`, and returning the problem statement as a string.
|
||||
|
||||
```python
|
||||
def prompt_func(self, identity: dict) -> str:
|
||||
return f"{identity['operand1']} {identity['operator']} {identity['operand2']} = ? Results should be rounded to {identity['precision']} decimal places and enclosed in double brackets like [[result]]."
|
||||
```
|
||||
|
||||
### 5. Implement the `extract_output` method to extract answers from the model output
|
||||
|
||||
`extract_output` is a static method that accepts the model output as input `output` and returns the answer in any form.
|
||||
|
||||
```python
|
||||
import re
|
||||
|
||||
@staticmethod
|
||||
def extract_output(output):
|
||||
matches = re.findall(r'\[\[([^\[\]]+)\]\]', output)
|
||||
return matches[-1].strip() if matches else None
|
||||
```
|
||||
|
||||
### 6. Implement the `_verify_correction` method to validate whether the model-generated answer is correct
|
||||
|
||||
`_verify_correction` is a class method that accepts the model output and the output of `case_generator` as input parameters `solution` and `identity`, returning a boolean indicating whether the answer is correct or a float/integer between 0 and 1 indicating the degree of correctness.
|
||||
|
||||
```python
|
||||
@classmethod
|
||||
def _verify_correction(cls, solution, identity: dict) -> bool:
|
||||
try:
|
||||
# Convert the string-form answer to a float
|
||||
solution = float(solution)
|
||||
except (ValueError, TypeError):
|
||||
return False
|
||||
|
||||
# Calculate the correct answer based on the problem parameters
|
||||
operand1 = identity['operand1']
|
||||
operand2 = identity['operand2']
|
||||
operator = identity['operator']
|
||||
precision = identity['precision']
|
||||
|
||||
if operator == '+':
|
||||
correct_answer = operand1 + operand2
|
||||
elif operator == '-':
|
||||
correct_answer = operand1 - operand2
|
||||
elif operator == '*':
|
||||
correct_answer = operand1 * operand2
|
||||
elif operator == '/':
|
||||
# Check for division by zero
|
||||
if operand2 == 0:
|
||||
return False
|
||||
correct_answer = operand1 / operand2
|
||||
else:
|
||||
raise ValueError(f"Unsupported operator: {operator}")
|
||||
|
||||
# Round the correct answer to the specified precision
|
||||
correct_answer = round(correct_answer, precision)
|
||||
|
||||
# Compare the model output with the correct answer
|
||||
return abs(solution - correct_answer) < 1e-6
|
||||
```
|
||||
|
||||
### 7. Configure the JSON file (optional if no data generation is needed)
|
||||
|
||||
Create two JSON configuration files (for training and testing) in the `examples/pipelines/puzzle_configs` directory, with names matching the Bootcamp class name but without the "bootcamp" suffix (e.g., `BinaryArithmeticOperations_train.json`, `BinaryArithmeticOperations_test.json`), to define the parameters for this task. Below is an example configuration:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"operands_range": [-10, 10],
|
||||
"precision": 3,
|
||||
"operators": ["+", "-", "*", "/"]
|
||||
},
|
||||
{
|
||||
"operands_range": [10, 1000],
|
||||
"precision": 4,
|
||||
"operators": ["+", "-", "*", "/"]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
Ensure that the keys in the JSON file match the parameter names in the `__init__` method.
|
||||
|
||||
### 8. Register the custom Bootcamp task
|
||||
|
||||
To enable the system to recognize and load the custom Bootcamp task, register the task in `internbootcamp/bootcamp/__init__.py`. For example:
|
||||
|
||||
```python
|
||||
from .binary_arithmetic_operations.binary_arithmetic_operations_default import BinaryArithmeticOperationsbootcamp
|
||||
```
|
||||
|
||||
### 9. Test the custom Bootcamp task
|
||||
|
||||
After completing the above steps, you can test the custom Bootcamp task as follows:
|
||||
1. Use `case_generator` to generate test cases.
|
||||
2. Call `prompt_func` to generate the problem statement.
|
||||
3. Use `extract_output` to extract answers from the model output.
|
||||
4. Call `_verify_correction` to verify the correctness of the answer.
|
||||
|
||||
Below is a simple test code example:
|
||||
|
||||
```python
|
||||
if __name__ == "__main__":
|
||||
# Initialize the Bootcamp task
|
||||
bootcamp = BinaryArithmeticOperationsbootcamp(
|
||||
operands_range=(0, 10),
|
||||
precision=3,
|
||||
operators=['+', '-', '*', '/']
|
||||
)
|
||||
|
||||
# Generate a test case
|
||||
case = bootcamp.case_generator()
|
||||
print("Generated Case:", case)
|
||||
|
||||
# Construct the problem statement
|
||||
prompt = bootcamp.prompt_func(case)
|
||||
print("Prompt:", prompt)
|
||||
|
||||
# Simulate model output
|
||||
answer = eval(f"{case['operand1']} {case['operator']} {case['operand2']}")
|
||||
model_output = f"[[{round(answer, 3)}]]"
|
||||
extracted_answer = bootcamp.extract_output(model_output)
|
||||
print("Extracted Answer:", extracted_answer)
|
||||
|
||||
# Verify the answer
|
||||
is_correct = bootcamp._verify_correction(extracted_answer, case)
|
||||
print("Is Correct:", is_correct)
|
||||
```
|
||||
|
||||
### 10. Integrate into Pipeline (optional if no data generation is needed)
|
||||
|
||||
To integrate the custom Bootcamp task into the data generation pipeline, simply add the corresponding task definition to the two configuration files in the `examples/pipelines/data_configs` directory. For example:
|
||||
|
||||
```json
|
||||
{
|
||||
"bootcamp_name": "BinaryArithmeticOperations",
|
||||
"sample_number": 64,
|
||||
"config_file": "BinaryArithmeticOperations",
|
||||
"bootcamp_cls_name": "BinaryArithmeticOperationsbootcamp"
|
||||
}
|
||||
```
|
||||
|
||||
Where:
|
||||
- `bootcamp_name` is the task name (i.e., the class name without the `bootcamp` suffix).
|
||||
- `sample_number` is the number of samples to generate.
|
||||
- `config_file` is the configuration file name.
|
||||
- `bootcamp_cls_name` is the task class name.
|
||||
|
||||
Alternatively, run the script [quickgen_data_configs.py](/examples/pipelines/quickgen_data_configs.py), which automatically generates corresponding pipeline configurations for Bootcamps configured in `examples/pipelines/puzzle_configs` and saves them in `examples/pipelines/data_configs`.
|
||||
|
||||
**By following the above steps, you can successfully create, test, and integrate a custom Bootcamp task!**
|
||||
|
|
@ -1,17 +1,23 @@
|
|||
# 快速开始
|
||||
InternBootcamp提供了数据生成、模型训练、模型评测等功能，可参考如下指南快速入门。
|
||||
为确保后续操作成功执行，请确保您已经安装了InternBootcamp，并将项目根目录设为工作目录。
|
||||
|
||||
InternBootcamp 提供了数据生成、模型训练、模型评测和自定义 Bootcamp 等功能。请参考以下指南快速入门。
|
||||
|
||||
为确保后续操作成功执行，请确保您已经安装了 InternBootcamp，并将项目根目录设为工作目录。
|
||||
|
||||
## 数据生成
|
||||
|
||||
运行[**run_pipeline.sh**](examples/pipelines/run_pipeline.sh)即可根据[默认配置](examples/pipelines/data_configs)生成对应的测试与训练数据。若有自定义配置等需求可参考[Pipeline Usage](examples/pipelines/README_zh.md)进行个性化配置。
|
||||
运行 [**run_pipeline.sh**](/examples/pipelines/run_pipeline.sh) 即可根据 **默认配置**`examples/pipelines/data_configs`生成对应的测试与训练数据。若有自定义配置需求，请参考 [Pipeline Usage](pipelines/README_zh.md) 进行个性化配置。
|
||||
|
||||
```bash
|
||||
source examples/pipelines/run_pipeline.sh
|
||||
```
|
||||
生成的数据保存于[bootcamp_generator_outputs目录](examples/bootcamp_generator_outputs)，数据批次以时间戳命名，具体目录结构如下
|
||||
|
||||
生成的数据保存于 **bootcamp_generator_outputs 目录**`examples/bootcamp_generator_outputs`，数据批次以时间戳命名，具体目录结构如下：
|
||||
|
||||
```
|
||||
examples/
|
||||
├── ...
|
||||
└──bootcamp_generator_outputs/
|
||||
└── bootcamp_generator_outputs/
|
||||
├── ...
|
||||
└── 2025-xx-xx-xx:xx:xx/
|
||||
├── test/
|
||||
|
|
@ -25,14 +31,21 @@ examples/
|
|||
```
|
||||
|
||||
## 模型训练
|
||||
我们提供了两种训练框架（Xpuyu,Verl）的支持体系。
|
||||
|
||||
我们提供了两种训练框架（Xpuyu 和 Verl）的支持体系。
|
||||
|
||||
### Xpuyu
|
||||
可参考[Xpuyu说明文档](examples/xpuyu_usage/README_zh.md)快速进行高效训练。
|
||||
|
||||
可参考 [Xpuyu 说明文档](/examples/xpuyu_usage/README_zh.md) 快速进行高效训练。
|
||||
|
||||
### Verl
|
||||
要在Verl框架下加入Bootcamp任务进行训练，需要参考说明文档将bootcamp奖励计算方法嵌入verl框架，具体参考[Verl说明文档](examples/verl_usage/README_zh.md)进行。
|
||||
|
||||
要在 Verl 框架下加入 Bootcamp 任务进行训练，需要参考说明文档将 Bootcamp 奖励计算方法嵌入 Verl 框架，具体参考 [Verl 说明文档](/examples/verl_usage/README_zh.md)。
|
||||
|
||||
## 模型评测
|
||||
我们针对bootcamp任务，提供了个性化的评测服务，在使用FastChat、Ollama等框架部署好待测模型并获取对应API Url 与API Key后,使用如下命令可快速评测已部署的模型在**InternBootcamp_eval**评测集上的性能：
|
||||
|
||||
我们针对 Bootcamp 任务提供了个性化的评测服务。在使用 FastChat、Ollama 等框架部署好待测模型并获取对应的 API URL 和 API Key 后，使用以下命令可快速评测已部署的模型在 **InternBootcamp_eval** 评测集上的性能：
|
||||
|
||||
```bash
|
||||
cd InternBootcamp
|
||||
python examples/unittests/run_eval.py \
|
||||
|
|
@ -49,4 +62,217 @@ python examples/unittests/run_eval.py \
|
|||
--max_retries 16 \
|
||||
--max_retrying_delay 60
|
||||
```
|
||||
注意当api_mode指定为completion时，需正确设置对应的template（支持r1、qwen、internthinker、chatml（with no system prompt））。更详细的内容参考[评测手册](examples/unittests/README_zh.md)。
|
||||
|
||||
注意：当 `api_mode` 指定为 `completion` 时，需正确设置对应的 `template`（支持 `r1`、`qwen`、`internthinker`、`chatml`（无系统提示））。更详细的内容请参考 [评测手册](/examples/unittests/README_zh.md)。
|
||||
|
||||
## 自定义 Bootcamp
|
||||
|
||||
若需新建自定义 Bootcamp 任务，可参考以下流程：
|
||||
|
||||
### 1. 在目录 `internbootcamp/bootcamp` 下新建一个子目录，命名为 Bootcamp 任务名称
|
||||
|
||||
在该目录下可定义该任务类的多个版本。以下将以 `binary_arithmetic_operations_default.py` 为例（实现一个二项四则运算 Bootcamp），展示新建自定义 Bootcamp 的全流程。
|
||||
|
||||
### 2. 创建一个 Bootcamp 类，继承 `Basebootcamp` 类，并实现 `__init__` 方法
|
||||
|
||||
- 类名必须以 `bootcamp` 为后缀。
|
||||
- 谨慎将类似 `list` 的可变变量定义在 `__init__` 方法中（推荐在 `case_generator` 方法里定义），避免导致未重置的变量被重复使用。
|
||||
- 谨慎将有随机性的变量在此处确定（推荐在 `case_generator` 方法里确定随机变量），避免生成数据时缺少随机性导致数据重复。
|
||||
- `__init__` 方法的输入参数将以 JSON 文件形式配置于 **examples/pipelines/puzzle_configs**，确保 JSON 配置文件命名与 Bootcamp 类名一致，例如 `BinaryArithmeticOperationsbootcamp_test.json`、`BinaryArithmeticOperationsbootcamp_train.json`。
|
||||
|
||||
```python
|
||||
from internbootcamp.bootcamp.base import Basebootcamp
|
||||
|
||||
class BinaryArithmeticOperationsbootcamp(Basebootcamp):
|
||||
def __init__(self, operands_range=(0, 10), precision=3, operators=['+', '-', '*', '/']):
|
||||
self.operands_range = operands_range
|
||||
self.precision = precision
|
||||
self.operators = operators
|
||||
|
||||
def case_generator(self):
|
||||
pass
|
||||
|
||||
def prompt_func(self, identity) -> dict:
|
||||
pass
|
||||
|
||||
@staticmethod
|
||||
def extract_output(output):
|
||||
pass
|
||||
|
||||
@classmethod
|
||||
def _verify_correction(cls, solution, identity: dict):
|
||||
pass
|
||||
```
|
||||
|
||||
### 3. 实现 `case_generator` 方法（可选，若无需生成数据，则不需要实现）
|
||||
|
||||
`case_generator` 为实例方法，用于生成构造单个题目或验证一次回答所需的参数（即可以唯一确定一个题目所需的参数），并返回参数字典。
|
||||
|
||||
```python
|
||||
import random
|
||||
|
||||
def case_generator(self) -> dict:
|
||||
operand1 = round(random.uniform(*self.operands_range), self.precision)
|
||||
operand2 = round(random.uniform(*self.operands_range), self.precision)
|
||||
operator = random.choice(self.operators)
|
||||
while operator == '/' and operand2 == 0:
|
||||
operand2 = round(random.uniform(*self.operands_range), self.precision)
|
||||
case = {}
|
||||
case['operand1'] = operand1
|
||||
case['operand2'] = operand2
|
||||
case['operator'] = operator
|
||||
case['precision'] = self.precision
|
||||
return case
|
||||
```
|
||||
|
||||
### 4. 实现 `prompt_func` 方法（可选，若无需生成数据，则不需要实现）
|
||||
|
||||
`prompt_func` 为实例、静态或类方法，用于构造单个题目的题面，接受 `case_generator` 返回的参数字典作为输入参数 `identity`，并以字符串形式返回题面。
|
||||
|
||||
```python
|
||||
def prompt_func(self, identity: dict) -> str:
|
||||
return f"{identity['operand1']} {identity['operator']} {identity['operand2']} = ? 结果请保留小数点后{identity['precision']}位，并以[[]]包裹你的答案，如[[result]]。"
|
||||
```
|
||||
|
||||
### 5. 实现 `extract_output` 方法，用于提取模型输出的答案
|
||||
|
||||
`extract_output` 为静态方法，接受模型输出作为输入参数 `output`，并以任意形式返回答案。
|
||||
|
||||
```python
|
||||
import re
|
||||
|
||||
@staticmethod
|
||||
def extract_output(output):
|
||||
matches = re.findall(r'\[\[([^\[\]]+)\]\]', output)
|
||||
return matches[-1].strip() if matches else None
|
||||
```
|
||||
|
||||
### 6. 实现 `_verify_correction` 方法，用于验证模型生成的答案是否正确
|
||||
|
||||
`_verify_correction` 为类方法，接受模型输出和 `case_generator` 的输出作为输入参数 `solution` 和 `identity`，返回一个布尔值表示答案是否正确，或返回 0 到 1 之间的一个浮点型或整数型数字表示答案的正确程度。
|
||||
|
||||
```python
|
||||
@classmethod
|
||||
def _verify_correction(cls, solution, identity: dict) -> bool:
|
||||
try:
|
||||
# 将字符串形式的答案转换为浮点数
|
||||
solution = float(solution)
|
||||
except (ValueError, TypeError):
|
||||
return False
|
||||
|
||||
# 根据题目参数计算正确答案
|
||||
operand1 = identity['operand1']
|
||||
operand2 = identity['operand2']
|
||||
operator = identity['operator']
|
||||
precision = identity['precision']
|
||||
|
||||
if operator == '+':
|
||||
correct_answer = operand1 + operand2
|
||||
elif operator == '-':
|
||||
correct_answer = operand1 - operand2
|
||||
elif operator == '*':
|
||||
correct_answer = operand1 * operand2
|
||||
elif operator == '/':
|
||||
# 检查除零错误
|
||||
if operand2 == 0:
|
||||
return False
|
||||
correct_answer = operand1 / operand2
|
||||
else:
|
||||
raise ValueError(f"Unsupported operator: {operator}")
|
||||
|
||||
# 对正确答案进行精度处理
|
||||
correct_answer = round(correct_answer, precision)
|
||||
|
||||
# 比较模型输出与正确答案
|
||||
return abs(solution - correct_answer) < 1e-6
|
||||
```
|
||||
|
||||
### 7. 配置 JSON 文件（可选，若无需生成数据，则不需要实现）
|
||||
|
||||
在 `examples/pipelines/puzzle_configs` 目录下创建两个(train与test)与 Bootcamp 类名一致但去除bootcamp后缀的 JSON 配置文件（如 `BinaryArithmeticOperations_train.json`、`BinaryArithmeticOperations_test.json`），用于定义该任务的参数。以下是一个示例配置：
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"operands_range": [-10, 10],
|
||||
"precision": 3,
|
||||
"operators": ["+", "-", "*", "/"]
|
||||
},
|
||||
{
|
||||
"operands_range": [10, 1000],
|
||||
"precision": 4,
|
||||
"operators": ["+", "-", "*", "/"]
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
确保 JSON 文件中的键与 `__init__` 方法的参数名称一致。
|
||||
|
||||
### 8. 注册自定义 Bootcamp 任务
|
||||
|
||||
为了使系统能够识别并加载自定义的 Bootcamp 任务，需要在 `internbootcamp/bootcamp/__init__.py` 中注册该任务。例如：
|
||||
|
||||
```python
|
||||
from .binary_arithmetic_operations.binary_arithmetic_operations_default import BinaryArithmeticOperationsbootcamp
|
||||
```
|
||||
|
||||
### 9. 测试自定义 Bootcamp 任务
|
||||
|
||||
在完成上述步骤后，可以通过以下方式测试自定义的 Bootcamp 任务：
|
||||
1. 使用 `case_generator` 生成测试用例。
|
||||
2. 调用 `prompt_func` 生成题面。
|
||||
3. 使用 `extract_output` 提取模型输出的答案。
|
||||
4. 调用 `_verify_correction` 验证答案的正确性。
|
||||
|
||||
以下是一个简单的测试代码示例：
|
||||
|
||||
```python
|
||||
if __name__ == "__main__":
|
||||
# 初始化 Bootcamp 任务
|
||||
bootcamp = BinaryArithmeticOperationsbootcamp(
|
||||
operands_range=(0, 10),
|
||||
precision=3,
|
||||
operators=['+', '-', '*', '/']
|
||||
)
|
||||
|
||||
# 生成测试用例
|
||||
case = bootcamp.case_generator()
|
||||
print("Generated Case:", case)
|
||||
|
||||
# 构造题面
|
||||
prompt = bootcamp.prompt_func(case)
|
||||
print("Prompt:", prompt)
|
||||
|
||||
# 模拟模型输出
|
||||
answer = eval(f"{case['operand1']} {case['operator']} {case['operand2']}")
|
||||
model_output = f"[[{round(answer, 3)}]]"
|
||||
extracted_answer = bootcamp.extract_output(model_output)
|
||||
print("Extracted Answer:", extracted_answer)
|
||||
|
||||
# 验证答案
|
||||
is_correct = bootcamp._verify_correction(extracted_answer, case)
|
||||
print("Is Correct:", is_correct)
|
||||
```
|
||||
|
||||
### 10. 集成到 Pipeline（可选，若无需生成数据，则不需要实现）
|
||||
|
||||
将自定义的 Bootcamp 任务集成到数据生成 Pipeline 中，只需要在 `examples/pipelines/data_configs` 目录下的两个配置文件中添加相应的任务定义。例如：
|
||||
|
||||
```json
|
||||
{
|
||||
"bootcamp_name": "BinaryArithmeticOperations",
|
||||
"sample_number": 64,
|
||||
"config_file": "BinaryArithmeticOperations",
|
||||
"bootcamp_cls_name": "BinaryArithmeticOperationsbootcamp"
|
||||
}
|
||||
```
|
||||
|
||||
其中：
|
||||
- `bootcamp_name` 为任务名称（即类名去除 `bootcamp` 后缀）。
|
||||
- `sample_number` 为生成样本数量。
|
||||
- `config_file` 为配置文件名。
|
||||
- `bootcamp_cls_name` 为任务类名。
|
||||
|
||||
或运行脚本 [quickgen_data_configs.py](/examples/pipelines/quickgen_data_configs.py)，默认可自动将 `examples/pipelines/puzzle_configs` 下配置的 Bootcamp 生成对应的 Pipeline 配置并覆盖保存于 `examples/pipelines/data_configs`。
|
||||
|
||||
**通过以上步骤，您可以成功创建、测试并集成一个自定义的 Bootcamp 任务！**
|
||||
Loading…
Add table
Add a link
Reference in a new issue