diff --git a/environments/README.md b/environments/README.md index 3d8ddf23..ef467152 100644 --- a/environments/README.md +++ b/environments/README.md @@ -88,6 +88,44 @@ You are a deep thinking AI, you may use extremely long chains of thought to deep - Linear penalty scaling from 1.0 down to 0.0 for responses between 50% and 100% of max length - Returns None if all scores are identical (no learning signal) +--- + +### Instruction Following Environment (`instruction_following_algorithm_environment.py`) + +Environment for training models to follow natural language instructions and constraints, based on the `allenai/RLVR-IFeval` dataset and environment. + +**Input Format:** +- Each item from the processed `allenai/RLVR-IFeval` dataset contains: + - `prompt`: The user's instruction string. + - `func_name`: The string name of the verifier function (from a predefined map) used to check if the instruction is followed. + - `args`: A dictionary of arguments for the specified verifier function. + +**System Prompt:** +``` +You are a deep thinking AI, you may use extremely long chains of thought to deeply consider the problem and deliberate with yourself via systematic reasoning processes to help come to a correct solution prior to answering. You should enclose your thoughts and internal monologue inside tags, and then provide your solution or response to the problem. +``` + +**Reward Function:** +- Score of 1.0 if the model's response correctly follows the instruction, as determined by the specific verifier function associated with the input prompt. +- Score of 0.0 if the response fails the verifier function. +- Length penalty applied if all responses in a batch are correct (receive a score of 1.0 before penalty): + - No penalty for responses under a certain percentage (e.g., 75%) of max token length. + - Linear penalty scaling from 1.0 down to 0.0 for responses between the threshold and 100% of max length. + - Returns None if all scores are identical after potential penalties (no learning signal). + +**Unique Configuration and Features:** +- **Dataset Configuration (`IFConfig`): + - `dataset_name`: Specifies the primary dataset to use (defaults to `allenai/RLVR-IFeval`). + - `dataset_config_name`: Optional name for a specific configuration or subset of the dataset. + - `test_set_ratio`: Defines the proportion of the dataset reserved for testing (defaults to 5%). + +- **Verifier-Based Scoring:** Utilizes a comprehensive map of verifier functions (`IF_FUNCTIONS_MAP`) to evaluate whether the model's +output adheres to diverse and specific constraints defined in the input instructions (e.g., keyword presence, response length, JSON format, etc.). + +- **Specialized Dataset Processing:** The `setup` method is specifically designed to parse the `allenai/RLVR-IFeval` dataset, extracting user instructions, the corresponding verifier function name, and its arguments. + +- **Fallback Mechanism:** Includes a fallback to a small, predefined dummy dataset if the primary dataset (`allenai/RLVR-IFeval`) cannot be loaded, ensuring operational continuity for testing or development. + ## Common Features All environments share these common features: