lint

examples/word_ladder/README.md

@@ -6,21 +6,21 @@ This project generates a dataset of word ladder puzzles and (optionally) submits
 
 The project consists of several key components:
 
-- **`main.py`**:  
+- **`main.py`**:
   Orchestrates the overall flow. It performs the following tasks:
   1. Generates a dataset of word ladder puzzles by calling functions from `utils/create_word_ladders.py`.
   2. (Optionally) Triggers the reasoning request process to augment puzzles with chain-of-thought reasoning via `utils/generate_reasoning.py`.
   3. (Planned) Additional steps such as checking results or uploading the final dataset.
-  
+
   The configuration for the dataset parameters (e.g., word length, chain length, and dataset size) is centralized here, making it easy to adjust the settings as needed.
 
-- **`utils/create_word_ladders.py`**:  
+- **`utils/create_word_ladders.py`**:
   Contains functions to create and validate a word ladder dataset. It leverages underlying modules (e.g., `reasoning_gym`) to generate individual puzzles and ensures uniqueness across the dataset.
 
-- **`utils/generate_reasoning.py`**:  
+- **`utils/generate_reasoning.py`**:
   Reads the generated dataset (in JSONL format), then filters out puzzles that already have reasoning. For puzzles missing chain-of-thought data, it splits them into batches (with a default batch size that you can adjust) and submits each batch to Anthropic's Message Batches API. Each API request includes the puzzle along with a custom system prompt (read from `system_prompt.txt`), and the resulting metadata is stored for later retrieval and analysis.
 
-- **`usage_stats.py`**:  
+- **`usage_stats.py`**:
   Analyzes API response files to compute detailed usage statistics. This script:
   - Extracts token usage metrics such as `input_tokens`, `cache_creation_input_tokens`, `cache_read_input_tokens`, and `output_tokens`.
   - Calculates costs based on pricing data and shows the savings achieved through prompt caching.
@@ -29,23 +29,23 @@ The project consists of several key components:
 
 ## Warning
 
-**Caution:**  
+**Caution:**
 Running large batches of requests via the Anthropic API (especially in `generate_reasoning.py`) can incur significant costs in Anthropic credits. **Please review and understand your API quota and budgeting before running the API call.** If you are just testing or working with a demo dataset, ensure you adjust the batch size or dataset size appropriately to avoid unexpected charges.
 
 ## Prerequisites
 
 - **Python Version:** Python 3.7+
-- **Dependencies:**  
+- **Dependencies:**
   - `tqdm`
   - `anthropic`
   - `reasoning_gym`
-- **Environment Variables:**  
+- **Environment Variables:**
   For generating reasoning batches, set your Anthropic API key:
   ```bash
   export ANTROPIC_API_KEY=your_api_key_here
   ```
 
-## Directory Structure 
+## Directory Structure
 
 ```
 examples/word_ladder/
@@ -62,15 +62,15 @@ examples/word_ladder/
 
 The dataset generation parameters are centralized in `main.py` under the `config` dictionary. You can adjust settings like:
 
-- **Word Length:**  
+- **Word Length:**
   - `min_word_length`
   - `max_word_length`
-  
-- **Chain Length:**  
+
+- **Chain Length:**
   - `min_chain_length` (e.g., set to -1 for the shortest possible chain)
   - `max_chain_length`
-  
-- **Dataset Size:**  
+
+- **Dataset Size:**
   - `size` — the number of puzzles to generate (e.g., `1000` for a demo)
 
 ## How to Run
@@ -114,19 +114,18 @@ The dataset generation parameters are centralized in `main.py` under the `config
 
 ## Troubleshooting
 
-- **File Paths:**  
+- **File Paths:**
   Verify that `system_prompt.txt` is in the `/examples/word_ladder` folder as expected. The modules use paths relative to their location.
-  
-- **Environment Variables:**  
+
+- **Environment Variables:**
   Make sure your `ANTHROPIC_API_KEY` is set correctly when submitting API requests.
-  
-- **Output Directory Permissions:**  
+
+- **Output Directory Permissions:**
   Ensure the `output` directory exists and is writable by your user.
-  
-- **Cost Monitoring:**  
+
+- **Cost Monitoring:**
   Check your Anthropic API usage and account balance before running large batches to avoid unexpected costs.
 
 ## License
 
 This project is licensed under the MIT License.
-

examples/word_ladder/main.py

@@ -6,10 +6,10 @@ main.py – Orchestrates the overall flow:
 3. Upload the final dataset to HuggingFace Hub (if needed)
 """
 
-import uuid
 import sys
+import uuid
 from pathlib import Path
-from typing import Dict, Any
+from typing import Any, Dict
 
 from examples.word_ladder.utils import create_word_ladders, generate_reasoning
 
@@ -17,7 +17,7 @@ from examples.word_ladder.utils import create_word_ladders, generate_reasoning
 def create_dataset(jsonl_path: Path, config: Dict[str, Any]) -> bool:
     """
     Creates the word ladder dataset, handling potential exhaustion gracefully.
-    
+
     Returns:
         bool: True if dataset was created (even if truncated), False if creation failed
     """
@@ -25,7 +25,7 @@ def create_dataset(jsonl_path: Path, config: Dict[str, Any]) -> bool:
         print("Step 1: Algorithmically creating word ladder chains...")
         create_word_ladders.create_word_ladder_dataset(str(jsonl_path), config=config)
         return True
-        
+
     except IndexError as e:
         # Dataset was exhausted but some examples were generated
         print("\nNote: Dataset generation stopped early due to exhaustion of unique puzzles.")
@@ -34,23 +34,24 @@ def create_dataset(jsonl_path: Path, config: Dict[str, Any]) -> bool:
             print("Continuing with the partial dataset that was successfully generated.")
             return True
         return False
-        
+
     except Exception as e:
         # Unexpected error during dataset creation
         print(f"\nError: Failed to create dataset: {str(e)}")
         return False
 
+
 def main():
     # Centralized configuration for the dataset
     config = {
-        'dataset_name': 'word_ladder',
-        'dataset_config': {
-            'min_word_length': 3,
-            'max_word_length': 5,
-            'min_chain_length':-1,  # set to -1 for the shortest possible path
-            'max_chain_length':10,
-            'size': 100,  # Generate a small-ish dataset for demonstration
-        }
+        "dataset_name": "word_ladder",
+        "dataset_config": {
+            "min_word_length": 3,
+            "max_word_length": 3,
+            "min_chain_length": -1,  # set to -1 for the shortest possible path
+            "max_chain_length": 7,
+            "size": 2000,  # Generate a small-ish dataset for demonstration
+        },
     }
 
     # Generate a friendly unique identifier and compose the file path
@@ -64,21 +65,20 @@ def main():
         print("Exiting due to dataset creation failure.")
         sys.exit(1)
 
-
     # Step 2: Generate reasoning
-    '''
+
     try:
         print("\nStep 2: Submitting reasoning batches for the dataset...")
         generate_reasoning.submit_reasoning_batches(input_path=str(jsonl_path))
     except Exception as e:
         print(f"\nError: Failed to submit reasoning batches: {str(e)}")
         sys.exit(1)
-    '''
 
     # Step 3: Check Anthropic batch results
     # Step 4: Upload to HuggingFace 🤗
-    
+
     print("\nComplete!")
 
+
 if __name__ == "__main__":
-    main() 
+    main()

examples/word_ladder/output/example_msgbatch_01J7Dev6petQwq41gNpeGhUK_results.jsonl

@@ -2,4 +2,4 @@
 {"custom_id":"CONKS_GRAYS_1","result":{"type":"succeeded","message":{"id":"msg_01JrxHJyxFxsTVsn8huufv8T","type":"message","role":"assistant","model":"claude-3-5-sonnet-20241022","content":[{"type":"text","text":"<think>\nAlright, let me try to transform CONKS into GRAYS by changing one letter at a time. First, let me write down both words to analyze what needs to change:\n\nCONKS\nGRAYS\n\nLooking at the letters that need to change:\nC needs to become G\nO needs to become R\nN needs to become A\nK needs to become Y\nS stays S\n\nHmm, four letters need to change, but S stays the same at the end. Let me start by looking at possible changes from CONKS.\n\nFirst letter options from CONKS:\n- Change C to B: BONKS (valid)\n- Change C to D: DONKS (valid)\n- Change C to G: GONKS (is this valid? Not sure)\n- Change C to H: HONKS (valid)\n\nSecond letter options:\n- Change O to A: CANKS (is this valid? Not sure)\n- Change O to I: CINKS (probably not valid)\n- Change O to O: stays same\n- Change O to U: CUNKS (not sure if valid)\n\nWait, maybe instead of changing the first letter right away to G, let's try working with the O. What if we change O to another letter that might help us get closer? \n\nLet me try: CONKS → COOKS (changing N to O)\nYes! COOKS is definitely a valid word. That feels like progress.\n\nFrom COOKS, what can we do?\n- Change first C to G? GOOKS (probably not appropriate)\n- Change second O to R? No, need to keep both O's for now\n- Change K to something? COOLS? No, moving away from target\n- Change S? No, need to keep S\n\nWhat about changing one of the O's to something? COOKS → COOPS (changing K to P)\nYes! COOPS is valid (like chicken coops). This feels promising.\n\nNow from COOPS:\n- Change C to G? GOOPS (not sure if valid)\n- Change first O to R? CROPS! Yes, that works!\n\nCROPS is a valid word. Now we're getting somewhere. Let's see what we can do from CROPS:\n- Change O to A? CRAPS! That's valid and gets us closer to GRAYS\n\nFrom CRAPS:\n- Change P to Y? CRAYS! Perfect!\n\nAnd finally, from CRAYS:\n- Change C to G: GRAYS! We've reached the target!\n\nLet me verify this path:\n1. CONKS → COOKS (N to O)\n2. COOKS → COOPS (K to P)\n3. COOPS → CROPS (O to R)\n4. CROPS → CRAPS (O to A)\n5. CRAPS → CRAYS (P to Y)\n6. CRAYS → GRAYS (C to G)\n\nLet me check each word again:\nCONKS - valid (to hit on head)\nCOOKS - valid\nCOOPS - valid (enclosures)\nCROPS - valid\nCRAPS - valid\nCRAYS - valid (types of fish)\nGRAYS - valid\n\nThis seems to work perfectly! Each step changes exactly one letter, and all words are valid English words. Let me double-check if there might be a shorter path.\n\nCould we go CONKS → CONES? No, that doesn't help get to GRAYS.\nWhat about CONKS → CRANKS? No, that's two letter changes.\nOr CONKS → CORKS? Again, doesn't seem to lead anywhere useful.\n\nThe path we found:\nCONKS → COOKS → COOPS → CROPS → CRAPS → CRAYS → GRAYS\nseems optimal because:\n1. Each step is a valid word\n2. Each change moves us closer to our target\n3. We maintain the final S throughout\n4. We gradually transform the word while keeping it valid at each step\n\nYes, I think this is the solution. Six steps, changing one letter at a time, all valid words, and each change gets us closer to our target GRAYS.\n</think>"}],"stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":64,"cache_creation_input_tokens":0,"cache_read_input_tokens":6434,"output_tokens":956}}}}
 {"custom_id":"WINGS_WORMY_2","result":{"type":"succeeded","message":{"id":"msg_01KXJZXTZgQoTnLvMqHphEcS","type":"message","role":"assistant","model":"claude-3-5-sonnet-20241022","content":[{"type":"text","text":"<think>\nOkay, let me tackle this word ladder from WINGS to WORMY. First, I'll write both words down and analyze what letters need to change:\n\nWINGS\nWORMY\n\nLooking at the letters:\nW stays the same\nI needs to become O\nN needs to become R\nG needs to become M\nS needs to become Y\n\nSo four letters need to change, keeping only the W. Let me start exploring possible paths by changing one letter at a time.\n\nStarting with WINGS, what can I change? Let's look at each position:\n- First letter (W): stays the same, so no need to change\n- Second letter (I): could become A, E, O, U\n- Third letter (N): could become other consonants\n- Fourth letter (G): could become other consonants\n- Fifth letter (S): could become other letters\n\nLet me try changing I first since it needs to become O eventually:\nWINGS → WONGS? Is that a word? Not sure. \n\nMaybe change N first:\nWINGS → WIGS? No, that removes a letter. Wait, I need to keep it 5 letters.\nWINGS → WINGS (same)\nWINGS → WINKS (changing G to K)! Yes, WINKS is definitely a valid word.\n\nOkay, so WINGS → WINKS works. Now from WINKS, what's next? I still need to:\n- Change I to O\n- Change K to R\n- Change S to Y\n\nLet me try changing I to O in WINKS:\nWINKS → WONKS\nIs WONKS a word? I think it is - it's the plural of wonk, someone who studies a subject extensively. Yes, that works!\n\nSo now I have:\nWINGS → WINKS → WONKS\n\nFrom WONKS, I need to:\n- Change K to R\n- Change S to Y\n\nLet me try changing K to R:\nWONKS → WORKS\nPerfect! WORKS is definitely a valid word.\n\nSo now:\nWINGS → WINKS → WONKS → WORKS\n\nFrom WORKS, I still need to:\n- Change S to Y\n\nBut wait, looking at the target WORMY, I notice I need M before Y. So from WORKS, maybe I should change S to M first?\n\nWORKS → WORMS\nYes! WORMS is a valid word.\n\nNow I have:\nWINGS → WINKS → WONKS → WORKS → WORMS\n\nFinally, from WORMS to WORMY, just need to change S to Y:\nWORMS → WORMY\n\nLet me verify the complete path:\n1. WINGS → WINKS (change G to K)\n2. WINKS → WONKS (change I to O)\n3. WONKS → WORKS (change K to R)\n4. WORKS → WORMS (change K to M)\n5. WORMS → WORMY (change S to Y)\n\nLet me check if each word is valid:\n- WINGS: valid (plural of wing)\n- WINKS: valid (plural of wink)\n- WONKS: valid (plural of wonk)\n- WORKS: valid (plural of work)\n- WORMS: valid (plural of worm)\n- WORMY: valid (full of worms)\n\nCould there be a shorter path? Let me think...\nCould I go WINGS → WONGS (I to O)? But is WONGS a valid word? Not sure about that.\nOr WINGS → WRINGS (add R)? No, that would be adding a letter, not changing one.\nOr WINGS → WORMS directly? No, that would require changing multiple letters at once.\n\nThe path I found requires five steps, which makes sense because we needed to change four letters (I→O, N→R, G→M, S→Y) and couldn't combine any changes since we can only change one letter at a time.\n\nSo after exploring alternatives, I believe the path:\nWINGS → WINKS → WONKS → WORKS → WORMS → WORMY\nis optimal, with each step changing exactly one letter and forming a valid English word.\n</think>"}],"stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":58,"cache_creation_input_tokens":0,"cache_read_input_tokens":6434,"output_tokens":962}}}}
 {"custom_id":"PINTO_RASPS_3","result":{"type":"succeeded","message":{"id":"msg_01GJQuw2NwcJQpx2civyj8Fu","type":"message","role":"assistant","model":"claude-3-5-sonnet-20241022","content":[{"type":"text","text":"<think>\nAlright, I need to transform PINTO into RASPS by changing one letter at a time. Let me analyze both words first to understand what needs to change.\n\nPINTO\nRASPS\n\nLooking at the letters:\nP -> R\nI -> A\nN -> S\nT -> P\nO -> S\n\nHmm, all five letters need to change. This might be tricky. Let me start by exploring possible changes from PINTO.\n\nFirst letter (P) possibilities:\n- MINTO (is that a word? Maybe a proper noun, but that's not allowed)\n- RINTO (not a word)\n- TINTO (might be Spanish, not valid English)\n\nSecond letter (I) possibilities:\n- PANTO (yes, that's valid!)\n- PENTO (not sure if that's valid)\n- PUNTO (Spanish, not English)\n\nThird letter (N) possibilities:\n- PINTO -> PINTS (changing O to S)\nThat's interesting! PINTS is definitely valid. Maybe that's a good first step since we need to end with 'S' anyway.\n\nLet me try PINTO -> PINTS first.\nNow from PINTS, what can we do? We need to eventually get to RASPS, so maybe changing that I to A would help?\nPINTS -> PANTS (changing I to A)\nYes! PANTS is valid. Now we have:\nPINTO -> PINTS -> PANTS\n\nFrom PANTS, looking at the letters compared to RASPS:\nP -> R\nA (matches!)\nN -> S\nT -> P\nS (matches!)\n\nMaybe we can change that T to R? No, that wouldn't help because we still need P in that position eventually. What about changing N to R? No, we need S there. \n\nLet me try changing T to something... PANTS -> PARTS (changing N to R)\nYes! PARTS is valid. So now:\nPINTO -> PINTS -> PANTS -> PARTS\n\nFrom PARTS to RASPS, we need to:\n- Change P to R at the start\n- Change T to P\n- Keep the S\n\nWait, but if I change P to R now, how would I get the P back later? Maybe I need an intermediate step.\n\nWhat if... PARTS -> WARTS (changing P to W)?\nYes! That's valid. Then from WARTS, maybe... WARPS (changing T to P)?\nWARTS -> WARPS works! \n\nSo now I have:\nPINTO -> PINTS -> PANTS -> PARTS -> WARTS -> WARPS\n\nGetting closer! From WARPS to RASPS, I need to:\n- Change W to R\n- Change P to S in the middle\n\nBut wait, if I change W to R directly, would that work? Let me try another approach...\nWARPS -> WASPS (changing R to S)\nYes! That works. Then finally:\nWASPS -> RASPS (changing W to R)\n\nLet me verify the complete path:\n1. PINTO -> PINTS (O to S)\n2. PINTS -> PANTS (I to A)\n3. PANTS -> PARTS (N to R)\n4. PARTS -> WARTS (P to W)\n5. WARTS -> WARPS (T to P)\n6. WARPS -> WASPS (R to S)\n7. WASPS -> RASPS (W to R)\n\nChecking each word:\nPINTO - valid (type of horse/car)\nPINTS - valid (plural of pint)\nPANTS - valid (clothing)\nPARTS - valid (components)\nWARTS - valid (skin growths)\nWARPS - valid (bends/distorts)\nWASPS - valid (insects)\nRASPS - valid (coarse files/harsh sounds)\n\nEach step changes exactly one letter and creates a valid English word. Seven steps in total. Let me double-check if there might be a shorter path...\n\nCould I have gone from PINTS directly to PARTS? No, that would be changing two letters (I to A and N to R).\nCould I have skipped WARTS? If I went PARTS -> WARPS directly, that would change two letters (P to W and T to P).\nCould I have gone from WARPS to RASPS directly? No, that would change two letters (W to R and P to S).\n\nNo, each step seems necessary. Therefore, this seven-step solution appears to be valid and efficient:\nPINTO -> PINTS -> PANTS -> PARTS -> WARTS -> WARPS -> WASPS -> RASPS\n</think>"}],"stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":67,"cache_creation_input_tokens":6434,"cache_read_input_tokens":0,"output_tokens":1081}}}}
-{"custom_id":"GRACE_WAINS_4","result":{"type":"succeeded","message":{"id":"msg_01He4e1HXh1DXdxPz2j7K3GZ","type":"message","role":"assistant","model":"claude-3-5-sonnet-20241022","content":[{"type":"text","text":"<think>\nLet me tackle this word ladder puzzle. I need to transform GRACE into WAINS by changing just one letter at a time, making sure each step creates a valid English word.\n\nFirst, let me compare the starting and ending words:\nGRACE\nWAINS\n\nLooking at what needs to change:\nG needs to become W\nR needs to become A\nA needs to become I\nC needs to become N\nE needs to become S\n\nWow, all five letters need to change! This might be tricky. Let me start by looking at possible changes from GRACE.\n\nFrom GRACE, I could change:\n- First letter (G): BRACE, TRACE, SPACE... but I need to get to W eventually\n- Second letter (R): GRACE to GLACE? That's valid\n- Third letter (A): GRACE to GRECE? No, that's not a word\n- Fourth letter (C): GRACE to GRADE, GRAVE, GRAPE... wait, GRAPE is interesting\n\nLet me try GRACE → GRAPE (changing C to P)\nThat works! Now from GRAPE, I need to somehow get closer to WAINS.\n\nLooking at GRAPE vs WAINS:\nG still needs to become W\nR still needs to become A\nA needs to become I\nP needs to become N\nE needs to become S\n\nMaybe I can change the A to I next? GRAPE → GRIPE\nYes! GRIPE is a valid word. That's progress because now the middle letter matches our target word.\n\nFrom GRIPE:\nG still needs to become W\nR still needs to become A\nP needs to become N\nE needs to become S\n\nLooking at GRIPE... what can I change? The final E needs to become S eventually. But wait, looking at the P... if I change P to S, would that work? GRIPE → GRISE? No, that's not common. What about GRIPS? Yes! GRIPE → GRIPS works by changing E to S.\n\nNow I have GRIPS. Getting closer! Looking at what's left:\nG needs to become W\nR needs to become A\nI stays I\nP needs to become N\nS stays S\n\nFrom GRIPS, maybe I can change P to N? GRIPS → GRINS\nYes! That works. Now I have GRINS.\n\nLooking at GRINS vs WAINS:\nG needs to become W\nR needs to become A\nI stays I\nN stays N\nS stays S\n\nFrom GRINS, I need to change R to A. GRINS → GAINS\nPerfect! Now I just need to change G to W to reach the target.\n\nGAINS → WAINS\n\nLet me verify the complete path:\n1. GRACE → GRAPE (C→P)\n2. GRAPE → GRIPE (A→I)\n3. GRIPE → GRIPS (E→S)\n4. GRIPS → GRINS (P→N)\n5. GRINS → GAINS (R→A)\n6. GAINS → WAINS (G→W)\n\nLet me check each word:\nGRACE - valid\nGRAPE - valid (fruit)\nGRIPE - valid (complaint)\nGRIPS - valid (holds)\nGRINS - valid (smiles)\nGAINS - valid (increases)\nWAINS - valid (wagons)\n\nAnd each step only changes one letter. This works! Though let me just double-check if there might be a shorter path...\n\nCould I have gone from GRACE directly to WAINS in fewer steps? No, because five letters need to change, so minimum five steps needed. Could I have gone:\nGRACE → GRANS? (Not a word)\nGRACE → GRINS? (No, that would be two letters)\nGRACE → WRACE? (Not a word)\n\nNo, I think the path I found is optimal because:\n1. Each step is a valid word\n2. Each step changes only one letter\n3. We needed to change all five letters\n4. Each change gets us closer to the target\n\nTherefore, the solution path GRACE → GRAPE → GRIPE → GRIPS → GRINS → GAINS → WAINS is correct and efficient.\n</think>"}],"stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":61,"cache_creation_input_tokens":0,"cache_read_input_tokens":6434,"output_tokens":967}}}}
+{"custom_id":"GRACE_WAINS_4","result":{"type":"succeeded","message":{"id":"msg_01He4e1HXh1DXdxPz2j7K3GZ","type":"message","role":"assistant","model":"claude-3-5-sonnet-20241022","content":[{"type":"text","text":"<think>\nLet me tackle this word ladder puzzle. I need to transform GRACE into WAINS by changing just one letter at a time, making sure each step creates a valid English word.\n\nFirst, let me compare the starting and ending words:\nGRACE\nWAINS\n\nLooking at what needs to change:\nG needs to become W\nR needs to become A\nA needs to become I\nC needs to become N\nE needs to become S\n\nWow, all five letters need to change! This might be tricky. Let me start by looking at possible changes from GRACE.\n\nFrom GRACE, I could change:\n- First letter (G): BRACE, TRACE, SPACE... but I need to get to W eventually\n- Second letter (R): GRACE to GLACE? That's valid\n- Third letter (A): GRACE to GRECE? No, that's not a word\n- Fourth letter (C): GRACE to GRADE, GRAVE, GRAPE... wait, GRAPE is interesting\n\nLet me try GRACE → GRAPE (changing C to P)\nThat works! Now from GRAPE, I need to somehow get closer to WAINS.\n\nLooking at GRAPE vs WAINS:\nG still needs to become W\nR still needs to become A\nA needs to become I\nP needs to become N\nE needs to become S\n\nMaybe I can change the A to I next? GRAPE → GRIPE\nYes! GRIPE is a valid word. That's progress because now the middle letter matches our target word.\n\nFrom GRIPE:\nG still needs to become W\nR still needs to become A\nP needs to become N\nE needs to become S\n\nLooking at GRIPE... what can I change? The final E needs to become S eventually. But wait, looking at the P... if I change P to S, would that work? GRIPE → GRISE? No, that's not common. What about GRIPS? Yes! GRIPE → GRIPS works by changing E to S.\n\nNow I have GRIPS. Getting closer! Looking at what's left:\nG needs to become W\nR needs to become A\nI stays I\nP needs to become N\nS stays S\n\nFrom GRIPS, maybe I can change P to N? GRIPS → GRINS\nYes! That works. Now I have GRINS.\n\nLooking at GRINS vs WAINS:\nG needs to become W\nR needs to become A\nI stays I\nN stays N\nS stays S\n\nFrom GRINS, I need to change R to A. GRINS → GAINS\nPerfect! Now I just need to change G to W to reach the target.\n\nGAINS → WAINS\n\nLet me verify the complete path:\n1. GRACE → GRAPE (C→P)\n2. GRAPE → GRIPE (A→I)\n3. GRIPE → GRIPS (E→S)\n4. GRIPS → GRINS (P→N)\n5. GRINS → GAINS (R→A)\n6. GAINS → WAINS (G→W)\n\nLet me check each word:\nGRACE - valid\nGRAPE - valid (fruit)\nGRIPE - valid (complaint)\nGRIPS - valid (holds)\nGRINS - valid (smiles)\nGAINS - valid (increases)\nWAINS - valid (wagons)\n\nAnd each step only changes one letter. This works! Though let me just double-check if there might be a shorter path...\n\nCould I have gone from GRACE directly to WAINS in fewer steps? No, because five letters need to change, so minimum five steps needed. Could I have gone:\nGRACE → GRANS? (Not a word)\nGRACE → GRINS? (No, that would be two letters)\nGRACE → WRACE? (Not a word)\n\nNo, I think the path I found is optimal because:\n1. Each step is a valid word\n2. Each step changes only one letter\n3. We needed to change all five letters\n4. Each change gets us closer to the target\n\nTherefore, the solution path GRACE → GRAPE → GRIPE → GRIPS → GRINS → GAINS → WAINS is correct and efficient.\n</think>"}],"stop_reason":"end_turn","stop_sequence":null,"usage":{"input_tokens":61,"cache_creation_input_tokens":0,"cache_read_input_tokens":6434,"output_tokens":967}}}}

examples/word_ladder/output/example_word_ladders_029573f4.jsonl

@@ -7,4 +7,4 @@
 {"question": "Transform the word ladder 'LAIRS' to 'FUELS' by changing one letter at a time.", "answer": "LAIRS,FAIRS,FAILS,FARLS,FURLS,FUELS", "reasoning": null, "metadata": {"start_word": "LAIRS", "end_word": "FUELS", "word_length": 5, "chain_length": 6}}
 {"question": "Transform the word ladder 'WINKS' to 'TENTH' by changing one letter at a time.", "answer": "WINKS,PINKS,PINTS,TINTS,TENTS,TENTH", "reasoning": null, "metadata": {"start_word": "WINKS", "end_word": "TENTH", "word_length": 5, "chain_length": 6}}
 {"question": "Transform the word ladder 'VERBS' to 'DATES' by changing one letter at a time.", "answer": "VERBS,HERBS,HERDS,HARDS,HARES,HATES,DATES", "reasoning": null, "metadata": {"start_word": "VERBS", "end_word": "DATES", "word_length": 5, "chain_length": 7}}
-{"question": "Transform the word ladder 'SCENT' to 'HEARS' by changing one letter at a time.", "answer": "SCENT,SCANT,SCART,SCARS,SEARS,HEARS", "reasoning": null, "metadata": {"start_word": "SCENT", "end_word": "HEARS", "word_length": 5, "chain_length": 6}}
+{"question": "Transform the word ladder 'SCENT' to 'HEARS' by changing one letter at a time.", "answer": "SCENT,SCANT,SCART,SCARS,SEARS,HEARS", "reasoning": null, "metadata": {"start_word": "SCENT", "end_word": "HEARS", "word_length": 5, "chain_length": 6}}

examples/word_ladder/requirements.txt

@@ -1,6 +1,5 @@
 # Required dependencies for the Word Ladder Puzzle Dataset Generator
 # Python 3.7+
 
-anthropic>=0.45.2        # Client library for interacting with Anthropic's API
+anthropic>=0.45.2       # Client library for interacting with Anthropic's API
 tqdm>=4.67.1            # For progress bars
-

examples/word_ladder/system_prompt.txt

@@ -1,8 +1,8 @@
-Word Ladder puzzles involve transforming a start word into an end word. You are allowed to change only one letter a time and you must keep the number of letters constant. 
-Each time you change one letter the word in the chain must be forming a new word that's valid in English. Plurals are allowed, but not proper nouns. 
+Word Ladder puzzles involve transforming a start word into an end word. You are allowed to change only one letter a time and you must keep the number of letters constant.
+Each time you change one letter the word in the chain must be forming a new word that's valid in English. Plurals are allowed, but not proper nouns.
 Your task will be to simulate a detailed step-by-step chain of thought reasoning of the transformation process from a start word to the end word of a word ladder.
 You will be given the word ladder question, as well as the correct solution path. So you don't need to solve the problem, you have the solution already, which should be your 'cheat sheet' to refer to as you go along.
-Your task is to provide a perfectly simulated chain of thought reasoning exactly in the style of the example below, 
+Your task is to provide a perfectly simulated chain of thought reasoning exactly in the style of the example below,
 including considering multiple possibilities, backtracking where needed, validating words, and showing the final path.
 Remember you're NOT trying to 'find' the shortest path necessarily. Make sure you're not lazy and show the user that you're working hard to find the intermediate words in the chain, and that you haven't gotten simply lucky in finding the steps.
 For example, you may go down a path that's no longer getting you closer to the solution, so you backtrack and try a different path, which ends up working.
@@ -285,4 +285,4 @@ Yes, this works. Therefore, the sequence is BALL → BOLL → POLL → POLE →
 
 
 Finally, I've seen in your previous answers to similar questions that you sometimes get complacent and deviate from the hard work required. You must maintain the style and level effort throughout the entirety of your answer until the end.
-Now, the user will provide you with the word ladder question and the correct solution path. Provide the user with a very verbose, first person chain of thought reasoning that is simulating how a human would methodically think to solve the word ladder step by step.
+Now, the user will provide you with the word ladder question and the correct solution path. Provide the user with a very verbose, first person chain of thought reasoning that is simulating how a human would methodically think to solve the word ladder step by step.

examples/word_ladder/tests/test_generate_reasoning.py

@@ -1,11 +1,12 @@
-import os
-import json
-import time
 import datetime
-import pytest
+import json
+import os
+import time
 from pathlib import Path
 from types import SimpleNamespace
 
+import pytest
+
 from examples.word_ladder import generate_reasoning
 
 # We alias the functions and globals for easier usage in our tests.
@@ -16,6 +17,7 @@ COMMON_UUID = generate_reasoning.COMMON_UUID
 BATCH_SIZE = generate_reasoning.BATCH_SIZE
 client = generate_reasoning.client
 
+
 # Define a mock batch response class mimicking Anthropic's API response.
 class MockBatchResponse:
     def __init__(self, batch_id="msgbatch_mock", processing_status="in_progress", fail=False):
@@ -23,13 +25,7 @@ class MockBatchResponse:
         self.type = "message_batch"
         self.processing_status = processing_status
         # Make request_counts a SimpleNamespace object with the required attributes
-        self.request_counts = SimpleNamespace(
-            processing=0,
-            succeeded=0,
-            errored=0,
-            canceled=0,
-            expired=0
-        )
+        self.request_counts = SimpleNamespace(processing=0, succeeded=0, errored=0, canceled=0, expired=0)
         self.ended_at = None
         # Use datetime objects so that isoformat() is available
         self.created_at = datetime.datetime.utcnow().replace(tzinfo=datetime.timezone.utc)
@@ -37,52 +33,68 @@ class MockBatchResponse:
         self.cancel_initiated_at = None
         self.results_url = None
 
+
 # Helper: Create a temporary system prompt file.
 @pytest.fixture
 def system_prompt_file(tmp_path, monkeypatch):
     prompt_text = "This is a system prompt."
     sys_file = tmp_path / "system_prompt.txt"
     sys_file.write_text(prompt_text, encoding="utf-8")
-    
+
     # Monkeypatch the system prompt path
     monkeypatch.setattr(generate_reasoning, "DEFAULT_SYSTEM_PROMPT", str(sys_file))
     return sys_file
 
+
 # Helper: Create necessary directories using a temporary location.
 @pytest.fixture
 def setup_directories(tmp_path, monkeypatch):
     # Create output directory in temporary path
     output_dir = tmp_path / "output"
     output_dir.mkdir(exist_ok=True)
-    
+
     # Monkeypatch the DEFAULT_OUTPUT_DIR to be a Path (temporary directory).
     monkeypatch.setattr(generate_reasoning, "DEFAULT_OUTPUT_DIR", output_dir)
-    
+
     # Ensure we're working in the temporary directory
     monkeypatch.chdir(tmp_path)
     return output_dir
 
+
 # Helper: Create a temporary input JSONL file with given entries.
 @pytest.fixture
 def input_jsonl_file(tmp_path, setup_directories, monkeypatch):
     # Create input file in temporary directory
     file_path = setup_directories / "word_ladder_examples.jsonl"
     entries = [
-        { "question": "Transform 'A' to 'B'", "answer": "A,X,B", "reasoning": None,
-          "metadata": { "start_word": "A", "end_word": "B", "word_length": 1, "chain_length": 3 } },
-        { "question": "Transform 'C' to 'D'", "answer": "C,Y,D", "reasoning": "Some reasoning",
-          "metadata": { "start_word": "C", "end_word": "D", "word_length": 1, "chain_length": 3 } },
-        { "question": "Transform 'E' to 'F'", "answer": "E,Z,F", "reasoning": None,
-          "metadata": { "start_word": "E", "end_word": "F", "word_length": 1, "chain_length": 3 } }
+        {
+            "question": "Transform 'A' to 'B'",
+            "answer": "A,X,B",
+            "reasoning": None,
+            "metadata": {"start_word": "A", "end_word": "B", "word_length": 1, "chain_length": 3},
+        },
+        {
+            "question": "Transform 'C' to 'D'",
+            "answer": "C,Y,D",
+            "reasoning": "Some reasoning",
+            "metadata": {"start_word": "C", "end_word": "D", "word_length": 1, "chain_length": 3},
+        },
+        {
+            "question": "Transform 'E' to 'F'",
+            "answer": "E,Z,F",
+            "reasoning": None,
+            "metadata": {"start_word": "E", "end_word": "F", "word_length": 1, "chain_length": 3},
+        },
     ]
     with file_path.open("w", encoding="utf-8") as f:
         for entry in entries:
             f.write(json.dumps(entry) + "\n")
-    
+
     # Monkeypatch DEFAULT_INPUT_JSONL to point to our temporary test file.
     monkeypatch.setattr(generate_reasoning, "DEFAULT_INPUT_JSONL", str(file_path))
     return file_path
 
+
 # Test that submit_reasoning_batches builds a batch skipping entries with existing reasoning.
 def test_submit_batches_success(system_prompt_file, input_jsonl_file, setup_directories, monkeypatch):
     def fake_create(requests):
@@ -112,17 +124,16 @@ def test_submit_batches_success(system_prompt_file, input_jsonl_file, setup_dire
             assert temperature == 0.5, "Incorrect temperature"
             assert "C_D_" not in custom_id
         return MockBatchResponse(batch_id="msgbatch_test_success")
-    
+
     monkeypatch.setattr(client.messages.batches, "create", fake_create)
-    
+
     batch_metadata_prefix = "test_metadata"
-    submit_reasoning_batches(input_path=str(input_jsonl_file),
-                             batch_metadata_prefix=batch_metadata_prefix)
-    
+    submit_reasoning_batches(input_path=str(input_jsonl_file), batch_metadata_prefix=batch_metadata_prefix)
+
     metadata_filename = f"{COMMON_UUID}_{batch_metadata_prefix}.jsonl"
     meta_file_path = setup_directories / metadata_filename
     assert meta_file_path.exists(), "Metadata file was not created as expected."
-    
+
     with meta_file_path.open("r", encoding="utf-8") as f:
         lines = f.readlines()
         # Expecting only those entries that did not already have a reasoning.
@@ -136,61 +147,65 @@ def test_submit_batches_success(system_prompt_file, input_jsonl_file, setup_dire
             custom_ids = metadata["custom_ids"]
             assert len(custom_ids) == 2
 
+
 # Test that _submit_single_batch retries once and eventually succeeds.
 def test_retry_logic(system_prompt_file, setup_directories, monkeypatch):
     call_count = {"count": 0}
+
     def fake_create_retry(requests):
         if call_count["count"] == 0:
             call_count["count"] += 1
             raise Exception("Temporary failure")
         return MockBatchResponse(batch_id="msgbatch_retry_success")
-    
+
     monkeypatch.setattr(client.messages.batches, "create", fake_create_retry)
-    
+
     dummy_request = type("DummyRequest", (), {"custom_id": "dummy_1"})()
     batch_requests = [dummy_request]
     custom_ids = ["dummy_1"]
-    
+
     _submit_single_batch(batch_requests, custom_ids, 0, "test_retry", "dummy_input.jsonl")
-    
+
     metadata_filename = f"{COMMON_UUID}_test_retry.jsonl"
     meta_file_path = setup_directories / metadata_filename
     assert meta_file_path.exists(), "Retry metadata file was not created."
-    
+
     with meta_file_path.open("r", encoding="utf-8") as f:
         metadata = json.loads(f.read())
         assert metadata["api_response"]["id"] == "msgbatch_retry_success"
-    
+
     assert call_count["count"] == 1
 
+
 # Test that when all attempts to submit a batch fail, the error is logged to the failed file.
 def test_failed_batch(system_prompt_file, setup_directories, monkeypatch):
     def fake_create_fail(requests):
         raise Exception("Permanent failure")
-    
+
     monkeypatch.setattr(client.messages.batches, "create", fake_create_fail)
-    
+
     dummy_request = type("DummyRequest", (), {"custom_id": "dummy_fail"})()
     batch_requests = [dummy_request]
     custom_ids = ["dummy_fail"]
-    
+
     _submit_single_batch(batch_requests, custom_ids, 0, "test_failed", "dummy_input.jsonl")
-    
+
     error_filename = f"{COMMON_UUID}_failed_batches.jsonl"
     error_file_path = setup_directories / error_filename
     assert error_file_path.exists(), "Failed batch log file was not created."
-    
+
     with error_file_path.open("r", encoding="utf-8") as f:
         error_entry = json.loads(f.readline())
         assert error_entry["batch_number"] == 0
         assert "Permanent failure" in error_entry["error"]
         assert error_entry["batch_requests"] == ["dummy_fail"]
 
+
 # Test batching behavior when multiple batches are needed.
 def test_multiple_batches(system_prompt_file, setup_directories, monkeypatch):
     test_batch_size = 2
     monkeypatch.setattr(generate_reasoning, "BATCH_SIZE", test_batch_size)
-    
+
     # Create input file
     input_file = setup_directories / "word_ladder_examples.jsonl"
     entries = [
@@ -198,42 +213,42 @@ def test_multiple_batches(system_prompt_file, setup_directories, monkeypatch):
             "question": f"Transform word ladder {idx}",
             "answer": f"start,mid,end_{idx}",
             "reasoning": None,
-            "metadata": {"start_word": f"start_{idx}", "end_word": f"end_{idx}"}
+            "metadata": {"start_word": f"start_{idx}", "end_word": f"end_{idx}"},
         }
         for idx in range(5)
     ]
-    
+
     with input_file.open("w", encoding="utf-8") as f:
         for entry in entries:
             f.write(json.dumps(entry) + "\n")
-    
+
     # Monkeypatch DEFAULT_INPUT_JSONL.
     monkeypatch.setattr(generate_reasoning, "DEFAULT_INPUT_JSONL", str(input_file))
-    
+
     batch_ids = []
+
     def fake_create(requests):
         new_id = f"msgbatch_batch_{len(batch_ids)}"
         batch_ids.append(new_id)
         return MockBatchResponse(batch_id=new_id)
-    
+
     monkeypatch.setattr(client.messages.batches, "create", fake_create)
-    
+
     batch_metadata_prefix = "test_multi"
-    submit_reasoning_batches(input_path=str(input_file),
-                             batch_metadata_prefix=batch_metadata_prefix)
-    
+    submit_reasoning_batches(input_path=str(input_file), batch_metadata_prefix=batch_metadata_prefix)
+
     metadata_filename = f"{COMMON_UUID}_{batch_metadata_prefix}.jsonl"
     meta_file_path = setup_directories / metadata_filename
     assert meta_file_path.exists(), "Multiple batch metadata file was not created."
-    
+
     with meta_file_path.open("r", encoding="utf-8") as f:
         metadata_lines = f.readlines()
     # With 5 qualifying entries and a batch size of 2 we expect 3 batches.
     assert len(metadata_lines) == 3
-    
+
     seen_custom_ids = []
     for line in metadata_lines:
         metadata = json.loads(line)
         seen_custom_ids.extend(metadata["custom_ids"])
         assert metadata["api_response"]["id"] in batch_ids
-    assert len(seen_custom_ids) == 5 
+    assert len(seen_custom_ids) == 5

examples/word_ladder/utils/create_word_ladders.py

@@ -11,29 +11,30 @@ from tqdm import tqdm
 
 import reasoning_gym
 
+
 def check_duplicates(jsonl_path: str) -> tuple[bool, dict]:
     """
     Check for duplicate word pairs in a word ladder JSONL file.
-    
+
     Returns:
         tuple[bool, dict]: (has_duplicates, valid_entries) where:
             - has_duplicates: True if any duplicates were found
             - valid_entries: Dict mapping line_number -> data for non-duplicate entries
-            
+
     Note: A pair is considered duplicate if either (word1, word2) or (word2, word1)
     already exists, since word ladder paths are bidirectional.
     """
     pairs_seen = {}  # (start, end) -> (line_number, data)
     valid_entries = {}
     duplicates_found = False
-    
-    with open(jsonl_path, 'r', encoding='utf-8') as f:
+
+    with open(jsonl_path, "r", encoding="utf-8") as f:
         for line_num, line in enumerate(f):
             data = json.loads(line)
-            metadata = data['metadata']
-            pair = (metadata['start_word'], metadata['end_word'])
-            reverse_pair = (metadata['end_word'], metadata['start_word'])
-            
+            metadata = data["metadata"]
+            pair = (metadata["start_word"], metadata["end_word"])
+            reverse_pair = (metadata["end_word"], metadata["start_word"])
+
             # Check both orientations of the pair
             if pair in pairs_seen or reverse_pair in pairs_seen:
                 duplicates_found = True
@@ -43,9 +44,10 @@ def check_duplicates(jsonl_path: str) -> tuple[bool, dict]:
                 # Store both the line number and data for valid entries
                 pairs_seen[pair] = (line_num, data)
                 valid_entries[line_num] = data
-    
+
     return duplicates_found, valid_entries
 
+
 def create_word_ladder_dataset(jsonl_path: str = None, config: dict = None) -> None:
     """
     Creates a word ladder dataset and writes each sample as a JSON line.
@@ -65,67 +67,67 @@ def create_word_ladder_dataset(jsonl_path: str = None, config: dict = None) -> N
         jsonl_path = output_dir / f"word_ladders_{unique_id}.jsonl"
     else:
         jsonl_path = Path(jsonl_path)
-    
-    target_size = config['dataset_config']['size']
+
+    target_size = config["dataset_config"]["size"]
     current_size = 0
     max_attempts = 3  # Limit total regeneration attempts
     attempt = 0
-    
+
     # Initial generation
-    dataset = reasoning_gym.create_dataset(config['dataset_name'], **config['dataset_config'])
-    with open(jsonl_path, 'w', encoding='utf-8') as f:
+    dataset = reasoning_gym.create_dataset(config["dataset_name"], **config["dataset_config"])
+    with open(jsonl_path, "w", encoding="utf-8") as f:
         for item in tqdm(dataset, desc="Generating initial ladder examples"):
             row = {
-                'question': item['question'],
-                'answer': item['answer'],
-                'reasoning': None,
-                'metadata': item.get('metadata', {})
+                "question": item["question"],
+                "answer": item["answer"],
+                "reasoning": None,
+                "metadata": item.get("metadata", {}),
             }
-            f.write(json.dumps(row) + '\n')
-    
+            f.write(json.dumps(row) + "\n")
+
     while attempt < max_attempts:
         # Check entire file for duplicates
         has_duplicates, valid_entries = check_duplicates(jsonl_path)
         current_size = len(valid_entries)
-        
+
         if not has_duplicates and current_size == target_size:
             print(f"\nSuccessfully created dataset with {current_size} unique examples.")
             return
-        
+
         # If we have duplicates or not enough entries, regenerate the missing amount
         needed = target_size - current_size
         if needed > 0:
             print(f"\nAttempt {attempt + 1}: Regenerating {needed} examples to replace duplicates/missing entries...")
-            
+
             # Generate additional examples
-            config['dataset_config']['size'] = needed
-            additional_dataset = reasoning_gym.create_dataset(config['dataset_name'], **config['dataset_config'])
-            
+            config["dataset_config"]["size"] = needed
+            additional_dataset = reasoning_gym.create_dataset(config["dataset_name"], **config["dataset_config"])
+
             # Write all entries to a temporary file
-            temp_path = jsonl_path.with_suffix('.tmp')
-            with open(temp_path, 'w', encoding='utf-8') as f:
+            temp_path = jsonl_path.with_suffix(".tmp")
+            with open(temp_path, "w", encoding="utf-8") as f:
                 # Write existing valid entries
                 for data in valid_entries.values():
-                    f.write(json.dumps(data) + '\n')
-                
+                    f.write(json.dumps(data) + "\n")
+
                 # Write new entries
                 for item in additional_dataset:
                     row = {
-                        'question': item['question'],
-                        'answer': item['answer'],
-                        'reasoning': None,
-                        'metadata': item.get('metadata', {})
+                        "question": item["question"],
+                        "answer": item["answer"],
+                        "reasoning": None,
+                        "metadata": item.get("metadata", {}),
                     }
-                    f.write(json.dumps(row) + '\n')
-            
+                    f.write(json.dumps(row) + "\n")
+
             # Replace original file with temporary file
             temp_path.replace(jsonl_path)
-            
+
             # Note: We'll check for duplicates again at the start of the next loop
-        
+
         attempt += 1
-    
+
     if current_size < target_size:
         print(f"\nWarning: Could only generate {current_size} unique examples after {max_attempts} attempts.")
     else:
-        print(f"\nSuccessfully created dataset with {current_size} unique examples.")
+        print(f"\nSuccessfully created dataset with {current_size} unique examples.")

examples/word_ladder/utils/generate_reasoning.py

@@ -14,17 +14,16 @@ In our informal testing, Sonnet was deemed best performance value.
 You can swap out to another API, but this will need a rewrite to remove anthropic-specific code.
 """
 
-import os
 import json
-import uuid
+import os
 import time
+import uuid
 from pathlib import Path
 
-from tqdm import tqdm
-
 import anthropic
 from anthropic.types.message_create_params import MessageCreateParamsNonStreaming
 from anthropic.types.messages.batch_create_params import Request
+from tqdm import tqdm
 
 # Updated default output directory to use the parent directory.
 DEFAULT_OUTPUT_DIR = Path(__file__).resolve().parent.parent / "output"
@@ -36,18 +35,19 @@ BATCH_SIZE = 2500
 COMMON_UUID = uuid.uuid4().hex[:8]
 
 # Set up the Anthropic client (ensure the API key is set in the environment)
-client = anthropic.Anthropic(api_key=os.environ['ANTHROPIC_API_KEY'])
+client = anthropic.Anthropic(api_key=os.environ["ANTHROPIC_API_KEY"])
+
 
 def submit_reasoning_batches(
     input_path: str = DEFAULT_INPUT_JSONL,
     batch_metadata_prefix: str = "batch_metadata",
-    system_prompt_path: str = DEFAULT_SYSTEM_PROMPT
+    system_prompt_path: str = DEFAULT_SYSTEM_PROMPT,
 ) -> None:
     """
     Reads the input JSONL file of word ladder examples, builds batch requests for any example that
     does not have reasoning, splits them into groups of BATCH_SIZE, and submits each batch using
     Anthropic's Message Batches API.
-    
+
     Args:
         input_path: Path to input JSONL file
         batch_metadata_prefix: Prefix for batch metadata files
@@ -59,34 +59,36 @@ def submit_reasoning_batches(
 
     # Read the system prompt from file (used as a preamble for every request)
     with open(system_prompt_path, "r", encoding="utf-8") as sys_file:
-        system_message = [{
-            "type": "text",
-            "text": sys_file.read(),
-            "cache_control": {"type": "ephemeral"}  # Enable anthropic prompt caching
-        }]
+        system_message = [
+            {
+                "type": "text",
+                "text": sys_file.read(),
+                "cache_control": {"type": "ephemeral"},  # Enable anthropic prompt caching
+            }
+        ]
     batch_requests = []
     custom_ids = []  # List of custom_ids for the current batch
     batch_num = 0
 
     # Get the total number of lines in advance for tqdm progress bar.
     total_lines = sum(1 for _ in open(input_path))
-    
-    with open(input_path, 'r', encoding="utf-8") as infile:
+
+    with open(input_path, "r", encoding="utf-8") as infile:
         for idx, line in tqdm(enumerate(infile), desc="Preparing batch requests", total=total_lines):
             data = json.loads(line)
-            
+
             # Skip example if 'reasoning' already exists.
-            if not data.get('reasoning'):
+            if not data.get("reasoning"):
                 # Build a custom id. Here we use the row position and the start/end words:
                 metadata = data.get("metadata", {})
                 start = metadata.get("start_word", "unknown")
                 end = metadata.get("end_word", "unknown")
                 custom_id = f"{start}_{end}_{idx}"
                 custom_ids.append(custom_id)
-                
+
                 # Build the prompt text exactly as before.
                 prompt = f"{data['question']}. The correct solution is {data['answer']}. "
-                
+
                 # Build the request payload using Request and MessageCreateParamsNonStreaming.
                 request_payload = Request(
                     custom_id=custom_id,
@@ -95,10 +97,8 @@ def submit_reasoning_batches(
                         max_tokens=8192,
                         temperature=0.5,
                         system=system_message,
-                        messages=[
-                            {"role": "user", "content": prompt}
-                        ]
-                    )
+                        messages=[{"role": "user", "content": prompt}],
+                    ),
                 )
                 # Instead of wrapping in SimpleNamespace, simply ensure custom_id is set.
                 if isinstance(request_payload, dict):
@@ -106,7 +106,7 @@ def submit_reasoning_batches(
                 else:
                     request_payload.custom_id = custom_id
                 batch_requests.append(request_payload)
-                
+
                 # If we have reached our batch size limit, submit the current batch.
                 if len(batch_requests) >= BATCH_SIZE:
                     _submit_single_batch(batch_requests, custom_ids, batch_num, batch_metadata_prefix, input_path)
@@ -114,7 +114,7 @@ def submit_reasoning_batches(
                     # Reset for the next batch
                     batch_requests = []
                     custom_ids = []
-    
+
     # Submit any remaining requests that didn't complete a full batch.
     if batch_requests:
         _submit_single_batch(batch_requests, custom_ids, batch_num, batch_metadata_prefix, input_path)
@@ -141,11 +141,11 @@ def _submit_single_batch(batch_requests, custom_ids, batch_num, batch_metadata_p
         if dt.tzinfo is not None and dt.tzinfo.utcoffset(dt) is not None:
             iso_str = iso_str.replace("+00:00", "Z")
         return iso_str
-    
+
     def extract_custom_id(req):
         # Safely extract the custom_id attribute whether req is an object or a dict.
         return req.custom_id if hasattr(req, "custom_id") else req.get("custom_id")
-        
+
     max_attempts = 2
     attempt = 0
     last_exception = None
@@ -153,9 +153,7 @@ def _submit_single_batch(batch_requests, custom_ids, batch_num, batch_metadata_p
     while attempt < max_attempts:
         try:
             print(f"Submitting batch {batch_num} with {len(batch_requests)} requests... (attempt {attempt+1})")
-            message_batch = client.messages.batches.create(
-                requests=batch_requests
-            )
+            message_batch = client.messages.batches.create(requests=batch_requests)
             time.sleep(1)
             print(f"Batch {batch_num} submitted with ID: {message_batch.id}")
             break  # Success: exit the loop.
@@ -166,17 +164,21 @@ def _submit_single_batch(batch_requests, custom_ids, batch_num, batch_metadata_p
             if attempt < max_attempts:
                 print("Retrying...")
                 time.sleep(1)
-    
+
     if message_batch is None:
         error_filename = output_dir / f"{COMMON_UUID}_failed_batches.jsonl"
-        error_msg = f"{str(last_exception)} after {max_attempts} attempts" if last_exception else f"Failed after {max_attempts} attempts"
+        error_msg = (
+            f"{str(last_exception)} after {max_attempts} attempts"
+            if last_exception
+            else f"Failed after {max_attempts} attempts"
+        )
         failed_info = {
             "batch_number": batch_num,
             "error": error_msg,
             "batch_requests": [extract_custom_id(req) for req in batch_requests],
             "input_file": input_path,
         }
-        with open(error_filename, 'a', encoding='utf-8') as error_file:
+        with open(error_filename, "a", encoding="utf-8") as error_file:
             error_file.write(json.dumps(failed_info) + "\n")
         print(f"Batch {batch_num} permanently failed. Logged to {error_filename}.")
         return
@@ -198,14 +200,15 @@ def _submit_single_batch(batch_requests, custom_ids, batch_num, batch_metadata_p
         "batch_id": message_batch.id,
         "api_response": api_response,
         "custom_ids": custom_ids,
-        "input_file": os.path.basename(input_path),  
+        "input_file": os.path.basename(input_path),
     }
     metadata_filename = output_dir / f"{COMMON_UUID}_{batch_metadata_prefix}.jsonl"
-    with open(metadata_filename, 'a', encoding='utf-8') as meta_file:
+    with open(metadata_filename, "a", encoding="utf-8") as meta_file:
         meta_file.write(json.dumps(batch_metadata) + "\n")
 
     print(f"Batch metadata for batch {batch_num} appended to {metadata_filename}.")
 
+
 if __name__ == "__main__":
     # When running this module directly, submit the reasoning batches.
     submit_reasoning_batches()

examples/word_ladder/utils/usage_stats.py

@@ -9,7 +9,7 @@ It then calculates and prints statistics for each usage token field:
     - cache_creation_input_tokens
     - cache_read_input_tokens
     - output_tokens
-+pricing calculations 
++pricing calculations
 +calculates the savings from caching (vs if we hadn't done any caching)
 +forecasts costs for 10,000, 20,000 and 50,000 jobs based on tokens per query
 
@@ -17,17 +17,14 @@ Usage:
     python usage_stats.py path/to/msgbatch_01X9LgZNVkLFhzrrBd9LNgWb_results.jsonl
 """
 
-import json
 import argparse
+import json
 from statistics import mean
 
+
 def main():
-    parser = argparse.ArgumentParser(
-        description="Compute usage token statistics from a JSONL file."
-    )
-    parser.add_argument(
-        "file", help="Path to the JSONL file containing usage token data."
-    )
+    parser = argparse.ArgumentParser(description="Compute usage token statistics from a JSONL file.")
+    parser.add_argument("file", help="Path to the JSONL file containing usage token data.")
     args = parser.parse_args()
 
     # Usage token fields that we want to track
@@ -43,7 +40,7 @@ def main():
     pricing = {
         "input_tokens": base_input_rate,
         "cache_creation_input_tokens": base_input_rate * 1.25,  # More expensive for initial computation
-        "cache_read_input_tokens": base_input_rate * 0.1,         # Cheaper for cache-read tokens
+        "cache_read_input_tokens": base_input_rate * 0.1,  # Cheaper for cache-read tokens
         "output_tokens": 7.50,
     }
 
@@ -82,7 +79,7 @@ def main():
     print(f"\nProcessed {total_lines} lines with {error_count} error(s).\n")
     print("Usage Tokens Statistics:")
     print("-" * 40)
-    
+
     grand_total_cost = 0.0
     # Calculate and print stats for each token type
     for key in usage_fields:
@@ -115,7 +112,7 @@ def main():
     # Without caching, all tokens would have been charged at the standard input rate.
     #
     # Baseline cost (if no caching were used):
-    #   = (input_tokens + cache_creation_input_tokens + cache_read_input_tokens) 
+    #   = (input_tokens + cache_creation_input_tokens + cache_read_input_tokens)
     #     / 1_000_000 * base_input_rate
     #
     # Actual cost (with caching):
@@ -129,9 +126,11 @@ def main():
     sum_cache_read = sum(usage_data["cache_read_input_tokens"])
 
     baseline_input_cost = (sum_input + sum_cache_creation + sum_cache_read) / 1_000_000 * pricing["input_tokens"]
-    actual_input_cost = (sum_input) / 1_000_000 * pricing["input_tokens"] \
-                        + (sum_cache_creation) / 1_000_000 * pricing["cache_creation_input_tokens"] \
-                        + (sum_cache_read) / 1_000_000 * pricing["cache_read_input_tokens"]
+    actual_input_cost = (
+        (sum_input) / 1_000_000 * pricing["input_tokens"]
+        + (sum_cache_creation) / 1_000_000 * pricing["cache_creation_input_tokens"]
+        + (sum_cache_read) / 1_000_000 * pricing["cache_read_input_tokens"]
+    )
     caching_savings = baseline_input_cost - actual_input_cost
 
     print(f"Caching Savings (input-related tokens): ${caching_savings:.2f}")
@@ -172,12 +171,16 @@ def main():
             forecast_output = avg_output_tokens * jobs
 
             # Forecast actual cost (with caching applied for input tokens):
-            actual_input_cost_forecast = (forecast_input) / 1_000_000 * pricing["input_tokens"] \
-                                         + (forecast_cache_creation) / 1_000_000 * pricing["cache_creation_input_tokens"] \
-                                         + (forecast_cache_read) / 1_000_000 * pricing["cache_read_input_tokens"]
+            actual_input_cost_forecast = (
+                (forecast_input) / 1_000_000 * pricing["input_tokens"]
+                + (forecast_cache_creation) / 1_000_000 * pricing["cache_creation_input_tokens"]
+                + (forecast_cache_read) / 1_000_000 * pricing["cache_read_input_tokens"]
+            )
 
             # Without caching, all input-related tokens would be at base_input_rate:
-            baseline_input_cost_forecast = (forecast_input + forecast_cache_creation + forecast_cache_read) / 1_000_000 * pricing["input_tokens"]
+            baseline_input_cost_forecast = (
+                (forecast_input + forecast_cache_creation + forecast_cache_read) / 1_000_000 * pricing["input_tokens"]
+            )
 
             caching_savings_forecast = baseline_input_cost_forecast - actual_input_cost_forecast
 
@@ -198,5 +201,6 @@ def main():
     else:
         print("No valid jobs to forecast future costs.")
 
+
 if __name__ == "__main__":
-    main()
+    main()