Here’s a highly optimized version of your code for both **runtime** and **memory**, based on the profile hot spots.
- **Avoid repeated summing** for checking lengths in a growing list — we keep a running sum.
- **Avoid repeatedly copying lists/dicts** by using lists of indices and marking to remove in one pass, and using set operations for fast membership checks.
- **Avoid creating lots of small dicts** and list extensions inside loops.
- **Combine related generator expressions** so costly operations are only done once.
- **Group similar linear scans** into one to minimize number of loops over `queue`.
- Use **pre-allocated lists and sets** where it saves time.
Here's the rewritten function (all comments preserved except where the code logic was changed).
**Key optimizations:**
- Only a *single pass* over queue for setup.
- No repeated `.append(dict)`; pass only indices around until the end.
- Use `.clear()` for lists inside dict to avoid reallocations.
- Use lists of lengths for O(1) access everywhere.
- Maintain a running sum for batch size check, not repeated `sum`.
This should **dramatically cut runtime**, especially at the hot spots from your line profiler output. If you need even more speed and the queue is huge/long-lived, consider reworking the data structure for the queue itself (`deque`, heap, etc.), but for code-level optimization this is near optimal for this algorithm!
