mirror of
https://github.com/thinking-machines-lab/tinker.git
synced 2026-04-19 12:58:01 +00:00
Sync contents
This commit is contained in:
parent
35a77e79fe
commit
f3c0b1f179
17 changed files with 2069 additions and 632 deletions
|
|
@ -30,6 +30,20 @@ future = sampling_client.sample(prompt=prompt, sampling_params=params, num_sampl
|
|||
result = future.result()
|
||||
```
|
||||
|
||||
Multi-processing support:
|
||||
This class is picklable, so it can be passed to a separate process/worker to sample. It is also
|
||||
safe to pass the same instance of SamplingClient to multiple processes/workers.
|
||||
|
||||
If you are using Tinker SDK with more than one process you should always create SamplingClient from
|
||||
the main process and then pass it to the other processes/workers.
|
||||
ServiceClient and TrainingClient should always be managed from the main process.
|
||||
|
||||
Subprocess isolation:
|
||||
Set ``TINKER_SUBPROCESS_SAMPLING=1`` to run sample() and compute_logprobs() in a dedicated
|
||||
subprocess, preventing GIL contention from CPU-heavy user code (grading, environment
|
||||
interactions) from stalling networking IO and heartbeats. This is transparent — the same
|
||||
API works with or without it.
|
||||
|
||||
#### `sample`
|
||||
|
||||
```python
|
||||
|
|
@ -121,3 +135,33 @@ Get the tokenizer for the current model.
|
|||
|
||||
Returns:
|
||||
- `PreTrainedTokenizer` compatible with the model
|
||||
|
||||
#### `get_base_model`
|
||||
|
||||
```python
|
||||
def get_base_model() -> str
|
||||
```
|
||||
|
||||
Get the base model name for the current sampling session.
|
||||
|
||||
#### `get_base_model_async`
|
||||
|
||||
```python
|
||||
async def get_base_model_async() -> str
|
||||
```
|
||||
|
||||
Async version of get_base_model.
|
||||
|
||||
#### `__reduce__`
|
||||
|
||||
```python
|
||||
def __reduce__() -> tuple[Any, tuple[_SamplingClientPickleState]]
|
||||
```
|
||||
|
||||
Enable pickling of SamplingClient for subprocess use.
|
||||
|
||||
Serializes into a ``_SamplingClientPickleState`` dataclass. The
|
||||
``_sampling_client_sidecar_handle`` handle is deliberately omitted — only a
|
||||
bool flag is stored. The unpickled copy creates its own handle via
|
||||
the per-process sidecar singleton. Do not add ``__getstate__``
|
||||
without preserving this behavior.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue