o
Zh @ s d Z ddlZddlmZ ddlmZmZmZ ddlm Z m
Z
m Z m Z er(ddl
Z
e e
ddeed f d
ed ed e fd
dZdd d edeef fddZdd d efddZe d ddZdd d efddZdd d efddZe d d d efd d ZdS )!z"Contains pytorch-specific helpers. N) lru_cache)
TYPE_CHECKINGDictTuple )FILENAME_PATTERNMAX_SHARD_SIZEStateDictSplit$split_state_dict_into_shards_factory)filename_patternmax_shard_size
state_dictz torch.Tensorr r returnc C s t | ||ttdS )a
Split a model state dictionary in shards so that each shard is smaller than a given size.
The shards are determined by iterating through the `state_dict` in the order of its keys. There is no optimization
made to make each shard as close as possible to the maximum size passed. For example, if the limit is 10GB and we
have tensors of sizes [6GB, 6GB, 2GB, 6GB, 2GB, 2GB] they will get sharded as [6GB], [6+2GB], [6+2+2GB] and not
[6+2+2GB], [6+2GB], [6GB].
If one of the model's tensor is bigger than `max_shard_size`, it will end up in its own shard which will have a
size greater than `max_shard_size`.
Args:
state_dict (`Dict[str, torch.Tensor]`):
The state dictionary to save.
filename_pattern (`str`, *optional*):
The pattern to generate the files names in which the model will be saved. Pattern must be a string that
can be formatted with `filename_pattern.format(suffix=...)` and must contain the keyword `suffix`
Defaults to `"model{suffix}.safetensors"`.
max_shard_size (`int` or `str`, *optional*):
The maximum size of each shard, in bytes. Defaults to 5GB.
Returns:
[`StateDictSplit`]: A `StateDictSplit` object containing the shards and the index to retrieve them.
Example:
```py
>>> import json
>>> import os
>>> from safetensors.torch import save_file as safe_save_file
>>> from huggingface_hub import split_torch_state_dict_into_shards
>>> def save_state_dict(state_dict: Dict[str, torch.Tensor], save_directory: str):
... state_dict_split = split_torch_state_dict_into_shards(state_dict)
... for filename, tensors in state_dict_split.filename_to_tensors.values():
... shard = {tensor: state_dict[tensor] for tensor in tensors}
... safe_save_file(
... shard,
... os.path.join(save_directory, filename),
... metadata={"format": "pt"},
... )
... if state_dict_split.is_sharded:
... index = {
... "metadata": state_dict_split.metadata,
... "weight_map": state_dict_split.tensor_to_filename,
... }
... with open(os.path.join(save_directory, "model.safetensors.index.json"), "w") as f:
... f.write(json.dumps(index, indent=2))
```
)r r get_tensor_sizeget_storage_id)r
r r )r
r r r [/var/www/html/lang_env/lib/python3.10/site-packages/huggingface_hub/serialization/_torch.py"split_torch_state_dict_into_shards s ;r tensorz torch.devicec C s@ | j jdkrt rddl}|j| }nt| }| j |t| fS )a1
Return unique identifier to a tensor storage.
Multiple different tensors can share the same underlying storage. For
example, "meta" tensors all share the same storage, and thus their identifier will all be equal. This identifier is
guaranteed to be unique and constant for this tensor's storage during its lifetime. Two tensor storages with
non-overlapping lifetimes may have the same id.
Taken from https://github.com/huggingface/transformers/blob/1ecf5f7c982d761b4daaa96719d162c324187c64/src/transformers/pytorch_utils.py#L278.
Zxlar N)Zdevicetypeis_torch_tpu_available torch_xlaZ_XLACZ_xla_get_tensor_id storage_ptrget_storage_size)r r unique_idr r r r _ s
r c C s | | S )N)ZnumelZ element_sizer r r r r x s r Tc C sR t jddur'| r%zddlm m} | }W dS ty$ Y dS w dS dS )z
Checks if `torch_xla` is installed and potentially if a TPU is in the environment
Taken from https://github.com/huggingface/transformers/blob/1ecf5f7c982d761b4daaa96719d162c324187c64/src/transformers/utils/import_utils.py#L463.
r Nr TF) importlibutil find_specZtorch_xla.core.xla_modelcoreZ xla_modelZ
xla_device RuntimeError)Z check_deviceZxm_r r r r | s r c C sH z| W S ty# z | W Y S ty" Y Y dS w w )z
Taken from https://github.com/huggingface/safetensors/blob/08db34094e9e59e2f9218f2df133b7b4aaff5a99/bindings/python/py_src/safetensors/torch.py#L11C1-L20C21.
r )untyped_storageZdata_ptr ExceptionstorageNotImplementedErrorr r r r r s r c C sd z| W S ty1 z| t| j W Y S ty0 | t| j Y Y S w w )z
Taken from https://github.com/huggingface/safetensors/blob/08db34094e9e59e2f9218f2df133b7b4aaff5a99/bindings/python/py_src/safetensors/torch.py#L31C1-L41C59
) r" nbytesAttributeErrorr$ size_get_dtype_sizedtyper% Znelementr r r r r s r r* z torch.dtypec C sp ddl }t|dd}t|dd}|jd|jd|jd|jd|jd|jd|jd|j d|j
d|j d|d|di }|| S ) z
Taken from https://github.com/huggingface/safetensors/blob/08db34094e9e59e2f9218f2df133b7b4aaff5a99/bindings/python/py_src/safetensors/torch.py#L344
r NZ
float8_e4m3fnZ float8_e5m2 r ) torchgetattrZint64Zfloat32Zint32Zbfloat16Zfloat16Zint16Zuint8Zint8boolZfloat64)r* r. Z_float8_e4m3fnZ _float8_e5m2Z_SIZEr r r r) s" r) )T)__doc__r functoolsr typingr r r _baser r r r
r. strintr r r r r r r) r r r r s4
D