o
ZhR @ sJ d Z ddlZddlmZmZ edZdee dedeee fdd ZdS )
z9Contains a utility to iterate by chunks over an iterator. N)IterableTypeVarTiterable
chunk_sizereturnc c sf t |tr
|dkrtdt| } zt|}W n
ty# Y dS w t|ft||d V q)a Iterates over an iterator chunk by chunk.
Taken from https://stackoverflow.com/a/8998040.
See also https://github.com/huggingface/huggingface_hub/pull/920#discussion_r938793088.
Args:
iterable (`Iterable`):
The iterable on which we want to iterate.
chunk_size (`int`):
Size of the chunks. Must be a strictly positive integer (e.g. >0).
Example:
```python
>>> from huggingface_hub.utils import chunk_iterable
>>> for items in chunk_iterable(range(17), chunk_size=8):
... print(items)
# [0, 1, 2, 3, 4, 5, 6, 7]
# [8, 9, 10, 11, 12, 13, 14, 15]
# [16] # smaller last chunk
```
Raises:
[`ValueError`](https://docs.python.org/3/library/exceptions.html#ValueError)
If `chunk_size` <= 0.
The last chunk can be smaller than `chunk_size`.
r z6`chunk_size` must be a strictly positive integer (>0).TN )
isinstanceint
ValueErroriternext
StopIteration itertoolschainislice)r r iteratorZ next_item r Y/var/www/html/lang_env/lib/python3.10/site-packages/huggingface_hub/utils/_chunk_utils.pychunk_iterable s r )__doc__r typingr r r r
r r r r r s
&