o
ZhZ2 @ s U d Z ddlZddlZddlZddlZddlZddlmZ ddlm Z ddl
m Z m Z m
Z
mZ ddlZddlmZ ddlmZ ddlmZ d d
lmZ d d lmZ d d
lmZ eeZ dZ dZ G dd de Z G dd deZ!G dd deZ"dej#fddZ$e g ej#f Z%e$a&e%e'd< e$fde%ddfdd Z(dej#fd d Z)d2d d Z*ed!e+d"e+dej#fd#d$Z,d%d d&ej-ej fe j.d'd(ed)e/d*e+d+e0d,e0d-ee
e1 e e
e1 d.f f d/ee+e e+d.f f defd0d1Z2dS )3z>Contains utilities to handle HTTP requests in Huggingface Hub. N) lru_cache)
HTTPStatus)CallableTupleTypeUnion)Response) HTTPAdapter)PreparedRequest ) constants )logging)
HTTP_METHOD_TX-Amzn-Trace-Idz x-request-idc @ s e Zd ZdZdS )OfflineModeIsEnabledzTRaised when a request is made but `HF_HUB_OFFLINE=1` is set as environment variable.N)__name__
__module__ __qualname____doc__ r r R/var/www/html/lang_env/lib/python3.10/site-packages/huggingface_hub/utils/_http.pyr - s r c s6 e Zd ZdZ fddZdedef fdd Z ZS )UniqueRequestIdAdapterr c s t j|fi | t|jvr |jtptt |jt< t|jdd d}t
d|jt d|j d|j
d| d d S ) N
authorization z
Bearer hf_zRequest z: z (authenticated: ))super add_headersX_AMZN_TRACE_IDheadersget X_REQUEST_IDstruuiduuid4
startswithloggerdebugmethodurl)selfrequestkwargsZ has_token __class__r r r 4 s
&z"UniqueRequestIdAdapter.add_headersr, returnc
sh zt j|g|R i |W S tjy3 } z|jt}|dur.g |jd| dR |_ d}~ww )zSCatch any RequestException to append request id to the error message for debugging.Nz
(Request ID: r )r sendrequestsRequestExceptionr r! r args)r+ r, r4 r- eZ
request_idr. r r r1 A s zUniqueRequestIdAdapter.send) r r r r r r
r r1
__classcell__r r r. r r 1 s
r c @ s e Zd ZdedefddZdS )OfflineAdapterr, r0 c O s t d|j d)Nz
Cannot reach za: offline mode is enabled. To disable it, please unset the `HF_HUB_OFFLINE` environment variable.)r r* )r+ r, r4 r- r r r r1 N s zOfflineAdapter.sendN)r r r r
r r1 r r r r r7 M s r7 r0 c C sN t } tjr| dt | dt | S | dt | dt | S )Nzhttp://zhttps://)r2 Sessionr ZHF_HUB_OFFLINEmountr7 r )sessionr r r _default_backend_factoryT s r; _GLOBAL_BACKEND_FACTORYbackend_factoryc C s | a t dS )a
Configure the HTTP backend by providing a `backend_factory`. Any HTTP calls made by `huggingface_hub` will use a
Session object instantiated by this factory. This can be useful if you are running your scripts in a specific
environment requiring custom configuration (e.g. custom proxy or certifications).
Use [`get_session`] to get a configured Session. Since `requests.Session` is not guaranteed to be thread-safe,
`huggingface_hub` creates 1 Session instance per thread. They are all instantiated using the same `backend_factory`
set in [`configure_http_backend`]. A LRU cache is used to cache the created sessions (and connections) between
calls. Max size is 128 to avoid memory leaks if thousands of threads are spawned.
See [this issue](https://github.com/psf/requests/issues/2766) to know more about thread-safety in `requests`.
Example:
```py
import requests
from huggingface_hub import configure_http_backend, get_session
# Create a factory function that returns a Session with configured proxies
def backend_factory() -> requests.Session:
session = requests.Session()
session.proxies = {"http": "http://10.10.1.10:3128", "https": "https://10.10.1.11:1080"}
return session
# Set it as the default session factory
configure_http_backend(backend_factory=backend_factory)
# In practice, this is mostly done internally in `huggingface_hub`
session = get_session()
```
N)r< reset_sessions)r= r r r configure_http_backendc s
r? c C s t t t dS )a
Get a `requests.Session` object, using the session factory from the user.
Use [`get_session`] to get a configured Session. Since `requests.Session` is not guaranteed to be thread-safe,
`huggingface_hub` creates 1 Session instance per thread. They are all instantiated using the same `backend_factory`
set in [`configure_http_backend`]. A LRU cache is used to cache the created sessions (and connections) between
calls. Max size is 128 to avoid memory leaks if thousands of threads are spawned.
See [this issue](https://github.com/psf/requests/issues/2766) to know more about thread-safety in `requests`.
Example:
```py
import requests
from huggingface_hub import configure_http_backend, get_session
# Create a factory function that returns a Session with configured proxies
def backend_factory() -> requests.Session:
session = requests.Session()
session.proxies = {"http": "http://10.10.1.10:3128", "https": "https://10.10.1.11:1080"}
return session
# Set it as the default session factory
configure_http_backend(backend_factory=backend_factory)
# In practice, this is mostly done internally in `huggingface_hub`
session = get_session()
```
process_id thread_id)_get_session_from_cacheosgetpid threading get_identr r r r get_session s rH c C s t dS )zReset the cache of sessions.
Mostly used internally when sessions are reconfigured or an SSLError is raised.
See [`configure_http_backend`] for more details.
N)rC cache_clearr r r r r> s r> rA rB c C s t S )z
Create a new session per thread using global factory. Using LRU cache (maxsize 128) to avoid memory leaks when
using thousands of threads. Cache is cleared when `configure_http_backend` is called.
)r< r@ r r r rC s rC ) max_retriesbase_wait_time
max_wait_timeretry_on_exceptionsretry_on_status_codesr) r* rL rM rN rO .rP c K sl t |tr|f}t |tr|f}d}|} d}
d|v r(t |d tjr(|d }
t } |d7 }z:|
dur<|d |
| jd| |d|} | j |vrO| W S t
d| j d| d | ||kri| | W S W n/ |y }
z#t
d
|
d | d | t |
t
jrt ||kr|
W Y d}
~
nd}
~
ww t
d | d
| d| d t| t|| d } q,)a# Wrapper around requests to retry calls on an endpoint, with exponential backoff.
Endpoint call is retried on exceptions (ex: connection timeout, proxy error,...)
and/or on specific status codes (ex: service unavailable). If the call failed more
than `max_retries`, the exception is thrown or `raise_for_status` is called on the
response object.
Re-implement mechanisms from the `backoff` library to avoid adding an external
dependencies to `hugging_face_hub`. See https://github.com/litl/backoff.
Args:
method (`Literal["GET", "OPTIONS", "HEAD", "POST", "PUT", "PATCH", "DELETE"]`):
HTTP method to perform.
url (`str`):
The URL of the resource to fetch.
max_retries (`int`, *optional*, defaults to `5`):
Maximum number of retries, defaults to 5 (no retries).
base_wait_time (`float`, *optional*, defaults to `1`):
Duration (in seconds) to wait before retrying the first time.
Wait time between retries then grows exponentially, capped by
`max_wait_time`.
max_wait_time (`float`, *optional*, defaults to `8`):
Maximum duration (in seconds) to wait before retrying.
retry_on_exceptions (`Type[Exception]` or `Tuple[Type[Exception]]`, *optional*):
Define which exceptions must be caught to retry the request. Can be a single type or a tuple of types.
By default, retry on `requests.Timeout` and `requests.ConnectionError`.
retry_on_status_codes (`int` or `Tuple[int]`, *optional*, defaults to `503`):
Define on which status codes the request must be retried. By default, only
HTTP 503 Service Unavailable is retried.
**kwargs (`dict`, *optional*):
kwargs to pass to `requests.request`.
Example:
```
>>> from huggingface_hub.utils import http_backoff
# Same usage as "requests.request".
>>> response = http_backoff("GET", "https://www.google.com")
>>> response.raise_for_status()
# If you expect a Gateway Timeout from time to time
>>> http_backoff("PUT", upload_url, data=data, retry_on_status_codes=504)
>>> response.raise_for_status()
```
When using `requests` it is possible to stream data by passing an iterator to the
`data` argument. On http backoff this is a problem as the iterator is not reset
after a failed call. This issue is mitigated for file objects or any IO streams
by saving the initial position of the cursor (with `data.tell()`) and resetting the
cursor between each call (with `data.seek()`). For arbitrary iterators, http backoff
will fail. If this is a hard constraint for you, please let us know by opening an
issue on [Github](https://github.com/huggingface/huggingface_hub).
r NdataTr
)r) r* z HTTP Error z thrown while requesting r 'z' thrown while requesting z Retrying in z s [Retry /z].r r )
isinstancetypeintioIOBasetellrH seekr, status_coder' warningraise_for_statusr2 ConnectionErrorr> timesleepmin)r) r* rL rM rN rO rP r- Znb_triesZ
sleep_timeZio_obj_initial_posr: responseerrr r r http_backoff sH
G
rd )r0 N)3r rW rD rF r_ r$ functoolsr httpr typingr r r r r2 r Zrequests.adaptersr Zrequests.modelsr
r r r Z_typingr Z
get_loggerr r' r r" r^ r r r7 r8 r; ZBACKEND_FACTORY_Tr< __annotations__r? rH r> rV rC TimeoutSERVICE_UNAVAILABLEr# float Exceptionrd r r r r sl
$