o TZhU!@s@dZddlZddlZddlZddZddZdZdd e d DZd d eDZd d e DZ ddZ ddZ ddZd:ddZd;ddZddZd:ddZdd8d9Z#dS)?a Binary tensor encodings for PyTorch and NumPy. This defines efficient binary encodings for tensors. The format is 8 byte aligned and can be used directly for computations when transmitted, say, via RDMA. The format is supported by WebDataset with the `.ten` filename extension. It is also used by Tensorcom, Tensorcom RDMA, and can be used for fast tensor storage with LMDB and in disk files (which can be memory mapped) Data is encoded as a series of chunks: - magic number (int64) - length in bytes (int64) - bytes (multiple of 64 bytes long) Arrays are a header chunk followed by a data chunk. Header chunks have the following structure: - dtype (int64) - 8 byte array name - ndim (int64) - dim[0] - dim[1] - ... NcCs0t|dr|jSt|ttfrt|St|d)z#Determine the length of a in bytes.nbyteszcannot determine nbytes)hasattrr isinstance bytearraybyteslen ValueErrorar c/var/www/html/lang_env/lib/python3.10/site-packages/datasets/packaged_modules/webdataset/_tenbin.pybytelen(s  r cCs.t|tttfr |St|dr|jSt|d)z)Return a the raw data corresponding to a.datazcannot return bytedata)rrr memoryviewrrrr r r r bytedata2s   rzl float16 f2 float32 f4 float64 f8 int8 i1 int16 i2 int32 i4 int64 i8 uint8 u1 uint16 u2 unit32 u4 uint64 u8 cCsg|]}|qSr )split.0xr r r Ksr cCsi|] }|d|dqS)rr rr r r LsrcCsi|]\}}||qSr r )rkvr r r rMscCs:|D]}|jjtvrtd|s|jjdvrtdqdS)zCheck that the data has an acceptable type for tensor encoding. :param data: array :param allow64: allow 64 bit types zunsupported dataypte)Zfloat64Zint64Zuint64z664 bit datatypes not allowed unless explicitly enabledN)dtypename long_to_shortr)rZallow64r r r r check_acceptable_input_typePs rcCs.|ddt|}|d}td|dS)zConvert a string to an int64.ascii@qr)rencodestructunpack)sr r r str64]s r'cCstd|}|ddS)zConvert an int64 to a string.r"r!r)r$packdecodestrip)ibr r r unstr64ds r-cCsd|dus|dur |S|dur||fSt|ttfstdt||D] \}}td|d||S)zVerify the info strings.FNTz$required_infos must be tuple or listz actual info z doesn't match required info )rtuplelistrzip)rinfosZrequired_infosrequiredactualr r r check_infosjsr4cCs|jdkr td|jt|j|jkrtd|jjt vr#tdt t |jjt |t |jgt |j}t tj|ddS)z'Encode an array header as a byte array. ztoo many dimensionszmismatch between size and shapeunsupported array typei8r)ndimrrnpprodshapeitemsizerrrr'rr/rarray)r infoheaderr r r encode_headerws  *rBcCsrtj|dd}t|dtvrtdttt|d}t|d}t|d}t|dd|}|||fS)z)Decode a byte array into an array header.r8r9rr7r)r; frombufferr- short_to_longrrintr.)hrr@Zrankr=r r r decode_headers   rIcCst|durdg}nt|t|krtd|d|g}t|D]\}}t|||t|}||t|g7}q |S)z?Given a list of arrays, encode them into a list of byte arrays.Nr5zlength of list z must muatch length of infos )rr enumeraterBr)lr1resultr+r rAr r r encode_listsrMFc Cszg}g}t|ddd|dddD]#\}}t|\}}}tj||t|dj|} || g7}||g7}qt|||S)z5Given a list of byte arrays, decode them into arrays.NrCr)rcount)r0rIr;rEr<Zreshaper4) rKr1rLZinfos0rArr=rr@r r r r decode_lists&   rOz~TenBin~r!@cCs|||d|S)z$Round up to the next multiple of 64.rr )nrr r r roundupsrRcCstdd|D}t|}d}|D]1}t|||d<|d7}td|j|||d<|d7}||||t|<|tt|7}q|S)zKEncode a list of chunks into a single byte array, with lengths and magics..css|] }dt|jVqdS)N)rRr)rr,r r r sz encode_chunks..rr r")sumr magic_bytesr$r(rr rR)rKsizerLoffsetr,r r r encode_chunkssrYcCsg}d}t|}||krGt|||dkrtd|d7}td|||dd}|d7}||||}|t|7}||||ks |S)z*Decode a byte array into a list of chunks.rr zmagic bytes mismatchr")r rVrr$r%rRappend)bufrLrXtotalrr,r r r decode_chunkss   r]cCs"t|ts tdtt||dS)z1Encode a list of arrays into a single byte array.z requires listr1)rr/rrYrM)rKr1r r r encode_buffers r_cCstt||dS)z*Decode a byte array into a list of arrays.r^)rOr])r[r1r r r decode_buffersr`cCs\t|}|t|td||t|t||}|dkr,|d|dSdS)zBWrite a byte chunk to the stream with magics, length, and padding.r"rN)r writerVr$r(rrR)streamr[rpaddingr r r write_chunks  recCs||d}|dkr dS|tkrtd|d}td|d}|dkr(td||}t||}|dkr<|||S)zARead a byte chunk from a stream with magics, length, and padding.r Nzmagic number does not matchr"rznegative nbytes)readrVrr$r%rR)rcmagicrrrdr r r read_chunks     ricCs t||dD]}t||qdS)zEWrite a list of arrays to a stream, with magics, length, and padding.r^N)rMre)rcrKr1chunkr r r rbs rbcCsVg}t|D]}t|}|durnt|}|durtd|||g7}qt||dS)zFRead a list of arrays from a stream, with magics, length, and padding.Nz premature EOFr^)rangerirrO)rcrQr1chunks_rArr r r rgs  rg)r1nocheckcGsT|s |ds tdt|d}t|||dWddS1s#wYdS)zBSave a list of arrays to a file, with magics, length, and padding..tenfile name should end in .tenwbr^N)endswithropenrb)fnamer1rnargsrcr r r saves  "rvcCsP|s |ds tdt|d}t||dWdS1s!wYdS)zDRead a list of arrays from a file, with magics, length, and padding.rorprbr^N)rrrrsrg)rtr1rnrcr r r loads   $rx)N)r5)F)rP)FF)$__doc__r$sysnumpyr;r rr*rritemsrFrr'r-r4rBrIrMrOZ magic_strrhr#rVrRrYr]r_r`rerirbmaxsizergrvrxr r r r sD