o TZhS1@s&ddlmZddlZddlmZddlmZmZm Z ddl Z ddl mZddl m Z dZedeZeeZd ed e eeffd d Zed \ZZgdZe eeeeefZGdddZe dGdddeZedkrddlmZeddZe de!Z"ee"j#Z#e$e#Z%dSdS)N)Path)AnyListTuple) resources) deprecatedzDhttps://github.com/huggingface/datasets/tree/main/src/datasets/utils/resourcereturncCs$tt|}t|td|fS)Nz /resources/) pkg_resources read_textryamlZ safe_load BASE_REF_URL)r contentrL/var/www/html/lang_env/lib/python3.10/site-packages/datasets/utils/readme.pyload_yaml_resources rzreadme_structure.yaml)z[Needs More Information]z[More Information Needed]zj(https://github.com/huggingface/datasets/blob/main/CONTRIBUTING.md#how-to-contribute-to-the-dataset-cards)c @s\eZdZddededeedefddZddefd d Zd ed e fd dZ d efddZ dS)SectionNFnamelevellinessuppress_parsing_errorscCsN||_||_||_d|_d|_i|_g|_g|_|jdur%|j|ddSdS)NTr) rrrtext is_empty_textrparsing_error_listparsing_warning_listparse)selfrrrrrrr__init__$s zSection.__init__cCsd}g}d}|jD]l}|ddkrq |ddddkr"| }q |d|jdkrp|sp|dkrBt||jd||j|<g}n|gkra|jd|7_|jdkr_|jtvr_d|_ g}d|d dd}q | |q |dkr||jvr|j d |d t||jd||j|<n|gkr|jd|7_|jdkr|jtvrd|_ |jdkr|s|j gks|j gkrd d d|j |j D}}d|j d|}t|dSdSdS)NrF z```r# rz)Multiple sections with the same heading `z:` have been found. Please keep only one of these sections. css|]}d|VqdS)- Nr.0xrrr Vsz Section.parse..z=The following issues were found while parsing the README at ``: )rstripsplitrrrrjoin FILLER_TEXTrappendrrr ValueError)r rZcurrent_sub_levelZ current_linesZ code_startlineerrors error_stringrrrr0sH     z Section.parse structurer c Csg}g}|ddur|jr|jikr|d|jd|ddur/|jr/|d|jd|dd ur|jikrYd d |dD}|d |jd ddd |Ddn_dd |dD}d}t|D]6\}}||jvr|d |jd|dd}qh|jdkrqh|j||d|\} } } || 7}|| 7}qh|r|jD]}||vr|d|jd|dq|ri||fS|||fS)aCValidates a Section class object recursively using the structure provided as a dictionary. Args: structute (:obj: `dict`): The dictionary representing expected structure. Returns: :obj: `ReadmeValidatorOutput`: The dictionary representation of the section, and the errors. Z allow_emptyFz"Expected some content in section `z` but it is empty.Zallow_empty_textzExpected some text in section `z4` but it is empty (text in subsections are ignored). subsectionsNcSg|]}|dqSrrr)Z subsectionrrr xz$Section.validate..z Section `z&` expected the following subsections: , cSg|]}d|dqS`rr(rrrr;{z. Found 'None'.cSr8r9rr:rrrr;r<z` is missing subsection: `z`.Tz###r@z` has an extra subsection: `z[`. Skipping further validation checks for this subsection as expected structure is unknown.) rrr1rr/ enumeratervalidateto_dict) r r6 error_list warning_listvaluesZstructure_namesZhas_missing_subsectionsidxr_Zsubsec_error_listZsubsec_warning_listrrrrCZsN      "        zSection.validatecCs$|j|j|jdd|jDdS)z3Returns the dictionary representation of a section.cSsg|]}|qSr)rD)r)valuerrrr;r<z#Section.to_dict..)rrrr7)rrrrrGr rrrrDs zSection.to_dictNFF) __name__ __module__ __qualname__strrboolr!rdictReadmeValidatorOutputrCrDrrrrr#s   *Grz*Use `huggingface_hub.DatasetCard` instead.c seZdZddedeededeffdd Zd d Ze dd e dedefd d Z e ddedededefddZ ddeffdd Z ddZddZZS)ReadMeNFrrr6rcsFtj|dd||_d|_d|_||_|jdur!|j|ddSdS)Nr)rrrr)superr!r6yaml_tags_line_count tag_countrr)r rrr6r __class__rrr!s zReadMe.__init__cCst|jdur|t\}}}n ||j\}}}|gks|gkr8ddd||D}d|jd|}t|dS)Nr&cSsg|]}d|qS)r'rr(rrrr;r<z#ReadMe.validate..z3The following issues were found for the README at `r,)r6 _validatereadme_structurer/rr2)r rrErFr4r5rrrrCs zReadMe.validatepathcCsDt|dd }|}Wdn1swY|||||dS)Nzutf-8)encodingr)open readlines)clsr^r6rfrrrr from_readmes zReadMe.from_readmerootstring root_namecCs|d}|||||dS)Nr&r)r.)rbrfr6rgrrrrr from_strings zReadMe.from_stringcsd}|jD]"}|jd7_|ddkr#|jd7_|jdkr#n|d7}q|jdkr8|j|dd|_n |j|jd|_tj|ddS)Nrrr"z---r)rrXr-rYrWr)r r line_countr3rZrrrs    z ReadMe.parsecCs t|S)z>Returns the string of dictionary representation of the ReadMe.)rQrDrKrrr__str__s zReadMe.__str__c Csg}g}|jdkr|dn|jdkr|dn |jdkr$|dt|j}|dkrF|dddd t|jDd n5|dkrP|d n+t|jd}|d rv|j| |d d\}}}||7}||7}n|d|ri||fS| ||fS)Nrz-Empty YAML markers are present in the README.z*No YAML markers are present in the README.rz2Only the start of YAML tags present in the README.z-The README has several first-level headings: r=cSr>r?rr(rrrr;rAz$ReadMe._validate..zL. Only one heading is expected. Skipping further validation for this README.zmThe README has no first-level headings. One heading is expected. Skipping further validation for this README.zDataset Card forr7zuNo first-level heading starting with `Dataset Card for` found in README. Skipping further validation for this README.) rXr1rYlenrkeysr/list startswithrCrD) r r]rErFZnum_first_level_keysZ start_keyrIZsec_error_listZsec_warning_listrrrr\s<      $      zReadMe._validaterL)NreFrM)rNrOrPrQrrSrRr!rC classmethodrrdrhrrkr\ __classcell__rrrZrrUs&$   rU__main__)ArgumentParserz?Validate the content (excluding YAML tags) of a README.md file.)usagereadme_filepath)&importlib.resourcesrr loggingpathlibrtypingrrrrrZdeprecation_utilsrr__file__Zthis_url getLoggerrNloggerrQrr]Zknown_readme_structure_urlr0rSrTrrUargparsersZap add_argument parse_argsargsrurdZreadmerrrrs4       b