o %ifu3@snddlmZddlmZddlmZddlZddlmZddl m Z ddl m Z ddl m Z Gd d d ZdS) ) annotations)cycle)SequenceN)Producer)report_collection_diff)parse_spec_config)WorkerControllerc@seZdZdZd8d9d d Zed:d d Zed;ddZed;ddZed;ddZ dd?d"d#Z d@d&d'Z dAd*d+Zd>dBd,d-ZdCd/d0ZdDd1d2ZdEd4d5Zd;d6d7ZdS)FLoadSchedulingauImplement load scheduling across nodes. This distributes the tests collected across all nodes so each test is run just once. All nodes collect and submit the test suite and when all collections are received it is verified they are identical collections. Then the collection gets divided up in chunks and chunks get submitted to nodes. Whenever a node finishes an item, it calls ``.mark_test_complete()`` which will trigger the scheduler to assign more tests if the number of pending tests for the node falls below a low-watermark. When created, ``numnodes`` defines how many nodes are expected to submit a collection. This is used to know when all nodes have finished collection or how large the chunks need to be created. Attributes:: :numnodes: The expected number of nodes taking part. The actual number of nodes will vary during the scheduler's lifetime as nodes are added by the DSession as they are brought up and removed either because of a dead node or normal shutdown. This number is primarily used to know when the initial collection is completed. :node2collection: Map of nodes and their test collection. All collections should always be identical. :node2pending: Map of nodes and the indices of their pending tests. The indices are an index into ``.pending`` (which is identical to their own collection stored in ``.node2collection``). :collection: The one collection once it is validated to be identical between all the nodes. It is initialised to None until ``.schedule()`` is called. :pending: List of indices of globally pending tests. These are tests which have not yet been allocated to a chunk for a node to process. :log: A py.log.Producer instance. :config: Config object, used for handling hooks. Nconfig pytest.ConfiglogProducer | NonereturnNonecCsZtt||_i|_i|_g|_d|_|durtd|_n|j |_||_ |j d|_ dS)N loadsched maxschedchunk) lenrnumnodesnode2collection node2pendingpending collectionrr rr getoptionr)selfr r rM/var/www/html/corbot_env/lib/python3.10/site-packages/xdist/scheduler/load.py__init__<s zLoadScheduling.__init__list[WorkerController]cCst|jS)z%A list of all nodes in the scheduler.)listrkeysrrrrnodesIszLoadScheduling.nodesboolcCst|j|jkS)zBoolean indication initial test collection is complete. This is a boolean indicating all initial participating nodes have finished collection. The required number of initial nodes is defined by ``.numnodes``. )rrrr rrrcollection_is_completedNsz&LoadScheduling.collection_is_completedcCs:|jsdS|jr dS|jD] }t|dkrdSqdS)z9Return True if all tests have been executed by the nodes.FT)r#rrvaluesrrrrrrtests_finishedXs zLoadScheduling.tests_finishedcCs(|jrdS|jD]}|rdSq dS)zReturn True if there are pending test items. This indicates that collection has finished and nodes are still processing test items, so this can be thought of as "the scheduler is active". TF)rrr%r&rrr has_pendingdszLoadScheduling.has_pendingnodercCs||jvsJg|j|<dS)zAdd a new node to the scheduler. From now on the node will be allocated chunks of tests to execute. Called by the ``DSession.worker_workerready`` hook when it successfully bootstraps a new node. N)r)rr)rrradd_nodess zLoadScheduling.add_noder Sequence[str]cCsr||jvsJ|jr0|jsJ||jkr0tt|j}t|j||jj |jj }| |dSt ||j|<dS)zAdd the collected test items from a node. The collection is stored in the ``.node2collection`` map. Called by the ``DSession.worker_collectionfinish`` hook. N) rr#rnextiterrrrgatewayidr r)rr)r other_nodemsgrrradd_node_collections   z"LoadScheduling.add_node_collectionr item_indexintdurationfloatcCs"|j|||j||ddS)zMark test item as completed by node. The duration it took to execute the item is used as a hint to the scheduler. This is called by the ``DSession.worker_testreport`` hook. )r5N)rremovecheck_schedule)rr)r3r5rrrmark_test_completes z!LoadScheduling.mark_test_completeitemstrcCs>|jdusJ|jd|j||jD]}||qdS)Nr)rrinsertindexrr8)rr:r)rrrmark_test_pendings   z LoadScheduling.mark_test_pendingindices Sequence[int]cCstN)NotImplementedError)rr)r?rrrremove_pending_tests_from_nodesz-LoadScheduling.remove_pending_tests_from_nodec Cs|jrdS|jrVt|j}tdt|j|d}tdt|j|d}|j|}t||krU|dkrrCr8rSrZrGrUrrrrr s, -        "  @r ) __future__r itertoolsrtypingrre xdist.remoter xdist.reportrxdist.workermanagerrr rrrrs