Z-&"dZdZddlZddlmZddlZddlZddlZddl m Z ddl Z ddl Z ddl mZddlZddlZddlmZe j&ZdaGd d Zd Ze j0ed Zd ZGddeZGddZdZGddeZ GddeZ!GddeZ"Gdde Z#dZ$dZ% d'dZ&d(dZ'Gdd e jPZ)da*da+d!Z,d"Z-Gd#d$ej\Z/Gd%d&ej`Z1y))a- Implements ProcessPoolExecutor. The following diagram and text describe the data-flow through the system: |======================= In-process =====================|== Out-of-process ==| +----------+ +----------+ +--------+ +-----------+ +---------+ | | => | Work Ids | | | | Call Q | | Process | | | +----------+ | | +-----------+ | Pool | | | | ... | | | | ... | +---------+ | | | 6 | => | | => | 5, call() | => | | | | | 7 | | | | ... | | | | Process | | ... | | Local | +-----------+ | Process | | Pool | +----------+ | Worker | | #1..n | | Executor | | Thread | | | | | +----------- + | | +-----------+ | | | | <=> | Work Items | <=> | | <= | Result Q | <= | | | | +------------+ | | +-----------+ | | | | | 6: call() | | | | ... | | | | | | future | | | | 4, result | | | | | | ... | | | | 3, except | | | +----------+ +------------+ +--------+ +-----------+ +---------+ Executor.submit() called: - creates a uniquely numbered _WorkItem and adds it to the "Work Items" dict - adds the id of the _WorkItem to the "Work Ids" queue Local worker thread: - reads work ids from the "Work Ids" queue and looks up the corresponding WorkItem from the "Work Items" dict: if the work item has been cancelled then it is simply removed from the dict, otherwise it is repackaged as a _CallItem and put in the "Call Q". New _CallItems are put in the "Call Q" until "Call Q" is full. NOTE: the size of the "Call Q" is kept small because calls placed in the "Call Q" can no longer be cancelled with Future.cancel(). - reads _ResultItems from "Result Q", updates the future stored in the "Work Items" dict and deletes the dict entry Process #1..n: - reads _CallItems from "Call Q", executes the calls, and puts the resulting _ResultItems in "Result Q" z"Brian Quinlan (brian@sweetapp.com)N)_base)Queue)partial)format_exceptionFc$eZdZdZdZdZdZy) _ThreadWakeupcd|_tj|_t j d\|_|_y)NF)duplex)_closed threadingLock_lockmpPipe_reader_writerselfs A/opt/alt/python312/lib64/python3.12/concurrent/futures/process.py__init__z_ThreadWakeup.__init__Es. ^^% %'WWE%:" dlc|j5|js;d|_|jj|jjdddy#1swYyxYwNT)rr rcloserrs rrz_ThreadWakeup.closeJsC ZZ<<#  ""$ ""$ ZZs AAA'c|j5|js|jjddddy#1swYyxYw)Nr)rr r send_bytesrs rwakeupz_ThreadWakeup.wakeupVs, ZZ<< '',ZZs (>Ac|jr td|jjr6|jj |jjr5yy)Nz!operation on closed _ThreadWakeup)r RuntimeErrorrpoll recv_bytesrs rclearz_ThreadWakeup.clear[sG <<BC Cll! LL # # %ll!rN)__name__ __module__ __qualname__rrrr"rrrrDs; %- &rrcdattj}|D]\}}|j |D]\}}|j yr)_global_shutdownlist_threads_wakeupsitemsrjoin)r+_ thread_wakeupts r _python_exitr0bsP !'') *E!="1 r=ceZdZdZdZy)_RemoteTracebackc||_yNtb)rr8s rrz_RemoteTraceback.__init__s rc|jSr6r7rs r__str__z_RemoteTraceback.__str__s wwrN)r#r$r%rr:r&rrr4r4s rr4ceZdZdZdZy)_ExceptionWithTracebackcdjtt|||}||_d|j_d|z|_y)Nz """ %s""")r,rtypeexc __traceback__r8)rr@r8s rrz _ExceptionWithTraceback.__init__s? WW%d3ib9 :"& 2%rc>t|j|jffSr6) _rebuild_excr@r8rs r __reduce__z"_ExceptionWithTraceback.__reduce__sdhh000rN)r#r$r%rrDr&rrr<r<s &1rr<c&t||_|Sr6)r4 __cause__)r@r8s rrCrCs$R(CM JrceZdZdZy) _WorkItemc<||_||_||_||_yr6)futurefnargskwargs)rrJrKrLrMs rrz_WorkItem.__init__s   rNr#r$r%rr&rrrHrHrrHceZdZddZy) _ResultItemNc<||_||_||_||_yr6)work_id exceptionresultexit_pid)rrSrTrUrVs rrz_ResultItem.__init__s "   rNNNrNr&rrrQrQs!rrQceZdZdZy) _CallItemc<||_||_||_||_yr6)rSrKrLrM)rrSrKrLrMs rrz_CallItem.__init__s   rNrNr&rrrYrYrOrrYc.eZdZdZdfd ZfdZxZS) _SafeQueuez=Safe Queue set exception to the future object linked to a jobcD||_||_t| ||y)N)ctx)pending_work_itemsr.superr)rmax_sizer^r_r. __class__s rrz_SafeQueue.__init__s%"4* s+rct|trtt|||j}t dj dj||_|jj|jd}|jj||jj|yyt |E||y)Nz """ {}"""r>) isinstancerYrr?rAr4formatr,rFr_poprSr.rrJ set_exceptionr`_on_queue_feeder_error)reobjr8 work_itemrbs rrhz!_SafeQueue._on_queue_feeder_errors c9 %!$q'1aoo>B*>+@+@+MNAK//33CKKFI    % % '$  ..q1% G *1c 2r)r)r#r$r%__doc__rrh __classcell__rbs@rr\r\sG, 3 3rr\c'jKt|} ttj||}|sy|'w)z, Iterates over zip()ed iterables in chunks. N)ziptuple itertoolsislice) chunksize iterablesitchunks r _get_chunksrxs9 iB i&&r956  s13c2|Dcgc]}|| c}Scc}w)z Processes a chunk of an iterable passed to map. Runs the function passed to map() on a chunk of the iterable passed to map. This function is run in a separate process. r&)rKrwrLs r_process_chunkrzs #( (%$BI% (( (s c |jt||||y#t$r=}t||j}|jt|||Yd}~yd}~wwxYw)z.Safely send back the given result or exception)rUrTrVrTrVN)putrQ BaseExceptionr<rA) result_queuerSrUrTrVrir@s r_sendback_resultrsh9WV/88M N 9%a9W.68 9 99s! A'3A""A'cB| ||d}d} |j d}|$|j t jy||dz }||k\rt j} |j|ji|j}t||j||~~|y#t$r$tjjddYywxYw#t$r9} t| | j} t||j| | Yd} ~ rd} ~ wwxYw) aEvaluates calls from call_queue and places the results in result_queue. This worker is run in a separate process. Args: call_queue: A ctx.Queue of _CallItems that will be read and evaluated by the worker. result_queue: A ctx.Queue of _ResultItems that will written to by the worker. initializer: A callable initializer, or None initargs: A tuple of args for the initializer NzException in initializer:T)exc_inforblockr1)rUrVr|)r~rLOGGERcriticalgetr}osgetpidrKrLrMrrSr<rA) call_queuer initializerinitargs max_tasks num_tasksrV call_itemrrir@s r_process_workerrs3   " IH NNN.     RYY[ )   NII%99;  innA 0@0@AA \9+<+.weakref_cb4s# GGMM1 2  "r)_executor_manager_thread_wakeupr._shutdown_lock shutdown_lockweakrefrefexecutor_reference _processes processes _call_queuer _result_queuer _work_idswork_ids_queue_max_tasks_per_childmax_tasks_per_child_pending_work_itemsr_r`r)rexecutorrrbs rrz_ExecutorManagerThread.__init__&s &EE%44&*%7%7 # #*++h "C",,#..%22'00$,#@#@ #+">"> rc |j|j \}}}|r|j|y||j ||j du}|r5|jj|j }|j~|jx}rC|r&|j5|jdddn|jj~|jr=|j!|j|j"s|j%y6#t$r&}t|}|j|Yd}~yd}~wwxYw#1swYxYwr6)add_call_item_to_queuer~rterminate_brokenwait_result_broken_or_wakeupprocess_result_itemrVrrfr,rr_adjust_process_count_idle_worker_semaphorereleaseis_shutting_downflag_executor_shutting_downr_join_executor_internals)rr@cause result_item is_brokenprocess_exitedprs rrunz_ExecutorManagerThread.runSsY ++- -1,M,M,O )KE%%e,&((5!,!5!5T!A!**;+?+?@AFFH #668888%!//$::<0/!77??A $$&002 ++-..002_ ! (-%%e, 00/s#D84E*8 E'E""E'*E3c |jjry |jjd}|j|}|j j rH|jjt||j|j|jdn|j|=#tj$rYywxYw)NTFr)rfullrrr_rJset_running_or_notify_cancelr}rYrKrLrMqueueEmpty)rrSrks rrz-_ExecutorManagerThread.add_call_item_to_queues##% --111>!33G< ##@@BOO'' '2;,,2;..2;2B2B)D/3 (4 //8% ;;  sB::CCc|jj}|jjrJ|jj}||g}t |j j Dcgc]}|j}}tjj||z}d}d}d} ||vr |j} d}n||vrd}|jj| ||fScc}w#t$r} t| }Yd} ~ =d} ~ wwxYwNTF)rrr.r r)rvaluessentinelr connectionwaitrecvr~rr") r result_reader wakeup_readerreadersrworker_sentinelsreadyrrrr@s rrz3_ExecutorManagerThread.wait_result_broken_or_wakeups  ))11 %%----**22  -004T^^5J5J5L0MN0M1AJJ0MN ""7-=#=>  E ! .+002 ! e #I   "Iu,,%O! .(- .s.C(0C-- D 6 DD ct|tr[|jsJ|jj |}|j |js|j yy|jj |jd}|X|jr&|jj|jy|jj|jyyr6)rdintrrrfr,rr_rSrTrJrg set_resultrU)rrrrks rrz*_ExecutorManagerThread.process_result_items k3 '((* **"";/A FFH>>,,." //33K4G4GNI$(($$22;3H3HI$$// 0B0BC %rcV|j}txs|duxs |jSr6)rr(_shutdown_thread)rrs rrz'_ExecutorManagerThread.is_shutting_downs4**, !-H$4-,, .rcB|j}|d|_d|_d}td}|#t ddj |d|_|jjD]"\}} |jj|~$|jj|jjD]}|j!|j"j%|j'dy#tj$rYwxYw)NzKA child process terminated abruptly, the process pool is not usable anymoreTz^A process in the process pool was terminated abruptly while the future was running or pending.z ''' r>z''')broken)r_brokenrBrokenProcessPoolr4r,rFr_r+rJrgrInvalidStateErrorr"rr terminater_terminate_broken_join_executor_internals)rrrbperSrkrs rrz(_ExecutorManagerThread._terminate_brokens! **,  !1H )-H %H !67  ,"''%.)-/CM#'"9"9"?"?"A GY   ..s3#B %%'&&(A KKM) ))+ %%T%2%**    s4DDDch|j5|j|dddy#1swYyxYwr6)rr)rrs rrz'_ExecutorManagerThread.terminate_broken s%     " "5 )  s(1cd|j}|}d|_|jrii}|jj D]%\}}|j j r!|||<'||_ |jjyyy#tj$rYnwxYwd|_$r) rr_cancel_pending_futuresr_r+rJcancelr get_nowaitrr)rrnew_pending_work_itemsrSrks rrz2_ExecutorManagerThread.flag_executor_shutting_downs**,  (,H %//*,&*.*A*A*G*G*I&GY$++224:C.w7+J+A'++6680 !;;490s3BB'&B'c,|j}d}||krc|jdkDrOt||z D]#} |jjd|dz }%||kr|jdkDrMyyyy#tj $rY3wxYw)Nrr1)get_n_children_aliveranger put_nowaitrFull)rn_children_to_stopn_sentinels_sentis rshutdown_workersz'_ExecutorManagerThread.shutdown_workers*s!668 "44--/!3-0@@AOO..t4$)$B "44--/!3535 zzs A<<BBcf|j5|jdddy#1swYyxYwr6)rrrs rrz._ExecutorManagerThread.join_executor_internals8s#     ) ) +  s'0cF|s|j|jj|jj|jj|j j D]$}|r|j|j&yr6) rrr join_threadr.rrrr,)rrrs rrz/_ExecutorManagerThread._join_executor_internals<ss  ! ! #  ##%   "&&(A FFH)rcVtd|jjDS)Nc3<K|]}|jywr6)is_alive).0rs r z>_ExecutorManagerThread.get_n_children_alive..OsA)@A1::<)@s)sumrrrs rrz+_ExecutorManagerThread.get_n_children_aliveMs A)>)>)@AAAr)F)r#r$r%rlrrrrrrrrrrrrrrmrns@rrrsN +Z2h.-<D..-3^*94 ,"Brrctrtrttda ddl} t j d}|dk(ry|dk\ryd|zatt#t$rdattwxYw#ttf$rYywxYw)NTrzxThis Python build lacks multiprocessing.synchronize, usually due to named semaphores being unavailable on this platform.SC_SEM_NSEMS_MAXz@system provides too few semaphores (%d available, 256 necessary)) _system_limits_checked_system_limitedNotImplementedErrormultiprocessing.synchronize ImportErrorrsysconfAttributeError ValueError)multiprocessing nsems_maxs r_check_system_limitsrVs %o6 6!3*JJ12 B C 46?@O o ..+ 3 F "/22 3 J 'sAA7A47B B c#jK|D]*}|j|s|j|r,yw)z Specialized implementation of itertools.chain.from_iterable. Each item in *iterable* should be a list. This function is careful not to keep references to yielded objects. N)reverserf)iterableelements r_chain_from_iterable_of_listsrvs0 ++- s 333ceZdZdZy)rzy Raised when a process in a ProcessPoolExecutor terminated abruptly while a future was in the running state. N)r#r$r%rlr&rrrrsrrceZdZ ddddZdZdZdZdZdZe jjje_ dd d fd Z dd d dZ e jjje _ xZS)ProcessPoolExecutorN)rct|Ptjxsd|_tj dk(ret t|j|_nE|dkr tdtj dk(r|tkDrtdt||_|,|tjd}ntj}||_ |jjd d k7|_ |t|s td ||_||_|Ut%|t&s td |dkr td |jjd d k(r td||_d|_i|_d|_t1j2|_t1j6d|_d|_d|_i|_d|_ tC|_"|jtFz}tI||j|j>|jD|_%d|jJ_&|jO|_(tSjT|_+y)aRInitializes a new ProcessPoolExecutor instance. Args: max_workers: The maximum number of processes that can be used to execute the given calls. If None or not given then as many worker processes will be created as the machine has processors. mp_context: A multiprocessing context to launch the workers created using the multiprocessing.get_context('start method') API. This object should provide SimpleQueue, Queue and Process. initializer: A callable used to initialize worker processes. initargs: A tuple of arguments to pass to the initializer. max_tasks_per_child: The maximum number of tasks a worker process can complete before it will exit and be replaced with a fresh worker process. The default of None means worker process will live as long as the executor. Requires a non-'fork' mp_context start method. When given, we default to using 'spawn' if no mp_context is supplied. Nr1win32rz"max_workers must be greater than 0zmax_workers must be <= spawnF) allow_noneforkzinitializer must be a callablez&max_tasks_per_child must be an integerz max_tasks_per_child must be >= 1zpmax_tasks_per_child is incompatible with the 'fork' multiprocessing start method; supply a different mp_context.)rar^r_r.T),rr cpu_count _max_workerssysplatformmin_MAX_WINDOWS_WORKERSrr get_context _mp_contextget_start_method#_safe_to_dynamically_spawn_childrencallable TypeError _initializer _initargsrdrr_executor_manager_threadrrr r r Semaphorerr _queue_countrrrrEXTRA_QUEUED_CALLSr\r _ignore_epipe SimpleQueuerrrr)r max_workers mp_contextrrr queue_sizes rrzProcessPoolExecutor.__init__sX(    "  3!D ||w&$'(<(,(9(9%;!a !EFF,,')22 -.B-CDFF!,D   ".^^G4 ^^- %  11U1CvM 0  "8K+@<= ='!  *137 HII$) !CDD00E0BfL "CDD%8!)-%!&'nn.&/&9&9!&<# #% ',$0=, &&);; %T%5%5#77>>@*.&'335rc|jd|js|jt||_|jj |j t |j<yyr6)rr_launch_processesrstartrr*rs r_start_executor_manager_threadz2ProcessPoolExecutor._start_executor_manager_threads`  ( ( 0;;&&(,B4,HD )  ) ) / / 144 T:: ; 1rc|jjdryt|j}||jkr|j yy)NF)blocking)racquirelenrr_spawn_process)r process_counts rrz)ProcessPoolExecutor._adjust_process_countsK  & & . . . > DOO, 4,, ,    ! -rc|jrJdtt|j|jD]}|j y)NzhProcesses cannot be fork()ed after the thread has started, deadlock in the child processes could result.)rrr+rrr,)rr-s rr%z%ProcessPoolExecutor._launch_processessO00 A@ A0s4??+T->->?A    !@rc|jjt|j|j|j |j |jf}|j||j|j<y)N)targetrL) rProcessrrrrrrr&rpid)rrs rr,z"ProcessPoolExecutor._spawn_processsm    $ $"""$$##..++ - % .  !"rcR|j5|jrt|j|jr t dt r t dt j}t||||}||j|j<|jj|j|xjdz c_ |jj|jr|j!|j#|cdddS#1swYyxYw)Nz*cannot schedule new futures after shutdownz6cannot schedule new futures after interpreter shutdownr1)rrrrrr(rFuturerHrrrr}rrrrr')rrKrLrMfws rsubmitzProcessPoolExecutor.submit"s  ||' 55$$"#OPP"$:;; A!Rv.A:;D $ $T%6%6 7 NN  t00 1    "   0 0 7 7 977**,  / / 1+! s DDD&r1)timeoutrtc|dkr tdt| tt|t |d|i|}t |S)ajReturns an iterator equivalent to map(fn, iter). Args: fn: A callable that will take as many arguments as there are passed iterables. timeout: The maximum number of seconds to wait. If None, then there is no limit on the wait time. chunksize: If greater than one, the iterables will be chopped into chunks of size chunksize and submitted to the process pool. If set to one, the items in the list will be sent one at a time. Returns: An iterator equivalent to: map(func, *iterables) but the calls may be evaluated out-of-order. Raises: TimeoutError: If the entire result iterator could not be generated before the given timeout. Exception: If fn(*args) raises for any values. r1zchunksize must be >= 1.rt)r8)rr`maprrzrxr)rrKr8rtruresultsrbs rr:zProcessPoolExecutor.map;sQ* q=67 7'+gnb9)9J J&-/-W55rF)cancel_futuresc|j5||_d|_|j|jj ddd|j |r|j j d|_d|_|j|r|jjd|_d|_ d|_y#1swY}xYwr) rrrrrrr,rrrr)rrr<s rshutdownzProcessPoolExecutor.shutdownXs  +9D ($(D !33?44;;= !  ( ( 4  ) ) . . 0)-%    )d    $ $ &!/3,#! s 5B>>C)NNNr&)T)r#r$r%rr'rr%r,r7rExecutorrlr:r>rmrns@rrrs~48,.j'GKj'X5 "" #.^^**22FN*.!6:4E4(~~..66HrrrWr6)2rl __author__rconcurrent.futuresrrrrmultiprocessing.connectionmultiprocessing.queuesrr r functoolsrrrr tracebackrWeakKeyDictionaryr*r(rr0_register_atexitrr Exceptionr4r<rCobjectrHrQrYr\rxrzrrThreadrrrrrBrokenExecutorrr?rr&rrrLs?(T2 $ "( &-7,,.&&< <( y 1 1!&!33, )DH" 93luBY--uBp /@  ,,c7%..c7r