`jj%<ddlZddlmZddlmZmZmZmZddlZddl m Z ddl m Z ddl mZmZddlmZdd lmZd d lmZmZmZmZmZmZej6d ej6d eddedddedddddeddddeddddedd d!d"edd#d$d%dd&ed'd(d)d*edd+d,d-ed.d/d0d1f d2ed3ed4eed5eed6ed7ed8eed9ed:ed;efd<Z dQd2ed3ed4eed6ed7ed8eed9ed=ed;ed:ed>eeeffd?Z ifd.dd@dAeeefdBeeefd;ed=ed>eeeff dCZ! dRdDeedEedFedGedHedIedJefdKZ"dLe dAeeeee#ffdMedNed>df dOZ$dLe dAeeeee#ffd>dfdPZ%y)SN)Path)AnyDictListOptional)fix_random_seed)Printer)displacyutil)Doc)Corpus)ArgOptapp benchmark_cli import_code setup_gpuaccuracyevaluate.zModel name or path)helpz3Location of binary evaluation data in .spacy formatT)rexistsz--outputz-ozOutput JSON file for metricsF)rdir_okayz--codez-czNPath to Python file with additional code (registered functions) to be importedz--gpu-idz-gzGPU ID or -1 for CPUz--gold-preprocz-GzUse gold preprocessingz--displacy-pathz-dpz+Directory to output rendered parses as HTML)rr file_okayz--displacy-limitz-dlz!Limit of parses to render as HTMLz--per-componentz-PzSReturn scores per component, only applicable when an output JSON file is specified.scz --spans-keyz-skz*Spans key to use when evaluating Doc.spansmodel data_pathoutput code_pathuse_gpu gold_preproc displacy_pathdisplacy_limit per_component spans_keyc Dt|t||||||||d|  y)am Evaluate a trained pipeline. Expects a loadable spaCy pipeline and evaluation data in the binary .spacy format. The --gold-preproc option sets up the evaluation examples with gold-standard sentences and tokens for the predictions. Gold preprocessing helps the annotations align to the tokenization, and may result in sequences of more consistent length. However, it may reduce runtime accuracy due to train/test skew. To render a sample of dependency parses in a HTML file, set as output directory as the displacy_path argument. DOCS: https://spacy.io/api/cli#benchmark-accuracy F)r!r#r$r%r&r'silentr(N)rr) rr r!r"r#r$r%r&r'r(s \/home/spazio/kingdom_calls/work5/transenv/lib/python3.12/site-packages/spacy/cli/evaluate.py evaluate_clir,s3r   !#%# r*returnc t|| } tt||tj|}tj|} tj|}|j s| j d|d|r$|j s| j d|dt||} tj|} t| | }| j|| }| r*|}|| jd n"| jd nid d d ddddddddddddddddddd d!d"d#d$d%d&d'd(|d)d*d(|d+d,d(|d-d.d/i}i}i}|jD]\}}||vs |d k(r|d0z|jd1d2zd3z}t||t t"fr|d/k(r ||d4||<n||d5zd6||<nd7||<|||t%j&d8d9|j)<| j+|d:;t-||||<}|r| j.Dcgc]}| j1|j2}}t| j5d=|d|D}d>|v}d?|v}d@|v}t7|||||||A| j9dB|dC|| *t;j<| || j9dD| |Scc}w)ENno_printpretty)r*zEvaluation data not foundr)exitsz(Visualization output directory not found)r$)r'zdThe per-component option is enabled but there is no output JSON file provided to save the scores to.z7Per-component scores will be saved to output JSON file.TOK token_accTAGtag_accPOSpos_accMORPH morph_accLEMMA lemma_accUASdep_uasLASdep_laszNER Pents_pzNER Rents_rzNER Fents_fTEXTCAT cats_scorezSENT Psents_pzSENT Rsents_rzSENT Fsents_fzSPAN Pspans__pzSPAN R_rzSPAN F_fSPEEDspeedz (cats_score_descunk)z.0fd.2f-z[\s/]_Results)titler(r*c3HK|]}|jjyw)N) referencetext).0exs r+ zevaluate..sV2R\\..Vs "parsernerspancat) model_namelimitdepsentsspansz Generated z parses as HTMLzSaved results to )r rrr ensure_pathrfailr load_modellistrwarninfoitemsget isinstanceintfloatresublowertablehandle_scores_per_type pipe_names get_pipe_metafactorypipe render_parsesgoodsrsly write_json)rr r!r#r$r%r&r*r(r'msg output_pathcorpusnlp dev_datasetscoresdatametricsresultsmetrickeyr{ factory_namesdocs render_deps render_ents render_spanss r+rrWs 6f* 5C gf%  +I""6*K$$]3M     ,iqA]113 ;]RST IL 9F //% Cvc{#K \\+]\ CF > HHv  HHN O ; 9  9  [  [  9   9  X  X  X  |  i  i  i  yk, yk,! " yk,# $ W% ("==? GKFCf},&#d]VZZ8I5-QQTWWFfSkC<8g~-3C[,=-3C[3->s,C&)GFO;A#;RVVHc399;78 G ' +%fdiPVWEH^^TT**4088T TCHHV_n9UVVW-/ },  M1      :n-_=}M d+ $[M23 K)Us"K*rYrrct|| }d|vr|drt||ddd|d|d<d|vr|drt||ddd|d|d<d|vr|drt||dd d|d|d<d |d |vr.|d |d r%t||d |d d d|d |d |d |d <d |vr|d rt||d dd|d |d <d|vr|drt||d|d|d<|S)Nr0morph_per_featr:featdep_las_per_typer@type ents_per_typeNERrJ _per_typeSPANScats_f_per_typez Textcat Flabelcats_auc_per_type)r print_prf_per_typeprint_textcats_auc_per_cat)rrr(r*rs r+rwrws 6f* 5C6! " # sF+;$v N'-.@'AD# $& / " sF?$;UF K$*?$;D !  {)$. F9+Y/ 0 VfYKy9:GV 396)I9V2WD6)I. /F" # $ sF+<$={G T&,->&?D" #f$ % & &sF3F,G H(./B(CD$ % Mr-rrrcrdrerfrgcF||djd<|rKtj|d|dd}|dz jdd 5}|j |ddd|rNtj|d|d dd di }|d z jdd 5}|j |ddd|rLtj|d|dd}|dz jdd 5}|j |dddyy#1swYxYw#1swYdxYw#1swYyxYw)NrrXentT)stylepagez entities.htmlwutf8)encodingdepcompact)rroptionsz parses.htmlspanz spans.html) user_datar renderopenwrite) rrrcrdrerfrghtmlfile_s r+r|r|s2",DGg tFU|5tDO+ 1 1# 1 G 5 KK    %LD9d:K M) / /f / E  KK   tFU|6EL( . .sV . D  KK            s$C?D #D?D DD rnamerc  g}|jD]S\}}|g}dD]5}||} |jt| ttfr| dzdn| 7|j|U|j |dd|d|dy) N)prfrSrT)PRF)lrrrz (per rRheaderalignsrX)rnappendrprqrrrv) rrrrrrvaluerowkvs r+rrs Dlln Ue  OAaA JJ:a#u+F!c'#A N O C  II "#fTF!$ r-c |j|jDcgc]#\}}|t|ttfr|dn|f%c}}dddycc}}w)NrT)rzROC AUC)rrzTextcat ROC AUC (per label)r)rvrnrprrrq)rrrrs r+rr s_II  1jUCL91S'q A + s(A )NrFNrTrF)rTTT)&rspathlibrtypingrrrrr~ thinc.apirwasabir rr r tokensr trainingr_utilrrrrrrcommandstrrqboolr,rrwr|rrrrr-r+rsy ,, %GG ZS34 GPT! j$%Ce!$  ] ! r:t2HI ,D%(  : % ,O  b   mU)UIA AA TN A~AA !A&D>'A67A<=AHIAAN"$(^ ^^ TN^ ^  ^ D> ^^ ^^^ #s(^^F" " cN" sCx." "  "  #s(^ "P s)       6 sDe$445=@HK $   sDe$445   r-