`j.ddlZddlZddlmZddlmZmZmZmZm Z m Z m Z m Z ddl mZddlmZddlmZmZddlmZdd lmZmZdd lmZdd lmZmZdd lmZdd lm Z m!Z!ddl"m#Z#ddl$m%Z%deedee&effdZ'dZ(Gdde%Z)dZ*y)N)Path)AnyCallableDictIterableListOptionalTupleUnion)Model)util)ErrorsWarnings)Language)Lookups load_lookups)Scorer)DocToken)Example)SimpleFrozenListlogger)Vocab)Pipeexamplesreturnc 0tj|dfi|S)Nlemma)rscore_token_attr)rkwargss c/home/spazio/kingdom_calls/work5/transenv/lib/python3.12/site-packages/spacy/pipeline/lemmatizer.pylemmatizer_scorer$s  " "8W ? ??ctSN)r$r%r#make_lemmatizer_scorerr)s r%ceZdZdZededeeeeeffdZ d%dde dde d e e d eded e d e edd fdZedZdedefdZ d&d d dde egeefde ede efdZej4fdedd fdZdedeefdZdedeefdZdede fdZe dde!ee"fdeefd Z#e dde!ee"fdeeddfd!Z$e ddeede%fd"Z&e dd#e%deeddfd$Z'y )' Lemmatizerz The Lemmatizer supports simple part-of-speech-sensitive suffix rules and lookup tables. DOCS: https://spacy.io/api/lemmatizer moderc6|dk(rdggfS|dk(rdgddgfSggfS)a Returns the lookups configuration settings for a given mode for use in Lemmatizer.load_lookups. mode (str): The lemmatizer mode. RETURNS (Tuple[List[str], List[str]]): The required and optional lookup tables for this mode. lookup lemma_lookuprule lemma_rules lemma_exc lemma_indexr()clsr,s r#get_lookups_configzLemmatizer.get_lookups_config$s< 8 #$b) ) V^"Ok=%AB BBxr%r.F)r, overwritescorervocabmodelnamer6r7Nc||_||_||_||_t |_||_d|_|jdk(r|j|_ nv|jdk(r|j|_ nU|jd}t||s)ttjj!|t#|||_ i|_||_y)a&Initialize a Lemmatizer. vocab (Vocab): The vocab. model (Model): A model (not yet implemented). name (str): The component name. Defaults to "lemmatizer". mode (str): The lemmatizer mode: "lookup", "rule". Defaults to "lookup". overwrite (bool): Whether to overwrite existing lemmas. Defaults to `False`. scorer (Optional[Callable]): The scoring method. Defaults to Scorer.score_token_attr for the attribute "lemma". DOCS: https://spacy.io/api/lemmatizer#init Fr.r0 _lemmatize)r,N)r8r9r:_moderlookupsr6 _validatedr,lookup_lemmatize lemmatizerule_lemmatizehasattr ValueErrorrE1003formatgetattrcacher7)selfr8r9r:r,r6r7 mode_attrs r#__init__zLemmatizer.__init__3s.    y " 99 !22DN YY& !00DN99+Z0I4+ !4!4$!4!?@@$T95DN  r%c|jSr')r=)rIs r#r,zLemmatizer.mode]s zzr%doccN|js|jtj|j } |D]7}|j s|j dk(s|j|d|_9|S#t$r }||j||g|Yd}~yd}~wwxYw)zApply the lemmatizer to one document. doc (Doc): The Doc to process. RETURNS (Doc): The processed Doc. DOCS: https://spacy.io/api/lemmatizer#call rN) r?_validate_tablesrE1004get_error_handlerr6r rAlemma_ Exceptionr:)rIrM error_handlertokenes r#__call__zLemmatizer.__call__as  ! !&,, /..0  5 <>>U[[A%5#'>>%#8#;EL <J 5 $))TC5! 4 4 5s A;A;; B$BB$)nlpr> get_examplesrXr>c|j|j\}}|tjdt |j j |}t |j j |d}|jD]#}|j||j|%||_ |jtjy)aInitialize the lemmatizer and load in data. get_examples (Callable[[], Iterable[Example]]): Function that returns a representative sample of gold-standard Example objects. nlp (Language): The current nlp object the component is part of. lookups (Lookups): The lookups object containing the (optional) tables such as "lemma_rules", "lemma_index", "lemma_exc" and "lemma_lookup". Defaults to None. Nz2Lemmatizer: loading tables from spacy-lookups-data)langtablesF)r[r\strict)r5r,rdebugrr8r[r\ set_table get_tabler>rOrrP)rIrYrXr>required_tablesoptional_tablesoptional_lookupstables r# initializezLemmatizer.initializets ,0+B+B499+M( ? LLM N" PG+ZZ___U  *00 L!!%)9)C)CE)JK L  fll+r% error_messagec|j|j\}}|D]K}||jvst|j |j||jj d|_y)z8Check that the lookups are correct for the current mode.)r,r\foundTN)r5r,r>rDrFr\r?)rIrfrarbrds r#rOzLemmatizer._validate_tablessv+/+B+B499+M($ EDLL( !((!YY."ll11) r%rUc|jjdi}|j|j|j}t |t r|g}|S)zLemmatize using a lookup-based approach. token (Token): The token to lemmatize. RETURNS (list): The available lemmas for the string. DOCS: https://spacy.io/api/lemmatizer#lookup_lemmatize r/)r>r`gettext isinstancestr)rIrU lookup_tableresults r#r@zLemmatizer.lookup_lemmatizesJ||--nbA !!%**ejj9 fc "XF r%c|j|j|jjf}||jvr|j|S|j }|j j}|dvr9|dk(r#tjtj|jgS|j|r|jgS|jjdi}|jjdi}|jjdi}t|j!||j!||j!|fs|dk(r|gS|jgS|j!|i}|j!|i} |j!|i} |} |j}g} g} | D]\}}|j#|s|dt%|t%|z |z}|s8||vs|j's)||vr| j)d|c| j+|u| j+|t-t.j1| } | j!|gD]}|| vs| j)d|| s| j3| | s| j+| | |j|<| S) zLemmatize using a rule-based approach. token (Token): The token to lemmatize. RETURNS (list): The available lemmas for the string. DOCS: https://spacy.io/api/lemmatizer#rule_lemmatize )eolspacerqr3r2r1propnNr)orthposmorphkeyrHrkpos_lowerwarningswarnrW108 is_base_formr>r`anyrjendswithlenisalphainsertappendlistdictfromkeysextend)rIrU cache_keystringuniv_pos index_table exc_table rules_tableindex exceptionsrulesorigforms oov_formsoldnewforms r#rBzLemmatizer.rule_lemmatizes{ZZEKKOO<  "::i( (::##% + +2~ hmm,LLN# #   U #LLN# #ll,,]B? LL**;; ll,,]B? ) h')  7"x ''"-]]8R0 "-  +HCs#6F c#h 67#=U]$,,.u} Q- T*$$T* +T]]5)* NN62. &D5  Q% & LL # LL  % 9 r%cy)aCheck whether the token is a base form that does not need further analysis for lemmatization. token (Token): The token. RETURNS (bool): Whether the token is a base form. DOCS: https://spacy.io/api/lemmatizer#is_base_form Fr()rIrUs r#r~zLemmatizer.is_base_formsr%excludepathrc\i}fd|d<fd|d<tj||y)zSerialize the pipe to disk. path (str / Path): Path to a directory. exclude (Iterable[str]): String names of serialization fields to exclude. DOCS: https://spacy.io/api/lemmatizer#to_disk c>jj|SNr)r8to_diskprrIs r#z$Lemmatizer.to_disk..stzz'9'9!W'9'Mr%r8c:jj|Sr')r>rrrIs r#rz$Lemmatizer.to_disk.. s)=)=a)@r%r>N)rr)rIrr serializes` ` r#rzLemmatizer.to_disks. M '@ ) T9g.r%c~i}fd|d<fd|d<tj||jS)aHLoad the pipe from disk. Modifies the object in place and returns it. path (str / Path): Path to a directory. exclude (Iterable[str]): String names of serialization fields to exclude. RETURNS (Lemmatizer): The modified Lemmatizer object. DOCS: https://spacy.io/api/lemmatizer#from_disk c>jj|Sr)r8 from_diskrs r#rz&Lemmatizer.from_disk..s)=)=a)=)Qr%r8c:jj|Sr')r>rrs r#rz&Lemmatizer.from_disk..s4<<+A+A!+Dr%r>)rrrO)rIrr deserializes` ` r#rzLemmatizer.from_disk s?8: Q G!D I t['2  r%czi}fd|d<jj|d<tj|S)zSerialize the pipe to a bytestring. exclude (Iterable[str]): String names of serialization fields to exclude. RETURNS (bytes): The serialized object. DOCS: https://spacy.io/api/lemmatizer#to_bytes c<jjSr)r8to_bytes)rrIsr#rz%Lemmatizer.to_bytes..'sTZZ%8%8%8%Ir%r8r>)r>rr)rIrrs`` r#rzLemmatizer.to_bytess9 I '#||44 )}}Y00r% bytes_datac~i}fd|d<fd|d<tj||jS)aLoad the pipe from a bytestring. bytes_data (bytes): The serialized pipe. exclude (Iterable[str]): String names of serialization fields to exclude. RETURNS (Lemmatizer): The loaded Lemmatizer. DOCS: https://spacy.io/api/lemmatizer#from_bytes c>jj|Sr)r8 from_bytes)brrIs r#rz'Lemmatizer.from_bytes..7s)>)>q')>)Rr%r8c:jj|Sr')r>r)rrIs r#rz'Lemmatizer.from_bytes..8s4<<+B+B1+Er%r>)rrrO)rIrrrs` ` r#rzLemmatizer.from_bytes+s?8: R G!E I  K9  r%) lemmatizerr')(__name__ __module__ __qualname____doc__ classmethodrmr rr5r$rr r boolrrKpropertyr,rrWrrrrrerE912rOrr@rBr~rr rrrbytesrrr(r%r#r+r+s( c eDItCy4H.I  $! ( %5((( (  (("( (T5C5C5*CG,#'%) ,xHW,=(=>?,h  , '" ,85;KK c D  e S DEDd3iDL % D CSBT /#t)$ /2:3- / CSBT#t)$2:3- $4D3E 18C= 1% 1>N=O-5c] r%r+cx|dk(r!tjd}|jStdtd|)Nmake_lemmatizerzspacy.pipeline.factorieszmodule z has no attribute ) importlib import_modulerAttributeErrorr)r:modules r# __getattr__r?sA   (()CD%%% 78*,>tfE FFr%)+rr{pathlibrtypingrrrrrr r r thinc.apir rqrerrorsrrlanguagerr>rrr7rtokensrrtrainingrrrr8rpiperrmr$r)r+rr(r%r#rsuNNN%++@x0@tCH~@__F Gr%