bj"3ddlZddlZddlmZddlmZmZmZmZm Z m Z m Z ddl m Z ddlmZmZerddlmZd ed ed e eee efd ed ede e edede gee ffdZdddddddddde d ed ed e eee efd ed ede e edee fdZdeded eee efdede gee ffdZdedede gee ffdZddde dee fdZddde dedee fdZddde fdZdddddde d eee efdededee f dZd d!ddd"ed#eee efd eee eee effdede eeee efff d$Zddde d%ed&ede f d'Z d(Z!y))N)partial) TYPE_CHECKINGCallableDictIteratorListOptionalTuple)Example)_doc_to_biluo_tags_with_partialsplit_bilu_label)Language lower_level orth_level orth_variantswhitespace_levelwhitespace_per_tokenwhitespace_variantsreturnrc .tt||||||S)a\Create a data augmentation callback that uses orth-variant replacement. The callback can be added to a corpus or other data iterator during training. lower_level (float): The percentage of texts that will be lowercased. orth_level (float): The percentage of texts that will be augmented. orth_variants (Optional[Dict[str, List[Dict]]]): A dictionary containing the single and paired orth variants. Typically loaded from a JSON file. whitespace_level (float): The percentage of texts that will have whitespace tokens inserted. whitespace_per_token (float): The number of whitespace tokens to insert in the modified doc as a percentage of the doc length. whitespace_variants (Optional[List[str]]): The whitespace token texts. RETURNS (Callable[[Language, Example], Iterator[Example]]): The augmenter. rrrrrr)rcombined_augmenterrs `/home/spazio/kingdom_calls/work5/transenv/lib/python3.12/site-packages/spacy/training/augment.pycreate_combined_augmenterr s&, #)1/ grnlpexamplec #Ktj|kr t||}|rtj|krs|j}|j} t |j | dd<t ||| d|d\} } | | d<|j|j| | }|rtj|krwttt|j |zD]J} t||tj|tjdt|j }L|yw)Ndoc_annotationentitiestoken_annotationFlowerr)randommake_lowercase_varianttextto_dictr referencemake_orth_variants from_dictmake_docrangeintlenmake_whitespace_variantchoice randrange) rrrrrrrrraw_text orig_dict variant_textvariant_token_annot_s rrr.s.}}$(g6:5<<OO% 2Q   3 "#J/-?   ( )  - ) ))< $%##CLL$> Jv}}1AAs3w0014HHIJ A- 12  C(9(9$:; G  MsEElevelr%c(tt|||S)aCreate a data augmentation callback that uses orth-variant replacement. The callback can be added to a corpus or other data iterator during training. level (float): The percentage of texts that will be augmented. lower (float): The percentage of texts that will be lowercased. orth_variants (Dict[str, List[Dict]]): A dictionary containing the single and paired orth variants. Typically loaded from a JSON file. RETURNS (Callable[[Language, Example], Iterator[Example]]): The augmenter. )rr9r%)rorth_variants_augmenter)r9r%rs rcreate_orth_variants_augmenterr<Us }EQV rc$tt|S)a3Create a data augmentation callback that converts documents to lowercase. The callback can be added to a corpus or other data iterator during training. level (float): The percentage of texts that will be augmented. RETURNS (Callable[[Language, Example], Iterator[Example]]): The augmenter. r9)rlower_casing_augmenterr>s rcreate_lower_casing_augmenterr@fs ) 77rc#K|ywN)rrs r dont_augmentrDrs Msc#`Ktj|k\r|yt||ywrB)r&r')rrr9s rr?r?vs(}}% $S'22s,.c,|j}t|j|dd<|j|jj }|jDcgc]}|j c}|dd<|j||Scc}w)Nr!r"r#ORTH)r)r r*r-r(r%lower_r,)rr example_dictdocts rr'r's??$L1P2L!":. ,,w||))+ ,CBIBSBS/TQ/TL#$V,   S, //0Us#B)r9r%c#dKtj|k\r|y|j}|j}t|j|dd<t |||d||duxrtj|k\}}||d<|j |j||yw)Nr!r"r#r$)r&r(r)r r*r+r,r-) rrrr9r%r4r5r6r7s rr;r;s}}% <<OO% 2Q   3 "#J/-?   ( ) $&B6==?U+B - ) ))< $% \ :IFFsB.B0Fr$raw token_dictcb|jdg}|jdg}|s||fS|r-|Dcgc]}|j}}|j}|s ||d<||fS|jdg}|D cgc]} tj| d} } t t |D]?} t t |D]&} || || dvs|| || dvs| | || <(A|jdg} | D cgc]} tj| d} } t t |D]} t t | D]} || | | dvs|| t jj| | dvs>% $D J $)*q**iik " 6J   Xr *D;?@aV]]1Z=1@M@#e*%;s4y) ;IX$y/&"99(OtIz'BB"/ ":h  ;;   Xr *D;?@aV]]1Z=1@M@#e*%Es4y) EIH~i!88U>..tIz/JK>L"==!Q0tIv./14#Iv6<x.H+ m.>x.H+ m J.$ 2%6!-1F##Hf5 Z vh 3wx/''.5  51 u ''07""8Z8 w ''.5  73 u ''07""8R0 w '''E q= v  % %h 2 v  % %h1 =s:f-./ +A&!!$06"1%*% + 5  51 v  u '' 5< ''%8 | $ %j 1C   S\\#. ==rcXd}t|d|dD]\}}||z }|s|dz }|S)z2Construct modified raw text from words and spaces.rwrGrq )zip)rNrMorthspacys rrZrZQsG C:f-z'/BC e t   3JC Jr)"rVr& functoolsrtypingrrrrrr r rr iob_utilsr rlanguagerfloatstrrrr<r@rDr?r'r;boolr+r/r1rZrCrrrs QQQH#Dd4j12    "$s), z7#Xg%667J59!"%/3$ $ $ $  $ Dd4j12 $$ $"$s),$g$N /3CdO/D z7#Xg%667" 8  8 z7#Xg%667 8j7x7H3 3%3163 g30 0W0 G G GT$Z(G  G  GgGB 4 4 4S$s)^$4T$sDI~"6778 4  4 3S$s)^$ $%4ns> s> s>s> s>  s>lr