jg.UddlZddlmZddlmZmZmZmZmZm Z ddl m Z ddl m Z ddlmZmZeeefZeeeeefZeeeeefZdd d d Zeeeeeeffed <d dddededd dZeeeeeeffed<e j6dedfedededede eededd ddedededededededefdZe j6dedfededededede eedd ddedededededededefd Ze j6d!edede eedd d"dededededef d#ZGd$d%eZ gd&Z!y)'N) defaultdict)DictListOptionalTupleUnioncast) get_array_ops)registry)FloatsXd GeneratorT?)L2L2_is_weight_decay grad_clip SGD_DEFAULTSgMbP?g?g+?g:0yE>rr) learn_ratebeta1beta2epsrrr ADAM_DEFAULTSzRAdam.v1rrrrr)rrrrrr use_averagesrc *t||||||||d S)NT)rrrrrrr use_radam Optimizer)rrrrrrrrs Z/home/spazio/kingdom_calls/work5/transenv/lib/python3.12/site-packages/thinc/optimizers.pyRAdamr s,  - !  zAdam.v1)rrrrrrrc *t||||||||d S)NF)rrrrrrrrr)rrrrrrrrs rAdamr#8s,   -!  r!zSGD.v1)rrrrc &t||||dd|S)Nr)rrrrrrr)rrrrrs rSGDr%Qs&  -! r!ceZdZUdZeeefed<eeefed<eeeefed<ee e fed<eee fed<eee fed<e ed<e ed <e ed <e ed <e ed <e ed <e ed<e ed<eeeeed<gdZed ededed edddddd ed ededed edede de de fdZdZdZdddee e fdeded e fd!Zd"Zd#Zy$)%rzDo various flavours of stochastic gradient descent, with first and second order momentum. Currently support 'vanilla' SGD, Adam, and RAdam. mom1mom2averages schedules nr_update last_seenrrb1b2rrrr _radam_buffer)r'r(r)r*r+r,rrr-r.rrrrr/rrTF)rrrrrrrrrci|_i|_|ri|_nd|_i|_t t |_t t |_|jd||jd||jd||jd||jd||jd|||_ | |_ tdD cgc]} gd c} |_ ycc} w) al Initialize an optimizer. learn_rate (float): The initial learning rate. L2 (float): The L2 regularization term. beta1 (float): First-order momentum. beta2 (float): Second-order momentum. eps (float): Epsilon term for Adam etc. grad_clip (float): Gradient clipping. use_averages (bool): Whether to track moving averages of the parameters. use_radam (bool): Whether to use the RAdam optimizer. L2_is_weight_decay (bool): Whether to interpret the L2 parameter as a weight decay term, in the style of the AdamW optimizer. Nrrr-r.rr )NNN) r'r(r)r*rintr+r,_set_attr_or_schedulerrranger/) selfrrrrrrrrr_s r__init__zOptimizer.__init__s6  DM DM$S)$S) ""; : ""<< ""4/ ""4/ ""5#. ""4,""4:?)DQ0DDs C*cDt|tttfrt |||yt|t r t |}||j|< t ||t|y#ttf$r$}d|dt|d|}t|d}~wwxYw)NzInvalid schedule for 'z' (z) ) isinstancefloatboolr2setattrlistiterr*next StopIteration TypeErrortype ValueError)r5namevalueeerrs rr3zOptimizer._set_attr_or_schedules eeT3/ 0 D$ &%&U #(DNN4  &dDK0!9- &.tfCU }CsK o% &sA,,B;BBc|jjD]\}} t|}t ||| y#t$rt ||}Y'wxYw)N)r*itemsr?r@getattrr<)r5keyschedulerEs rstep_scheduleszOptimizer.step_scheduless[!^^113 &MC +X D#u %  &! +c* +s =AAr)lr_scalerKweightsgradientrNct|dkr||fSt|}|j|xxdz cc<|j|}|jdk7r|js||j|zz }|j r|j ||j }|jr|j||||||\}}nb|jdkDr)|jdkDr|j||||||\}}n*|jdkDrt|||jz|zz}|dz}|jdk7r.|jr"|||jz|jz|zz}|jX||jvr*|j|j d|j|<|j#|j|||||fS)zCall the optimizer with weights and a gradient. The key is the identifier for the parameter, usually the node ID and parameter name. r rrfloat32)dtype)lenr r+rrr clip_gradientr_radamr-r._adamNotImplementedErrorrr)allocshapeupdate_averages)r5rKrOrPrNopsnr_upds r__call__zOptimizer.__call__s x=1 H$ $G$ sq $ 77a< 7 7 ') )H >>((4>>BH >> $ Wh#v! GXWWs]tww} $ Wh#v! GXWWs]% % x$//1H<  #HQK A:^$)u^44uT{BZOO FFKK +eEl:E 9*uT{2goF FJ ]^$)u^44uT{BZOO 9*uT{2W< rs #;;#& S#X 5$u+y0 1 d3i* + 4 d3eT3.//0    t k*5 tCudC/001Z *<8&g.%g.#E*"4(#D-8L*MN)+6       !0Y*<8#4(%g.%g.#E*)+6#D-8L*MN        0X"$'(5#D,7K*LM       &{ { | Qr!