jQddlmZmZddlmZmZddlmZmZddl m Z Gdde Z e jZ dZdZd Zd Zdd Zdd Zy ))tee zip_longest)escapeunescape)Paralleldelayed)tqdmc ZeZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd Zeeeeee e e e eeg Zy)CJKCharsz An object that enumerates the code points of the CJK characters as listed on http://en.wikipedia.org/wiki/Basic_Multilingual_Plane#Basic_Multilingual_Plane )ii)i.iϤ)i@i)ii)ii)i0iO)iei)ioio)ipi)ii/)ipi)iiN)__name__ __module__ __qualname____doc__ Hangul_Jamo CJK_RadicalsPhags_PaHangul_SyllablesCJK_Compatibility_IdeographsCJK_Compatibility_FormsKatakana_Hangul_Halfwidth#Ideographic_Symbols_And_PunctuationTangutKana_SupplementNushuSupplementary_Ideographic_PlanerangesY/home/spazio/kingdom_calls/work5/transenv/lib/python3.12/site-packages/sacremoses/util.pyr r s K*"LH&$2 -!/+'F'O E'# $! ' Frr cNt|}tD]\}}||ks ||kDcSy)u This checks for CJK character. >>> CJKChars().ranges [(4352, 4607), (11904, 42191), (43072, 43135), (44032, 55215), (63744, 64255), (65072, 65103), (65381, 65500), (94208, 101119), (110592, 110895), (110960, 111359), (131072, 196607)] >>> is_cjk('㏾') True >>> is_cjk('﹟') False :param character: The character that needs to be checked. :type character: char :return: bool F)ord_CJKChars_ranges) charactercharstartends ris_cjkr'as6 y>D& s #:%<   rc (t|ddddddS)a This function transforms the input text into an "escaped" version suitable for well-formed XML formatting. Note that the default xml.sax.saxutils.escape() function don't escape some characters that Moses does so we have to manually add them to the entities dictionary. >>> input_str = ''')| & < > ' " ] [''' >>> expected_output = ''')| & < > ' " ] [''' >>> escape(input_str) == expected_output True >>> xml_escape(input_str) ')| & < > ' " ] [' :param text: The text that needs to be escaped. :type text: str :rtype: str '"|[])'"|[]entities)rtexts r xml_escaper7ws(&      rc (t|ddddddS)ai This function transforms the "escaped" version suitable for well-formed XML formatting into humanly-readable string. Note that the default xml.sax.saxutils.unescape() function don't unescape some characters that Moses does so we have to manually add them to the entities dictionary. >>> from xml.sax.saxutils import unescape >>> s = ')| & < > ' " ] [' >>> expected = ''')| & < > ' " ] [''' >>> xml_unescape(s) == expected True :param text: The text that needs to be unescaped. :type text: str :rtype: str r.r/r0r1r2)r)r*r+r,r-r3)rr5s r xml_unescaper9s($      rcNt|\}}t|dt||S)zp From https://docs.python.org/3/library/itertools.html#recipes s -> (s0,s1), (s1,s2), (s2, s3), ... N)rnextzip)iterableabs rpairwiser@s& x=DAqDM q!9rNc6t|g|z}t|d|iS)ziCollect data into fixed-length chunks or blocks from https://stackoverflow.com/a/16789869/610569 fillvalue)iterr)r=nrBargss rgrouperrFs% N a D  2 22rcz|r t|n|}|dkr t|St|fd|DS)N)n_jobsc3@K|]}t|ywN)r).0linefuncs r z)parallelize_preprocess..s%OdmgdmD&9%Os)r mapr)rNiterator processes progress_bars` rparallelize_preprocessrTs<!-tH~8HA~4"" %89 %%Oh%O OOrrK)F) itertoolsrrxml.sax.saxutilsrrjoblibrrr objectr rr"r'r7r9r@rFrTrrrrYsQ'-$PvPf:$$,><3Pr