a Z^?Rc @sdZdZddlZddlZddlmZmZddlTddlm Z dd e e Dgd Z [ ze Wne yveZ Yn0d d Zd dZddZddZdZdZeedeeeZdZdZdZdZeddZeeeeeZdZeddeeZdeZeeeZed ed!Z ee eeZ!d"Z"d#Z#d$Z$d%Z%d&Z&ee&d'e&d(Z'ee&d)e&d*Z(ed+d,d-d.d/d0d1d2d3 Z)d4Z*ed5d6d7Z+ee)e*e+Z,ee!e,e(eZ-ee-Z.ee&d8ed9de&d:ed;dZ/edee'Z0eee0e!e,e/eZ1e2ej3e.e1e$e%f\Z4Z5Z6Z7edd?edGdPdQdQe=Z?dRdSZ@e@fdTdUZAdVdWZBGdXdYdYZCe3dZejDZEe3d[ejDZFd\d]ZGd^d_ZHd`daZIdbdcZJeKddkrddlLZLeMeLjNdkrxeAeOeLjNdjPn eAeLjQjPdS)ezKa-Ping Yee z@GvR, ESR, Tim Peters, Thomas Wouters, Fred Drake, Skip MontanaroN)BOM_UTF8lookup)*)tokencCsg|]}|ddkr|qS)r_).0xrr./usr/lib64/python3.9/lib2to3/pgen2/tokenize.py %r )tokenizegenerate_tokens untokenizecGsdd|dS)Nz(z|z))joinZchoicesrrr group0r rcG t|dS)Nrrrrrr any1r rcGr)Nz?rrrrr maybe2r rcstfddDS)Nc3s4|],}dD]}||kr||VqqdS))N)Zcasefold)rr ZyZlrr Z 4sz _combinations..)Zsetrrrr _combinations3s rz[ \f\t]*z #[^\r\n]*z\\\r?\nz\w+z0[bB]_?[01]+(?:_[01]+)*z(0[xX]_?[\da-fA-F]+(?:_[\da-fA-F]+)*[lL]?z0[oO]?_?[0-7]+(?:_[0-7]+)*[lL]?z[1-9]\d*(?:_\d+)*[lL]?z0[lL]?z[eE][-+]?\d+(?:_\d+)*z\d+(?:_\d+)*\.(?:\d+(?:_\d+)*)?z\.\d+(?:_\d+)*z \d+(?:_\d+)*z\d+(?:_\d+)*[jJ]z[jJ]z[^'\\]*(?:\\.[^'\\]*)*'z[^"\\]*(?:\\.[^"\\]*)*"z%[^'\\]*(?:(?:\\.|'(?!''))[^'\\]*)*'''z%[^"\\]*(?:(?:\\.|"(?!""))[^"\\]*)*"""z'(?:[uUrRbBfF]|[rR][fFbB]|[fFbBuU][rR])?'''"""z'[^\n'\\]*(?:\\.[^\n'\\]*)*'z"[^\n"\\]*(?:\\.[^\n"\\]*)*"z\*\*=?z>>=?z<<=?z<>z!=z//=?z->z[+\-*/%&@|^=<>]=?z~z[][(){}]z\r?\nz:=z[:;.,`@]z'[^\n'\\]*(?:\\.[^\n'\\]*)*'z"[^\n"\\]*(?:\\.[^\n"\\]*)*"ZrZRZfZFZbZB>ZuRZurZURZUZuZUr)rrrrcCi|]}|dtqSr) single3progrZprefixrrr yr r"cCrr) double3progr!rrr r"zr cCsi|] }|dqSNrr!rrr r"{r cCh|]}|dqSrrr!rrr r r'cCr&r#rr!rrr r'r cCr&)rrr!rrr r'r cCr&)rrr!rrr r'r ic@ eZdZdS) TokenErrorN__name__ __module__ __qualname__rrrr r)r r)c@r()StopTokenizingNr*rrrr r.r r.c Cs4|\}}|\}}td||||t|t|fdS)Nz%d,%d-%d,%d: %s %s)ZprintZtok_nameZrepr) ZtyperZxxx_todo_changemeZxxx_todo_changeme1lineZsrowZscolZerowZecolrrr printtokens r0cCs&zt||Wnty Yn0dSr%) tokenize_loopr.)readline tokeneaterrrr r s  r cCst|D] }||qdSr%)r)r2r3Z token_inforrr r1s r1c@s,eZdZddZddZddZddZd S) UntokenizercCsg|_d|_d|_dS)Nrr)tokensprev_rowprev_col)selfrrr __init__szUntokenizer.__init__cCs*|\}}||j}|r&|jd|dS)N )r7r5append)r8startZrowZcolZ col_offsetrrr add_whitespaces zUntokenizer.add_whitespacecCs|D]p}t|dkr$|||qv|\}}}}}|||j||\|_|_|ttfvr|jd7_d|_qd |jS)Nrrr) lencompatr=r5r;r6r7NEWLINENLr)r8iterableZtZtok_typerr<endr/rrr rs      zUntokenizer.untokenizec Csd}g}|jj}|\}}|ttfvr,|d7}|ttfvr|t kr| q@n*|ttfvrd}n|r|r||dd}||q@dS)NFr:Tr>) r5r;NAMENUMBERrArBASYNCAWAITINDENTDEDENTZpop) r8rrCZ startlineindentsZ toks_appendZtoknumZtokvaltokrrr r@s0     zUntokenizer.compatN)r+r,r-r9r=rr@rrrr r4sr4z&^[ \t\f]*#.*?coding[:=][ \t]*([-\w.]+)s^[ \t\f]*(?:[#\r\n]|$)cCsH|dddd}|dks*|dr.dS|dvs@|drDdS|S) Ni rz-utf-8zutf-8-)zlatin-1 iso-8859-1z iso-latin-1)zlatin-1-z iso-8859-1-z iso-latin-1-rO)ZlowerZreplace startswith)Zorig_encZencrrr _get_normal_namesrQcsdd}d}fdd}fdd}|}|trHd|dd}d }|sT|gfS||}|rj||gfSt|s~||gfS|}|s||gfS||}|r|||gfS|||gfS) NFrNcs&zWSty tYS0dSr%) StopIterationbytesr)r2rr read_or_stops z%detect_encoding..read_or_stopcsz|d}Wnty"YdS0t|}|s6dSt|d}z t|}Wntyntd|Yn0r|j dkrtd|d7}|S)NZasciirzunknown encoding: rNzencoding problem: utf-8z-sig) ZdecodeZUnicodeDecodeError cookie_rematchrQrrZ LookupErrorZ SyntaxErrorZname)r/Z line_stringrVencodingZcodec) bom_foundrr find_cookie s"     z$detect_encoding..find_cookieTz utf-8-sig)rPrblank_rerV)r2rWZdefaultrTrYZfirstZsecondr)rXr2r detect_encodings0         r\cCst}||Sr%)r4r)rCZutrrr r:srccsrd}}}d\}}d}dg}d}d} d} d} z |} WntyNd} Yn0|d}dt| } }|r0| sztd||| }|r|d} }t|| d||||f|| fVd\}}d}nd|r| dddkr| d dd krt|| ||t| f|fVd}d}q.n|| }|| }q.nB|dkrZ|sZ| sJqd}| |kr| | d krp|d}n8| | d kr|tdt}n| | d krd}nq| d} qN| |krq|r|Vd}| | dvr| | dkrR| | dd}| t|}t ||| f|| t|f| fVt | |d||f|t| f| fVq.t t f| | dk| | d|| f|t| f| fVq.||dkr| |t | d| |df|| f| fV||dkr2||vrt dd|| | f|dd}| r| |dkrd} d} d} td|| f|| f| fVq| rr| rr| |dkrrd} d} d} n| sntd|dfd}| |kr.t| | }|r|d\}}||f||f|}}} | ||| |}}|tjvs|dkr|dkrt|||| fVq|dvrHt}|dkrt }n | r$d} |r4|Vd}||||| fVq|dkrv|rb|Vd}t |||| fVq|tvrt|}|| | }|r|d} | || }|r|Vd}t|||| f| fVn||f}| |d}| }q.q|tvs"|ddtvs"|ddtvr|ddkrv||f}t|pZt|dpZt|d}| |dd}}| }q.n |r|Vd}t|||| fVq|rp|dvr| r|dkrtnt|||| fVqrt|||| f}|dkr|s|}qr|dkrX|rX|dtkrX|ddkrXd} |d} t|d|d|d|dfVd}|rh|Vd}|Vnz|dkr|r|Vd}t |||| f| fVd}nF|dvr|d}n|d vr|d}|r|Vd}t|||| fVn(t| | || f|| df| fV| d} qrq.|r*|Vd}|ddD]}td|df|dfdfVq6td|df|dfdfVdS)!Nr)rrFrrzEOF in multi-line stringiz\ iz\ r:z z z# z#z rEz3unindent does not match any outer indentation levelz zEOF in multi-line statementZ.Tr>rZZ )asyncZawaitr]Zdefiz\z([{z)]})rRr?r)rVrDZSTRINGZ ERRORTOKENtabsizeZrstripZCOMMENTrBr;rJZIndentationErrorrK pseudoprogZspanstringZdigitsrGrA triple_quotedendprogs single_quotedZ isidentifierrHrIrFZOPZ ENDMARKER)r2ZlnumZparenlevZ continuedZcontstrZneedcontZcontlinerLZstashedZ async_defZasync_def_indentZ async_def_nlr/ZposZmaxZstrstartZendprogZendmatchrDZcolumnZ comment_tokenZnl_posZ pseudomatchr<ZsposZeposrZinitialZnewlinerMZindentrrr rOs      *                              rZ__main__)RZ __author__Z __credits__r`ZreZcodecsrrZlib2to3.pgen2.tokenrrZdirZ__all__rSZ NameErrorZstrrrrrZ WhitespaceZCommentZIgnoreZNameZ BinnumberZ HexnumberZ OctnumberZ DecnumberZ IntnumberZExponentZ PointfloatZExpfloatZ FloatnumberZ ImagnumberZNumberZSingleZDoubleZSingle3ZDouble3Z _litprefixZTripleZStringZOperatorZBracketZSpecialZFunnyZ PlainTokenZTokenZContStrZ PseudoExtrasZ PseudoTokenZmapZcompileZ tokenprogr_r r$Z _strprefixesrbrarcr^Z Exceptionr)r.r0r r1r4ZASCIIrUr[rQr\rrr+Zsysr?ZargvZopenr2Zstdinrrrr Zs                     8 Ib &