From fdb178303b92e2cfe411cd86585316c3ad7e8e10 Mon Sep 17 00:00:00 2001 From: Ezra-Yu <1105212286@qq.com> Date: Tue, 7 Dec 2021 11:27:34 +0800 Subject: [PATCH] [Docs] Add analysis&misc docs (#525) * refactor ananlysis_log and add docs * refactor ananlysis_log and fix lint * improve docs * improve docs * improve docs * fix bugs and refactor analysis_log * rename analysis folder to analysis_tools * fix failure link * add result analysis docs * add eval-metrics docs * add misc doc * fix lint * improve docs * improve misc docs * fix docs * Change the `eval_options` in `tools/analysis_tools/eval_metric.py` to `metric_options` * Improve tutorials Co-authored-by: mzr1996 --- .../image/tools/analysis/analyze_log.jpg | Bin 0 -> 68146 bytes docs/getting_started.md | 2 +- docs/index.rst | 2 + docs/tools/analysis.md | 211 ++++++++++++++++++ docs/tools/miscellaneous.md | 59 +++++ docs/tutorials/config.md | 2 +- .../image/tools/analysis/analyze_log.jpg | Bin 0 -> 68146 bytes docs_zh-CN/getting_started.md | 2 +- docs_zh-CN/index.rst | 2 + docs_zh-CN/tools/analysis.md | 211 ++++++++++++++++++ docs_zh-CN/tools/miscellaneous.md | 59 +++++ tools/analysis_tools/eval_metric.py | 6 +- 12 files changed, 550 insertions(+), 6 deletions(-) create mode 100644 docs/_static/image/tools/analysis/analyze_log.jpg create mode 100644 docs/tools/analysis.md create mode 100644 docs/tools/miscellaneous.md create mode 100644 docs_zh-CN/_static/image/tools/analysis/analyze_log.jpg create mode 100644 docs_zh-CN/tools/analysis.md create mode 100644 docs_zh-CN/tools/miscellaneous.md diff --git a/docs/_static/image/tools/analysis/analyze_log.jpg b/docs/_static/image/tools/analysis/analyze_log.jpg new file mode 100644 index 0000000000000000000000000000000000000000..8eb1a27d6464d255b84b23a7460a5f622f51712f GIT binary patch literal 68146 zcmeFa2|Sd2zdwG7C@r!@#8i|mv{4(ldbF*aO~>e*o-0EiKsTonVi_yFI|q0XpWLEJsgY zpl8)GXE^4-cE&68?JmBH*;VY?9XS58w;UhuW<1Ejd5G)y2?0SNVF^ho>2or&mo6(^ zQNF68s&hkEPv5}M$l|u;9V=@a+xt$=4<5R>x_LkG@%4M^9}pJ)A|f*CWpqq@LSj;K z%DdF}Ik|cH1%*Y$CDk>xb@dG&8=E@2x_f&2KKBn`$Hu=-OioSD%;J|Z#i}_9^{w69>;$h+TTX@#|HNJKQyxcH?V(D<&C=8QBxdSd#Tw#~cF z)0+CkI~RxMR-4ze38G^m1JdjEcyuLo3I781d6ps zGaI{nZiT!l)eyk1tnVi9;8WI>dNH?yAvL|c+#kwt>26ni(hX)D%C_V`9P@ZKb4u&Y z+epFNv~H27$M3aT(|#gJ67@WXF}z|VOFUxToS=hSx`BguQ;v)f?$lJ547GeR+6d6s zC?y_W>*h$lotI4>)A-tOQX9d%Z}F8c51e>|G7LK`uY?zZRWz1P)Ap`;a zTB3(l1LY8bhiGm)yp!OHSF2^6GdqK;3-Pr;YZsQ4+%VqglDs6V5HfJl-mc@i*3j^o zCpa2WzlBo#tvM{y0&IMfbULoa11F;2n}HO#yPnHdY^PcU>lH?T ze!sy=1t<9f%8n@rowwm#v4eEHOOeaQ?Li7rkpLf)_IJh{nBnqiu2WcB=qay@DAOFO zm&&P5mU^J5r93pbq*+jLk_uFvuHbV?9v|CyLSdrxMValziPsNmBdxH-^4z>ht~^~f zp*Ee(a1YU@JcpLO%)7P5Lt7!uVaKYXIz4xL){Wp&6cugIhf!Edlr22D7J?Dv&Dw^LFye2!tClTBH;Ky)P85(4qM zC<_#TJXeexsfMyh`|)>?Y>tn1BTdt)YJCv&oTuQ+P zm7?=@$7@?p6q%Zy$O*e@s8x7bS6>FIMUq4CLTkwTC_+@=4yyW^le@*((fC??f8df3 zk_sHX{m?S-(@@^o>^(u*@_h{`PRt!sxpf~Z@Hhy`+b)6fF_X*2QGv<~E){v$xTss} zjV^f=wp;ac${zw|wXdg~)bBXnRrJw6BS~NG>-fR|AQ&I;Kjvz|-zk2GU4RfT*4NAV z$IR~H3os0_H8RvbrMB}HARg$B=@R*Zpck zw%DCuB;IB!@Ua_xlK{4SNh|`K@Iopuo`<YhZ3B;12e%ZXC~ zK3zN3+F@wlDqP*Mt=dSn#`SJ^Z4Nw)p0SD3>fl!^hTilFH2y#;Lv0C+kK%=(0u8ok zUlMe@g|DVpRu}D`*TtTn=%9qT<~XQ({*WTVgb^quzs2)Wd`vm007DKHSeAqgKv@cy2VJ#bmyRGy*O?7eCa%^%!+ld+Pc5n*t~`{PIR zs@DD1JD3)j{2{Bt(-?w66cq@4w|x-{1>^QY|99TA{}zOmK(VtQzo4<(M$tJxPv>KU zakz~kmy#w^|H|w7{_&zT)c4b?1evNXftqNe1HjC@RC=ptJ6dAhYGs6e_W4()@=m_9=8e!vd`ZQG#?LYV2ez zOBsHDf^)jffViK$T*{|VwfD188xDp8#=H49kLktS9^$+A^w0-~$28MU`b~W(oEGe< zEG*b~{caEHPT+0Ah*-ImMGApyq+!pA2nVbCmdGnJ( zM96#{omSUeYAe0Jmb zjj%Y{_usFAOD^S7EH9|Y(oi$-a6UDel@`LvwxC}19V7=) zf$4>cLsS6!5OhB9v#Vq8M_ey#1;`gsfvFgZZWqJjuO|2i(rY_HU2Y_tr*-rA9;Xt} zN#4!x>Q64fSja;N%`jgS!E}ZSkld1SQ49BpLN4*d>$pN+j7^r`*0U53{s-$u&LKhR zma$`Zj_OGSTb^800R-v79L%0|G0 zJuLU2d{LxO1>_+R8|XmgEmfikLOzqPa4WUE+&vlcD!7emHx3_p(TtgmR69G5)Hyew z52Z2Ii|bjD?adPD&+TIb;|jSCA;&k~xaco{L#3k-REXD*nt@{@f%0}^WXO$>3wd3; zaKlMEOTw+(Hr6?8muQ>WTQa&sP9j+2Q#9!eZVjIqYDKVqH(WkWwYj&Qe7;t z75R9vhWg=sAG-2i8H>y*9HauOc)GqweCztc{gdlm7bJaJ*H)7D2Kcyoq)nUcz+)DK z@RX6Ul=)~p?U;Sk=m@Wih<0auF@B>a<4~kX`3cI-B11d7rKH#WLu_g*c0#&ZN9FO9Y6N;0t#O6v;mwY;0X&o4xG zf%NpP@44{+iOT_k2};15(&(3uT-!)2+5%um^!Pqf__1;0;vtDcx+x_u8&4*hUYC~R zH4M?#;17cjk_5hq;{hu0Sr@e$EBuihTR~6IScenN#*y~n#1h^O%_#EZ!`(%KjjX>G zJKOMHG)=gy?KlJ>h%$5c_V6Uab2K>fF;|I#=cTVz$RVuJ4#Fa(% zx{F%18?ygnio(EC6ny4}FJ@kX3T!FgH;&lm^3fFkL4C7BBg(UJIC2?RNShHm&nJ|G>rRh9bQP)~8e!K)0k83q_bfKQt`#Jvh4u8>qmQq239A zO-tz~m#%$&IoHQf_q2g(WtbFJ9}=DV0=v^y&TaaJg;1#ytUX`pzS~H>#uGDcFd*|D z3-L?$pDvXBw5#Fsj(&j|Lo}|P3hc2OSMHWHy@4(z2V&ER?ITxw55gMGWim$B>!>2b zYcD8%!6O#0YvANkCuA}VRxUmnSZe04SdYiB7RDE$38q&NF*R6$;7qnv#Qf3ysNHXN zkM^UkM1hEep$p*?N-fN0?~qz!g<5< z#e2`}3*>Nr7~4#iQ63dgIa`np{kK?w0S5PiLhSuSOz4+gA9Ex_f5~Olno_ zQ}jecYE@%*N{a1s#Mj7-P2NByHvL4`Do}p@OJw5*`El7;7#~g3;GukS8;)|u&D9Q{ zfU94+)-Adq&x$PA8hQ~q;8QmD&Sw52Hgn;DNAkG>v}7Ff&84b!EGXZtuRb4EzCI0%`+a=diS!eOn1-%8!N(i2og^Z%YfL<9@ zrIQq$MKRDVr&JEJ;3Lskji)enq|IoHMoku8d@M6<%;W6E-i08J-bg0){5RP}PgeHk zDBTI?d^M?YP6ViWv=JakizY-Pct(6#diCz*%nrDnes(d4ru50ZrVBcY`<1Nb6j?e~ zLzt~u2Zn6L+EQgh@27pacKWNT3yTbZ@4jz@26fI?S_ZOj-V*twZ^%K^$NM!S>+A@! zK;$FIiYkVUNh*L@qDXi=W?E{Y0;mxZoTIKJs!mOWFpd-OcG`EPyLxY~?J$3AlTx_A z#BQF~RZ*OBg|uN;&NVhshRE@l=|}>Iz3gemP;~py`e%}^BzUK24Zcs2s!+FX-7<7m z?e6YTwI%ahOn;ap$67ZM92QbxwL4EiswYst7eT4+0*-9KV9alMqNbdw6yW#FtR z98oAzS_tPif_V+}V71$YEtEv$yXW zIc%y+9`=6POc);pLj8B%NRNm3E|u*YL;32XEIabvs7UzxkgH9`rDC81Ms)B%^)#Nr zNc{_w=KwM-L2f+c08$fIHhdIsRVB_z*fA+GRjAHUFJ0n3r0T`|WY%eLuBa@>;N2&$ z6>dvlz;HnNNnSau)`+yBZxnFG_G+B5?Z$y1S7zSJ_B8Tq@_iv7vUm<@l}V^MPY5W1 z)MmKXkL+`z0uafq+Af6si~Cb-8v{Y=IfPNODoebXQL;W#kER3bX% zB>^u+INw~J(Oaw*eXzjB$`#9gn@0Hqz&QW)w?MXS3$lG(;f?`7&5vuFux(xcmJ#@? zsXsPX!pVhM(`H;BJoIbbdXmv_$VhQ{u|WbX?KE(s^d)NI{Xqqq zn1(d8`AI86yR+mK#4<9KPK4hHRmW0+{ll=!6Ss@`{muOo8a|yG{NP$O+S^7FZDScB zRqznl$L8YQRGcCoID}-O?`3TsRqUBW*-dDSzE7gCg9*`@17Rdjt0Zz@jQF$g5W;J7 zDv;DFj@#t^0~YRRAJkgGIFQM2_)sV43O1W+)Sfl@2wiv~HH+Ci=ruYLJb-S?K-%VI zw~=yj8$RP92omG0S`hi3;tQ0mDypicCyHDWSFsfqLJ5QPhx*|i*v6>+bsibSF zAuojLJkQ=oJ!Z5=sF!P*By{qq&{#(8IR2J3^Vz>eoko)NPzw8*c_+-}s}fal!v8{Z}5DG1H8 zt}$}4wb`(BBpp|TRmB=pfxX1>(hvfP98c1KWUae0Q_SEhdh;kN&SO=*o>It5-5Pk; zVL@xq5vsf$$UeQQMGlU_1gpZvQne`i<6)051*hjm9YP4^>Yl={Q2tXu>wP%J+SNV8j+~SAsrIE0G zDIV$4C{21EDNk$P#E!x*LO6@S^AP~U%`}Ql{`}gnpmh9VRk;keYeYv1_*q82RDsMszkgsHpsxEKG zOl{l%efYWtD!`!zs=f_y%FsjlU?!deLNW#Hr7x|%b6O(c{3ylZ%jvF)5NmVJz#;t< zGQAj-Q1S^U?Jd!wUuAS9#r>Uauw?x!x||O^DTnR^vD|~uhM(sNX7~G{QP3B4f^L#~ zP&fw%FfMGTSFR87s$eC|*kIG32WMr2O9;mGZV3Opl@xWZLt^umE4Ngn6DE&+ImK=8 zET|lwn+_%Dmq19ASq$ME2-%KJgBI5PE$BThf?dR7vV+VCY$>|Av z-4_mibFtjtqS_Wg*1+HH$y*KPR&T%Si%v3n8o$>dxHi(F@z!-U)>+rMJvV*|GyMWR znS-G3vbH3QxZ^X3BM+8*NmqbURIh-KzG)00b3ZhG@mnz7ckk;#0#RQTuvx@?P)g{R z-U)L;A?4CAC@vqmK-0j}@=pQ5?^+5&r6D%dZ;?R%GVCGfM1JNSqykE!c0YsY{%ux) zJ$y`@*9B6|e28Q?t{|eObhsknUEMB7;D_|t-h_+$oo~lS>2<=~Z9GrBNY%}fWOQ&u z#H(Y9Pk@%xC}50fVK;7)xZ%kZu7l@KSrqV-@LvqYEaYT`B@Df0W)azme;KcZ_nk;t*Y{cd^qD9-FTldk%)B~8BR zdUprH0!fh46I8*-4w41#4Unt^-=Zv=pggu(RG`S8!WVsQ@l@^bvz(|Q=>C5iUNcJq5 zR%V!!N21@Fd_`>jX`#)Ftl`9okHI6XBo%`5Xej!S%Z5@XDBu@FwFRN8QtVN7#Pdk{ zvO{@v7jc^X%PEqweG!9bwB()k(1;~UL(O(WG(Q)NRk?SA5nJ{rH7 zY}_1MaaVZclh)m}IXAVY9@sdT6rMSX47IV%YF+Ka+Ms5Xxjdy({6j)<-c^MZK>?rc zc5?VN7%E@&v@o$UhYY$K{g}M#Szu&tCW4E6>bG<}GSmTeWJ`s=J!{MYs=81K+}umk z&M0X3;6y>u>&t@+4D;gzGel1-&90MK6ZxqU?f=e@+R@)^0 z2$4*LX$(So1fo$Ndf$M)+jDReZ?iaGeq{#J@c%r37zigDqe|c$*ep_bIRm9~ePm0> zn$|u*FmA_x;@bNi^b3Og-x~$TM~s5;k61L;>?kwous_vm)quL|YoaY?eP=cm=qPk9 zHuxUW|6?@abm&2lo#tKxTjxTN`HC}|eu$s{KCeHiz^Lt|oIpzBx|5lhN#{ExHaV7# zFJ|n$?76o(C&SE__i)dh_|nG2iuF#Cx{YSm(QIrE&!ES!Cn1(&>CTmb&uB-KrHcA|P^=srxKe|Y*~ozHLJAPvN8yD%@=)wM-?zeJ9!3iTlgs9m zq1vP*)Z4q7bCYX%DOV&&Q(b7GyYGFgRJ7{>iktn)Ge^#8iuIDb7OVt!6 zf>pwxi^au)9OE~#yEB8Bfv%>GR2zQ#%-Xd({GD)l^x|L&WGV9wMKdCT^yQ1b)rYRe zqf1HJurJ6|

+QtDunL^!o}~Li3Tm=p7HN@`eQq={z2!F*QqIaM4Hvq8N9*64j@{ zafG7DweT*5j1HI!Ox*zW*u-rzYERc5jU~{NXx@Kh7Ls>8=YbIcQ7y~p#A-VGkSm;w zOne5q@S#v)DAs^`5mRgx*TZy0{35m4#x4$0k4+L0*plll(8 zqUTXiG14;;_t}hW8Sb;!1`uWY^?84P7?7v^_vfwfQGwXN&GsD~|9>@e3sjEZyXD-w zSt$pvynh(7No%k{1(FMOOMe|w+|Ky$Uq+?;%a9%&4AA&xRJ^}Ag*4WQzmBToPj9BU zjfHfRpgua7U@OofJ9A+{&~4gA&v z2v=SFOfb=<%68bf851)5QGUvf>@szH>D7wfGN@FaJ#hVyXf!lySR$uTl_6eau+2RZPS=(LL40Cej{yqgbuO=105^F%+1AD`Y8%jZpG zw;1-2Zah|x(YvWI6AugVJgIoAG0x>;-}{DR1*tAIofP z-Ql3uc?-i@lpezALaeF)vehru^ByhCR)JZTIZ9a;p%#_6CMFQ;nyd>Z8ziA5KgtEr zOv%F$;3}|UrJze&3N|iKEX<3a*!PIl;ua zDRz&~TEWt!&X_Q>(8uZ+efFWp2`{uaQjByc^Ebyv(vt_&I#kxL+#`&qG)WsFCGmN2 zt5hIBZ%iHxJt~Vx-=bLbI9X=`hpDu4^_p?W7%?(b+T4Bn1!@G-T8j8ih*3J9k`nP6 zl%W)LX!#1|!rG@2*#<_;!fZHv5nT1UeeUalus0PPE|OkTH}Q4tmg^u~VnhFY7Mf@F z4q1d08eQOYTAAw2%%?@eMD0d7sel(ncW~Pt?;uBOP2+X#D1upa+8T?uD4ogq2DTuA zuopc#o4=6|z`KVqpvd{Ht8#X;#?NIMMeIiC@GyDQUe;mG22qSEcxyCQ;^vMs7 zL!=$Yyo55a4{ZLbxcoHVd@K9|?|2WpPt!0b{ z3`|WdHrk##mzJ&}2z1LiH1ljswqM#QTu2$*r;AMc2(*{GIy`3$Z8U2cM&FCTztaH) z2@#eqt}9({cZ!j+F%-J4D9SY}V~J054UtK%`GYZf#BePT%(=jK^*O&ayYl%2K)Z9> zRP?!4{t=}}^HUpF8>T!_xWus-yB3V=y$spEdc*|vk#`V7K4S!Y`4`hyzo$KGe~1I5 zSOiEXgKsQBmg%#XsDQeC+Qa`Dc13hsOgH$eMD}vj3LTg}W!^yYby}nXtDE}(#P8co zmY-uacUAa3jXm+6GfM_k;7@XL*vDaosA_~iinD@FIgWI))6O|?LArX}x?6LN~P|8Y8+?{t=B{3FGk;1b?}4uo7JXC{NeMIb?LUEfZi zx%X%u;xW!8N;OcsEZalOU9Cl@<=Op|q`6Obl_Tv>;SBb5rbTG2X3Z<(qQ(Sjpner6 z@X~cHEF)L(;yt?q#>$cP8-dF2kai`afzENYM z>=3TlmkVN}Dz~Eax;hOKHFdj(FYY?BM)x2Ug*76{fmKbcpnFnv7e-PSsD*9!?6&2Z$l{VMoZjVxW{gw?K#zZXfUmdvzMReS8pHh9?gdX!lP5>tPDS zDJKf1+8O7-%90qJbc|l2IZ^%ldzyW9LdCeiS+s zWL8gOv_-InH~bF1_AcK*dZ&r+5ewjXTKI_dY6kj;`BMKdK6d532ASq>3`(uu3OVFK z+}g^FQCl5~>B7T9HT`dl_|A|m_8PBdBCns_?Hv$#Eb%7VQ1vviI0n-+bp^6Ju&%M* zM@Bs}G$;v7RizLP=KLBPEk)Og7?iTVrxsd;vXCi8X-KEUwUT*NYy4J*kAUk)_uY)#% z8JPp6w!+J9S6FFJUOTR>t@-5Pi!69~e3*Ixk$*lsJjZAayCAX;uIB#7)zM3AGJm#&vf@{&k>t3kJ>HtJ6-ZQnWXB$?<) z0-ff>D`{95)1We$7k*_%O)%78;-%3|&VN zxRce_(_l}`4xim9dZp}Zs6pD-olTlv?{CrY4dSB+2V+VO6jIbzm@LA(2x-@0*vD#1 zua^kO;5{&aB#wDvVnC3)6mz!oYWh)2=u@^0f&*?$LR`Qr&r(xHH03e?az)@LmeySe z><>?uKz2*)I)Ny6uuN7)fPtY`Ejz~OmS-!Jrf|1d1dqX|nZ=l-%JR|GvK;3dhWDP1m%DwlqRxLq!5Ye_cywC8qreNd zQ5*UU8CXt}SL_*U%k2Z*nAcC$(jf9NL9pGa9243ITzCvh0y5AuK~i=4S~gSVd0G?x zwM_)EhcIXc?cdUvCt`VM8dz=Jy0cB|nbI`}-{&QeH~gk;)nNS^m2NN6-0}O@$4y?z zNa3%3!L}PS^p?xa!EA%V zAm%C_?*C~9w$qboT62#bHx}WSq2+dxOJ0QVT3lcl1iOATOXb2htlN{{stTVr<_}Zr z;Ex)0U9;`U(rQh7ow#9FFw*iZvASKYgFGY(gupUlgd?4DIjedZt>+Z2U44olmSZBt zbU5h!Yd5bM)1>VwChI%m+a(`lrhI3}dWYC({I?skZ$Zd+=2fy#3ApuxiT%Nl*>#-f z`~w@`*h!OK;Qv6!%o}li*w4XFaVJ7RyYWd1wCp*a5RxM9_=cB+adR8JjdqJzg#B{! z2{iZF>|hoC&@h!H)1aS1VGREV=IdWp!WO z@?q5J$aCd;>JPT%&8%sDh&=u?p80+aU(Ej6yUJb?o_-YbZ!yaE=(e%;&qGDPuH&~0 zg}&nj{3fQqEgkrQ0y+=>237uBEzzG)#L>_HjO~otvG>o?8vwm4sd#SeqHpkt{PA+G z7EQ~6xf=ne5=QRt)zgKyzvb?U30Sf5wBzc0YgU?nP6o<9@-;eF>E06Y67o#xPNabQ zJD0Falni;vgfB2Yw1%w2Y4M&n`ojsvkYLyW1I5&L0*@sMWDZ8|V;@cxTy-^&I7pcd z6j}{ccWslY*4?-ll_A!i3zM0l!z@e^Pq^(7tn1py)=Pja4MAT>zCx|fu2?ULaMz;_ zAR)M|Cb>5DIjuo%zkZv$PdiOap&LhiXk3lSFSOxwq~mzxB{J>0A3o8{YpgQ~vdL15 z?#&||!lhK3F_wp$6b5))GAvRrbX(7m9uPkP8CqfbqH(odcz=_iOVDs(@%6`GI342{ z6t)^98NO^UQaa7(tE)VcYS;~H=4=w(YI0L@O%mvLK06PV)vf3-yf2SslFpBs=HUSxUI`3?t_LStb0OWX`y?3~8rAL!80g&jOro)| z`WJp*%; z_Z9JK?PD8xi_z%liaVf(^2BlR^dV?9KhXj~o4gPhTf`$M3t6DaK#S<$m;9WrD$`95 zoQyhKSQIvBnZ3H$X#21WSy_X1InjcRY>l==q=!MbB zi>8Cj2kk681&iJQaRFL0w2;0^YdQxS~RuK=7fNUs`0}(@fJU z%7|ECpJ8uiUk>>sa<v-{I%R@>2`Z4A2qSMvBLXuLmG=J;(oIe#Z1`fI5p|IeJgKWZ5)UE1nB zA-Mfs;!jB`4?3#2g_RF$MMb(AV2=eeoVohxyzb#JxNN?SW|D9Oe*$M(!K#Y*+}2!@ zur9nOSk}Q_z5jSAs&8%Zy3N}{|4gyt2IER67Z)DIIrGOpE(*Rhp5^B(gjOQxS_uT^ zt4~U*^M_mRd3bjCX3YVfN>KsJEAqN_eAwW=+EO&(T7}GX`xlNQGiIzs=90~d????ekH+l;S3Z%fqmfhjgSKe9#@H4(+I1h|6E|1H1L3nK+B&g31*e957 znkMX7d{O*0uS{n{KSj{;b%+`|6@-%?NYH z(E*5(s$>moZndtJ0q%z%AI#qgzaKkDGa$Q99Q&|Ffw@ zA`b$FeRU$+kxeg5N}5zu?KbX*IU***`UTVY82d9VOdU&dv}QO9Igj^W5mgcoN2DP% z0h&i3WoT7H{@)UZ{srhT-!X3fYiLM&xA7k%eyCRd8{z01>FA$c-)~IGVuSDZ$Rp{$ zPp!&NxwQpgvh=4Xp$z^?!uknW=)2kZ$>+`Z3h#76LbZ{fUzx(VhjPuszQ`8#%<{x4 zJJFWF*s%I2&6Yi<=J|vBdei}1Em&+9(vqBul*U8rxS(FpyICv*bd5BI#VOdnv2=R8g;_wZ2AIhJ0O4L4fhq_~&H^Cjl|W+W3kWcM^e8r*D8v~k#eU$@XB>;K{oxBKz6U5Mhz8s zsz{fOI}vE*I<6or6yKQv0{Y#2dRLg=9eM1yOW_@D&L4Z|bh-w?5cUoAcK?QYTY{)J zyE5ry7KnN)OoOQR84&djhy!U{VBTLJgyVTvQKUoPI-!3W&pN*3lc$BiqG<^sSGwgd zq)yyTh|<)^9XLYllitfc;g2faNW3e&cH9+j@cx{vScZP`fbF0gf8?V2blA|kN`8k= zb#J}ijnoX}s*0#F;&xb1Vc{zqg)DB8<_BCw0>sBjLD&OPtAE-xvV~cnt$To!&S-N^5TZ#RA!@=Dr6+kZngHLT@Oyb~Ey6 z>!pztN(hFdJZC_p1$%oY#y15eJs|n!)|{4KYsKpzj=NIq=V^2qngVt_i@<gakys%E|5)&crZ&rDS^dY!)!N>HcOZfBTLvZq%)t1SgBcilEfoF+>OU%Kf2eCe zTlF71MkDcknWR341)jFjf1IqkMzTuUb-VL!r;Iwu-7r!utn%>a^Fxku_PqJI|8vIo zFFWynv}4R8|}#i#YDGu z1{)<1uh^#8eo$&)9BqK!uGm6;&@Uj#Ob^Ulf3IV}Qysq3FPij2>>x_|{XTrRz4JG~ z6Mgf zc6A@PAVbI7P1CRS*E;>GL;o8DXot->R)7oa3j?Tjsd15_XL1mMfSA;gs|o~%ty|GTB3H#_|-!5tR811 zGwkRoQ@h)Bdexw(IeQ)Geete0LqsPqd|92M60Mtc@rmpEMkUugj8uqHeA#u>U?J8l zO{joifrYTU^+_bJEh-n(6-2>3T zRfX0f4K7L-CU>c3L4yb$bipBUxkqL&zJDk%ZKGks?&io@&t32!SU(k5zuXNVcnHyD z<{+MDgoKZmX6VT@9 z?MF&GM#XEv0>eqPuTIBy1SuGY6T<>X*Jy~>hiAthdZF&L`EQLCZlyB$nRiAsSbXD4 zEEpY{M`K{$^b8)qUlJG;$atUz70xM0t|lEqQlHEgIS|E*e_p@ zum|ruMHbg2=OIx%9kUWc95s03tegh1Xb}tYl*N)M=J)is*4k{^XpqIsF3S6Bif@%j zjxWls#5?K^Um&pK<6&P+C+q^fFaKpi@&w|K&$Y8v2=t>lWf$PNz-dEuUu07Tyvidg7FcWXqR3=N~JvA_i0!6a$GTNe2jNqahrm z^Mncf;!;gbJ&f5IE@Z!o+5M%>A;m^YN-xk34nMrXtaQ8vl^HQ%J-#zV_y8Rp@i-fe z4}LFg-1%Kp;AQJCl|oZd5Yh^)DO+(H^uEK@0VHfolHM(kg9K{$rgq2qy^y8 z*Fbg!SG^y0_=zghChgz@kkPDW?D$(rf*;%2J<&h~VoQ4ptNs4lh))jw=nQytD=|z$?E!Ki!{rzTWyv?~#)gODVRDq{M!g)tlduI!g zMZX*l=%2FfD7Htezf7p|FMWv&05pE-4(~60F%qQtdv|^vRH}6x-LLDb_3NS}#s60~ zWA2h8K^eyNeT05JL&nUsP8Oyy)nS%-aeZW8u>S4pqBSkzf5brkF|A?BkjRlzSc$uy zbd5nfNv_@h0=Qo8_9Lr}aQmOp^!8=_E3V8x2K1wx?B(9o^{t3O3f%GrAU*qA5yP({ zlP)5Cwc^-p=T)q(lN!sX27pOwDP}?vJwRNCcjT+&NfL^aXZ3rRa>kl#=1!zYXprmG zBqAa({d|f&8N%#y1*ug{L*b~XISa1OL5e&AdbZMwoy5$wO_SAM_tUQ(Tkaz2P7kCs zk7_)Iv69YDE$%AY?VOxO=I_~=?y@lya0NfLU@_MA#YKPggb&twBx55Osh8V)fC^-p zDjYXtDjd*6B;h? zE(Lj!;Ga0t*GDTAqjR0Vyco1edL4@~Fv1S5NsakUJa;`tx*jZ$sx^8cPAR;MbF!#e z-#06LzRgYfE_XWfKnZxR17v3fEAnVz~opq^^(kn~1!6z}a8N>?uH zoN{iJHeqs-sAPy5)g~X~?c&iNef)Sy07S(eHLQg;ph0Bp0LgqA$wiJXQ-4=a1!mV! zoNTzYsBDpYT&r$)TNe^YNj3BAt~j?@hQK#m#RaE1cadn~`1^$AyxrgV>ypnDWQWeh z#%3r%jGsV#uAg4=R3?0~dJ&WQ>Px@Nkv!M*DP>yIIa)ETQL=GtPYKwlu*{PGm za($W3S5?;Yqg!%3SdOy~yXas?Ks;@Su}5GfWoJj8wL^%AYS*zs6SrHEI|>BTXUFim z3C%JErKW?69iH@gA6^=c^e}b^=+!qR92S*r z)rHGx`@D`B8d2h;0&ePtPAaMzSNo@tqV4%}3u-!q)NuvL^n6iQ^JiV^FD6zO6pz$% z9$sD9BOliIj8>)a&7;`X01*@3gPiiRNe71)3d<6QCIlu2D`G@hI`|Ho3A;2cCaByE zGZsv;u5;zj$js7Py9*-ovoeo=qh$F{%MWZjMeEBpv?y+0)9y%$YA|CYIAAx+^zpQH1NQ#O#-N7mmV|c~`Y6ymJaT)TNBy1<@iu91 zG+hB=ocra=J2LYW&bbxkBl92e{p`*PmxPWyyO6=Olky3|ND>*_g>=X?8Ybk{wjZt= zLvarCTwG@TQdQ>q;1LygFmw!NyDIyzQm3;M{bp0Y(MV&dFfQzqFx;i4>C4jO68i?M zd%1?FV>sv6NU+3-jA{5M$?uRqDi(S`ad~#K<0?;65W915T{FnCW~S2A^ZCjuN`$(;7NI$Du-7Z+p?`= z_WNPqi19GhxJf3XE$vHh=4KZf#0R+v3j$1W=?mW<-VqUDa{X9|YT_%k6s75iuuVSO zb0$(eh1YjWm(G%nOHoo_UFl8xe_mXw1hD^(Hu=Zl9$`Os;D7hr(zXYkNA~Pr9b!C$&_0E@up>|7{AcU*rx6VPyU%fPO->`;qI{C&&2mn#9pRF&=Fcv&7F*=iBT6`CjRbw!xm+lO zjq<$l^)Pe5v+?x~a>Qlk3O{^b4dqDeJwGU{dTvGN0)e2J4^QFklb@` zfav?9W>UsRolYIn7L1zhZ9!kqnAHiv z3vg~VN^dAf2hO!e;jlt(N=X~d5xl7e!zT&BxQ$9pa!P}lTt}V(9(`7?=U9P>+vFAY zYNmd^$}KEWxef+#8LZDmS2!=RYwRmt1GAYqQtOz_Jp%EuaIKJLVekHJv z$=j3v(%4-3f^T!x!JN4>pEfI%h{rz0O$B8$%7y_a-UefAQGv_5$HGRi*PnP=_rw?W zNNzURRTcL46Ens9*DjfT2u-}o=fC59knp?89g*nXQ>z$n4YOrim}5@JaH4qHEn6zU zmHa0Dwdck|@#5NL@ZZLn>T6Eb!SXihL~o$WgWdWx7{EfN^#TG(5QLp%3F3~>G53e# zb|&a@axPxZv;?#()_J6h)*7f)YoX zQbeTJpooBo2uK&AA~ix(dX0+o5(NbXBmx2=B`Cc{x`1>c5=sbIXn}+pNQnQZ&YU|i zo*B=%p8wu^jTZUP=v<%eXyEd9(p0L!(S)NgPAl{rx3TyYEG|T^#7My`-iBTkELM z`yN6#@)C;zwXq0+}wpfKMP*9xR=`7)5iuA^YyzEQ`yiT?OpCEO{^$7Fx^a zHvuX;Wl|%bBoyf&9-D0?E;7hnpKMtc&97f8cKOKCrU12lX`ju(wzB}2b`&!V@ZIZg zr_1OM&~!+HX%%rEvhg*VY+QHyYq2*(Zbs`(XUayG^XQiI^Aa#LU@Kv*fReY-?XAm^ z$GUloW9+$K*LEa4%ut_vhtAm7>VFe;)Gs z`7XS4&MN_)sdZ7vdfjB`WvLiFl(LBI&A?C&Y4{>_6kya-X)=X!uPP5wQQ;A&*8?ee z0KB?3e18lb`s^HYvnb2D03s=--Yhf<3`#jOgdjfyl)c~67I3uvKiSgyd^@Mj=p#;L zzf3D$aSQ8lS^LNKCN?+C4r@0rdog_A7Pv4MF+2+7j@Ax2xr2b=+nT7Y>$UG4yZ_-0 z<~rRNzeV|Z)7~A4 zm*+QiFv2u00?Cb9z;L_P9Qek4?h`2TGtpx0ELP*aXAGu^sW8%gWzaNXIE2yzCgsh8 zmq~zY_oL6QyXOGj<-d1JF7d(jdRrZ2TYa7(9Qu!AykU_{L=HdgaL#5JzQ}H>mSCR* zm{nMJUD+)eCl}Q@N?*}8H%v9pDmZ{ofzO_9PIHttn$XtlNn0-PcHBC%DSQNOan2U>TQ{`Xtd{ z?s|^}O}x0%%@RlN!+_Ah{qeS@1y)(F#DBbke|XT)C`#|RKw)vTGNQng#T!bo$MZO1 zBmxkZXF?T2Anrb|qz7dsr)y>x4}n%r>J_?Yk{N0{A4jPunHlK^>JwUCf`Ix0=P>I= zZeRxAmcit#2%Jm$1G0pQz zp%d~+U&_s7%h=7AONB+>zhilD;ch^iR6ry!X8wbY1m8DGe&~T`Eye+OL9E@rjOC(y zT~Y?e^6EKbv$ytS?|mpOTEJ~l@?f^}X5vmpgR5J#H{MV(|6vEP(r?n29}3rrZ)&KW z;%Qg^1UmZV@SXAF^4}&Xe}F@>7)RCd6^xoFhwI#hyg=@dapre3KzG_PCKqfT^Xq`m_*?i|wGnbuV@KfcNxMHXCXN zoVnHZ;kAR^DY=Ll=W#C=(dXrC4@*Eb*D6M}9%~>tBu=b649$$=_7skK}UMWx4_+%>M#E`&#enA5&~x2ZIYmA4v?O zL*M856-$kEoA;r#KR|^bG{4@ml@V3>PoRvq6j)6qqL5~n!7s5CJEZPgJ6_b2PmG_F z$~>i@;l`B}9=7o4F=kIvtS)zp%;R?j1_u6jD)0mbbXP&Y6R%EDmsrdd*`o0sqGQjJ z)o6jHfFvN5yGBIPSVz6>UYde}_F!I0*rAZ$nC##B`adKZc&94l>qXHgMZx@(8io#s zshpW-&#*T>(&Uk8^$CRCuEqHwZsMuhw1%akk*HpDqJ>{Q3)yZ4YNWb|D$t zX=RM!#@Hh{_Q$=1zDTQ2JnFK!J&I=GlpbXD`ci>K(^(JPSkBu(<;^_8L=+nEGMK58 z2e8@}M0=MtEnmj_CD+8(L76TLM$YVy}UdefHQe&~XNc+Fxi# zd*7Q-GSQUO#=FUz#guJEwmE0JO!PXuz^PA@WtFUjkQZ+w>wzOJAp%9e-yG0CDoE-q zrlbL8VNd3cI+-k%Q3Q9VsRj(UTI-XO(wsjiO!;n4J?}_A9q>#?=7sbTxu>xw>W^b~ zLmZH-wxiXk7f+03KdzdQ+oPQ3mo4xu&x5G;i+&W z<|Q}>l+%F^>@ZS(*#g#M1mYP#`9U#zNrLl|Dir^C_yl zt67!@Kh7dE;i|vT*EoEee;oMxid`7rT$G4%iOWX4=QDimI`bAVmNVX9QmJq!j8M{v)oBO{M>eYppUYnQc$Fc&LP+kAx4f5{CUPx`y z&?9Ts`B*b&TflSfz)(dt79Yt3X*UMDT08~rmAAbgLQ6$@5V48$Wx>sW)n0yaN|DVnD3$qOjv%s-B^o97xeyqq@YyiM9g&fc^Sv^j(rFmzGLb{d z77k#v+slN=a|cC?63k5d-sLxnw-`yW37lOy?J;ddxT>Vd%qHAg?n?bg3&=-MX5j zya7N2htkr|SYsuEo{70WlEKcXEu|}Lic_fwx!a6dNZ)@<(vsd(CY$DrJA)?Ij@W#- zFS~n5tCSoO$vNOR;w8IGU^%{9L*O#|tz+|u9iGryeXDVR>Fv;6G?O`Bi`5KY%cl)^#Ay=x{$q0=Le9+!aAYUFu!kl? z5hi63HkmLGu(!nK$7d|ulb?dRhOCO0GD`0hbQf1dHI&7Rh*(R?TzvK3e$iNGS{wTD z`uK3(n~b(pFAt}t)mbO>eSiayYyI?F$C`hUXz#zdZ2Zn>eLdmhkIa?d=mkHHKJXpI zK-OMf{-BC1{<5h4gTl@dWB`gR*`w?*lG?tAjBq{>iFsQX#H-A5VR06m39R_+R0S5f z;i}+g#X~)p5)ALGRKDpt|D1L@y&_w0M#@(i5-TH2Ik_Du{A-sPygFnQf|@&(h}Z*I zq>mpYK1~T*&=dRC82J6O?>*+uDq+IFT=XNJZ9q|r)r3OG&bRie`{H#hE$Wj5I_=(w zFYG&J0qZl@WNt0B3Ev@DGIUNmq{l2)H=(n(t#YbM$9-O>ShK20pr4ouW1@tdec@2F zsAEhWcYD!O)MdG#z4D&(J?6@4;rz@10BEX^icQ@O~hg`ZHIq3?j9ga1k-W&&M8V0m8 zJk5D6cfO6Hc=E#vX@@;m<}4w#pGzO8vMOUCEUMt~6}R_Tn*T(Se(d7ZN~_B+4(Y1X zQ}&CVB8a^M6)D*m(jou5EDljwB*U|FG39{KiS^2>4-i7C@5K>R3;{gVGwNi+yK0q& z#x*`R8-0G!x5+Jl-db*mA;A4RaUQfF%w>!IY`+ zpJsWL$}J8012lb|9<0Y~1RFK`6G*BVI6LRk!S;+Tl+t0KnvCbRMtoeo|GqnB8M%{_ zhB6Omw%LB_4Llh{4?l_9{_LL77J5M)T9?RNkp)@NXmlQ7O1ke3nT6Eq8Qd#>hB&K9 zy!SRGY-`=whNodio~45PIrZxZnLhL5IKRWwaq+rVtGPIfi7p}2^SaRMIXbmDsm=Hp z2V8qjG~9{Zj$s7S2VL|f%c3&O&eO!Y#h!E$*{~}A6g`zY23woxc{f=4h3r-M6CVc{#+)o&)JQEf&`rN^+ZIap> zN57W_zaJjUz<9{=cssQiD5Eusc+@W5jj(OOuzH4d*P*~wLROxnaf9qKp%}09T8DJp z>x={Mi7Cs@!i>{Rhc5DMh}3`4_~vQKi=2=w%UqA5mJ@L)#yah~-UGNOzNmew8dWpK z_$&{2i>0FJA=z9a^`u(EwYEd{*}AvZWBEYtbRC)Z`rlz=ZCv+!)3=<#9lyz*E&EP- z)sgwNGE3t>6xaa$ekJ&O->)}`|DP|fAzF(WPQ~x65~5kHMk$+-@(IGVqa%-?b?oD@ zQua{XF7uq8fC=N2Wpj(5b4FA!Zd;r77z2A4Xo*L34yr+ldkHx0y`=-|<}FXpghgJG zT}`-bRXJI%4!+4#%|bVsQq?EjfAfj9ScgSFZ%M38m2Gi6XTM;4^gj2P1r{26aBGJu zn5ItAuy#=zgR;;iVRSGl(S(advap+Vq$H@vvdMby7Q(QaHty~LSA8`;>^iR>fJYk| z-NYEg=9!aZt81ZW6Lge8f$J(Q^%{e4h5*Yo14NA(8qq1!i&#uW}SLDVw3a zoB(tCexTao`^wJ^e$j)oPa4xVbv_O!&~p@p?mK5m8=7yvvlaV+H-nK8y&vodnsA%^ z1e&Vcxnw8|wk?Q0{|QtED9%rPtYR=l*!b=wImer9!Pk`eUiJyMug=DY56SgT0cAN- zR`h~#6{0M`@XHQKFx}@j86GvV$K1r;(6y&%p2xZ(w~c$Br8s4eV9(t8B{Z8n+mK54 zp_;^z?UL?c=AU6!GskHP!a309IG|dMTLF;5TK9K?NwF~eysjIX3J@!R6B_3P-^H(a ztTj;s5@PEj_BA2)7p-ypniKm`ZMnq9A!npv+X^kAFUuaKwEG?~g9wqaV7X*O(V6J=2#t|O3$BGKBFvBVWYce>W1w`ijb7&pCCGl>HGw?KDG`=u@+8G{b= zSVR3*t}OBMedMELClEarZ>^?WK8IlN%zCNH@Q{Idwf;OHJZDM>G=}|MRO@aaHbWUZ#{Ap7Z ztJ%aNWcq=OLawX?^+Ntc0Pd6rND9!cq)Xh zl}607cqReV=mHc%U2sY=b+3CHAhCJLkBF=IY@-w#N;UJ|Hk5^WN7FK6>=54^IxX%*0ItORa@8P6exJgCtHCWCW8D(%utA7GI=IzyN$ zwo0d3VM^MuuOH>>5BN8LH(pW0P`pyCC{iZj2~LO_^Lw#z*H9U+AwhSW1T zYD8{`;V9xE!y^bQ|SC>h*%cs4IuBJq>thTQv>0)6w1m}8e*&>A*uVD9f5G4C@Q|g?c}q{-JG)IuSC3>Q4)5C8` zb}s#)Ex?2ZAfv47v-LCbE0yYJOMe-sZ;7^?|k!mO_Su>;cwxZWowMjuhS9C7EUJwxI5{Zl}O z{4e*A{%>429ilz+t!De_;zju#fM&o4IK=$O)b}3p+_1?lh&?_i4zfepOE%TXN&J1< zLTp;9RRU~KcBOYjDcqvmOHK8Br!I%{lUqkiE;OG4L{OmRW++7nRV$j2(4dHHz=s0D zJd{;YPco%c2z|M1wIFt}8>g|!N36HW={A>~WCG&Fg9IX*l&Hdx3T#;TC1x|d4YTtX ziqmlTqm$FW__-mTPT+bz;v@=nsF_0}A7($CXF2L%b9XRg!^z{%o;`ay1=Mx;(WM(D zAr5R7!@}V?=jPlV512rWX1^pqJbXY1beHa_yuSBkgqm1*gMUzAEQr_-+V! zg_zZxHe+Xma&*PwMWR&{tEP1|JTmoHJ2Q5DN#?x#TQa8v843aWa+2`SfND+MUf_7g zI%w|q3}2hX-6c?2HY?}E)>O>ef6|nb?F>0}mzqF8)b)v_#T-f@c+m+Nd_|SZelGF6 zqlb5dVp19Z^z&nEg^wzNAKy(Y*&aj9zX8kTR(uhbh>L)lDLK3=i~-b8O@_Ruq^{m_ zi5}W$TgAu6Q}C=H*|87 zE#WozfGQA0DNQfKQ&h_}X$1~3?twe=jZh6y+*0QUv`)uBt~{6DdyomVPs1?tRyvW# z+5fr|ls*{M((RcF5e&cx9LG-z~jh~Um3hF4NL06}S zs~5VXihKMB%t{%XV%s_YFfl}Im=QL?6;V+S+rtm3=O2-7kp&!R!FyyTSy(fwF}GH9 ztDcXBx7Z{db1R=xWX@tsSzM00^CW0?*g)XXy+GWvSs`>a;9JNeTm$26!S)<>L$ksR zvgv~4GwB?I6EQb$SlHKH+b?Jxc;qgcNkL%AFinqTv+;|{g$SSJs!p2Cix(A9Oofh@ zgvL7Di=yM;2bbaq`vwP7B&yK?Wp*`|eA7CSR(`NL#h#btWX(|Yo$ZkZlg@IxnwzLpL;Ye008sVEqYnrk^@-4)GY^ z+)N~0FhS9rDCJq6vKB1CkR9`{WZ?TeBQ0I`$-1j*Wb52Koa}CLuQ2$kFt@|rC)#uJ z{G~lIx=JUy54+`9Eh@ouwD^3oLK8(acn;aR3qJ<2=@ohfPZg$EQi3&mAodszhn(H{ z13Trkt2m`Q;rcHy1sQ&Sg?c|l)+}$JcE`sac;1WAmv%> zKyIc`?q%ypfWfS9>BM28aw;ZAzs;HW2?WL3q$nOT$T%5)n=_h@Aq)`*D?f9Nw%pYC zqiS}`A953aNKAf_#Qv2B_U83eil1)UIhaA2wF!a3!f%njI<^e{W=Z}Je>I7u52E(c z>`1)BdsncvkG5NsVhab2`_;Q;#JOrb%CL*P=VJjb1k7efzBXrtjzIZwR)O%nC(sW| zKps3xjqNx6caxIPqj+|j11a`VvM0hI<_xwl3pmE&fQ;P`|7IUpXp-8U=IrJj&gvUq zvbMrKVRb{iIeWwGXk(gIxd4w_%&7zFM>Ws+^r=;N>VCprAxg42e!{93R%*BUT~z zq-kqcPMv{H_N$1N_qp5joz>)tF-Su-W+1PmwVGNm1b9G^be9Mpt;`W*WLLT4-1hA= z-~pgSb3Qt#AVwx5z?JmM?V;?mRu8X$@~P#fBT15Pue&tbttONC>4BD9pFoTN$UQ6L zWoC>qjq?U*0$TLojECb>{J8s&#w2Lwmo$yNuNGizY@z|kJf|?UNK*uBmB=KNNqL{w zD0;7hqIf#WHmf8zUpZQr|A;9eOr}s=Ecd-W|5>xXTQ{`PR=XZ)KdrZ{y!-yl)eJ_3 za>1-CA2AMbh>)`2V1(^Hg(g0!QXeG86l9_T;;4^i@ZL5!oWEp|!9UHjaQj5$rfZJ@ zS&0!h$lD}}2Hcn$CfnI35qwn#C@k8~dujS`82n;Nd#*yviLT)to#sknJJM~V&C>5> z_LpfKD>;$v2d1izXs$AqNzhqr0Ib+?qW;to{bVk;eZ>42d@oJ|BJ9Lh@j$EjdhCHy zZ9c+fVH7`!?D#p89*1xJ@tTXlb;uP?S166I<#Ts6xF-Ojt+fx|Al60f3m)`;?VT;a z0alRT=o2XK*E7dRHoC$WkRd91!nLLL3BA_d(|ehT>vbtk-47)YQ_j!8y*H7ZzkX-| zrVXYD9GDFiAarpC<2JG;Va|8kcB zWMV(x`IBAvJL2;VJ=*;xWH7Q&SdV#J$offND8J(Co5z>ql(gXa@1i{}U$AC~Ns>hr zs0tuDvH*wX;YBF@A*x~&KD^Of(6%X<8M-DET!Qm7)&jpc>6Gmjrc;d(bod%up* z@yiuY-7;|10+jSJCLrQVYVwE7h3`ALy#XlBnAtP)U@AjDVk`~t*8fi8*SE@hD5L99 z&o!r7M-faD{yWJmwGVl^jmb}{4woc#TUHg72Uy&3@|lD^wbLdFir#T^X}k~rmcsE?IqJ_z|Dz) zW&P=QUFq(DFZzM+C%o;@+lqMN>q^e~2o*imNpzu)mB)dXc^7hVC8!|GIGBxoz{-u4 za;xnPcuBfkTM{@xXBD9(gt6NWd1L~|jrV5g)q?psKgnxIdjsh5831f{W+J0+drPAV zgd;xFF9N;~ zeN;pG0MzRC%(r^`JPM2Q^DT$jBNk&ti*kjBAu|hs9PcIPq7@&->9OO zWzu2o*K`s&D}c&cN~081X|@mhQQC^#i`Ue#N>qx*~p!#n-`(f6N>s;pk7nFAT(m_99 z%g_tn`J5)-@y_K&nap$H4lf-wZcKh-J1MDo4oIpppPW>!a+;t0_z48g4QL!XoJV53 z>4xTWeBu{(qRSaRef=7>o5O9{p^!$GB__i42ZlVUo*09BW=7#CZF z^(osB!}80BJ&t8|K^4zw`i@*3xOU1`frrQ!@gsz=Q$?1yZWrA}@xejr2Cvt)#XWEQ=AUJOwdEmn6$(wd#tQkzPsa6s(Wh z1~nsYhQ}NxY3A8kz4tXM5)eM{1?3fv)Z@1uH0_^My^9+IEcGrs{zv%g{ShqQ@H+tc z#{tJQfai2)2Y8XE8#Dic2XHkFLCkIlpx2sv&|}pBz(rpJ^f})Dty+NJ?}rPN3^N?b zsKwv10==m^z$%KLkh2I&|7hBI{HfUV^jJvPcBQgsw_a};6Af8?Oc(a$ zqZ}hGLiYK(TiB4Mcx6~~n!aQk1B#jkKXIorHCz!60txF}uFxbZG`;Kt755|o4B zsW{=P(9siXCl}`t*U_GC&U|~?YcIx_fex=cDC%j`FphxMz#tmQZPc z|9#otm^Y1>j&;zFe#5EP9{!xUT{_TBd_d?|JqB)pF#HFiik zKA*=)DfKCd9+>WFq4u2A77BmY=g@o29CzY;(W_LokW%HugABs)bBIU;*}&I>qB9J- zerHlKF7)syxRLpOcz>>6U_*|QIyNb{4dy>|cZ|NaQyh49X4)kZ z?|1$k9(}kfqX>ZG0QCWY$2kIcoCbi$33RTa05~`(2Y`b&831sw?)68ufPi8BQ}DW9 z14^C#$G@~8V_jW?k}Od$@Smerl0esqTJ8L!TCNUE_va3~-@w?v+CSaY)TYU>7oca_ z(LQ79S)Va=4h${kD?*F_pkIE9|9Xmifo<1Kh(FAu zpD*ctUkLt=5bB>=k`FQzd?sG5T}uD36?`H1|9t1~Zk68?AE0}d5AmQ^Z(c#nI}JLU zaN1apj*b$w+3Vu99yrL_YVVBv#h}Wjt%HlWXVbnu1%+3`){juY;s+#8W$ZDO13>aL z+Ws@|pWgmCd8%$44ru%i0Xr}9=S%-=2-XSm*KgXra#2@&edEQ1=IY{gG8PMOZ0CzS za0d?ycW5e6-crq9vgxwj{;$;KA22(<&07XYI&8{C?O_GZ!SoA&d}vk|LO|1EDjDf7 z7x62H?18$MzjGM)N4|dsE87`761jV()?cVIM`oWQeQ>xc4^wQGToM|@qrh5iVz6zq z*8TwFvBx`kx)(35#%HVU)*~HlT=XJIt8Od1*WxxjRv~CxcCO7-(zbxrOUy$g@qU|{ zO4c(HCa^+s7|CKr(X960o7W}%NOAB&_k88yR&gTnl73*6{OAUKwXw$_k=1{R5topZy0_IoN|LUlfS+m*piEc#S056}d38(1-}k2g1iXE@I*V9qiK(>FoL}@5 zt8TBH<-=UDDQq15XiY}zgAOvKf?PM2^F8$S{Q#7HNn1(OU&`O(b%mBOWHdA z0s;KVB>l-GrKUgk=K*e7N*#zP7C>;H5_>})iJMuKBBX~Cs<9@NNCM25?|Gzio0;Y) zhk;3^Ni4k+KTemTXcLm#`{}}({Aj8y&N$5j^%&+#(?T2MHsdag<M|c9o<#MI<#Wji8bB^RdQpyNikz6^$O2RA4GfOQLYST|5%ZcCr%zn{VDO~* zo!F_IU*y%*_PU}-28bZzOCM|E8>^uqy^vkW*s1w&`0XVg-53})-_kG4MfZ?0Ni6>Tn#FiSrrQ^ z7DZ7Wa>P(B?hkuUueM+M zQ{%l9izi=qOA4l&erfD74MJ)tyf&=M<5{)ev4elycq*@+IoY&2(qF%9<5k7$+Dngj zfWrWfsMQQr4*R$5eHXC{SG>K9pPK`T4B_V#Y^F(=qZM~=p+vvb$zsSZ8CxeXA@<3L zHFNqqs%`S6Bamwm;O8EDmOc3sR}ky|6!%L787EZ$7gazxa<~3=Z<>Qg0*!w&{WDSr z?0`uCB~(N7v(?CM1P!UV?w7AUO8=hMzAIe+z72f?%#+wLsXIzMHQ=k(6ObEY+bSQ_ z?Ghl^#z7zD1`01Q%%k}-oE!;e7_!y#LFGy_#L#l6b8ta%%6~|?-bcZ;} z$savW(3_ULdFvzyBH5&DZTE9=`&VrM>(RWm^I#!z6t!dWtbJYuW|xe^&7N!yfjcL05MS?8%v7eWB zQIzppm8U9-hvR6c{gr$#V--V`_0z^r4W;bc-8_t}M)Q(ez3;nQ7vyj+MsU$jp(d_f zkV!q{aSu;Fjiuv&zu?-VZLx~b$=9t#WMQcvv_~L1Q&IzOUnCIvf&6hrYS7+T& z?4AKe;;RIY?4bdM2!XYFTaw!{9+YGR?=kxaZ}$PX>s+4LrkGnlt{*YAZJskbM%hJoo?vpri|~;vuU( zl1a*P8Yap+_QtVtMDrFEpM1DSPo(DfaKNhrl}urx<_nhIbI+Dy;4@D2^mB(aS&nnW zc53@INpt{!*sC|+;)wqK*^fesb^{lGpIYc54`P+S^e12RU&Q$SjaB=%ui1K`U{>FK zq?niS8FlCop3>)?1+bH)Zw?1m#2h^dDc|eZ&d;W{HOvffn$+Sf2CC+uhtZTnwrOM& z0ywC`gw(Ql{mnaaTqIp^vT7FzeN$N{`xD6aRAict=i%JKZ4qp|oLX0&c=5BI9_N3e zrl6Gd7Go=c{J0oQ+PoAZ1Kr#TRO(bCb1vQ-X=iTtj?>hAlOxb408`v-Hew^$viGKm z>-A)jC-=dEp2CDrpn75mdKb3SZA7{Z%aKLSU!Sqc)Q%~)u0l( zSW+xYI(B}?qn8nGtv0Kv!!xTK>40xr7lIB^+gHh1=eOpOCQfxH0Aihj)5L{=M9ch! zer@_UcEwvuU`F@7zS_>Hc?ICm);K9wEkzUIW9~YcMg~=1l^#>3APC|~65@+Ha;uMQ z3ux%rq6o;P4~LDYc9J~kVTK5T0|cPxrV_!7fZ|1tzB>l+gI)m<$8!1CK3e%Fb*-JTo35IF^1xl66CsPRLyE2qpl0oSp}ScY5q5qyhTISJzM`P|G~|y*AdjM z3)PDXbx2KWmVHq7$cAXi%wimr&enu#(aYBjYI-oWTohRg_Sv(Xr>1iB=lK3@eE*2K zj{AnOdNx-HMJaD|M8_52QMLi=pjp*66yYq%Ia)>{WXtxO8OJ`>0*2C8$X@U~8~1HD zPV>p$knHBzFz#x?x`+(G%o`*C1!n*-ow5qHrzipF`5_Fc0LTa+I~d^_*VaAjT=y#= z_xYdA{2H%6#KsA@{m&Z6K&czwegcj4mkpq$3Qz3c7;MszTy>;H)0e&6B4hmg!j=nX)w`~dmv zXR#&g{qq;^v5sl*okz^IB1Fs&Zcr`(jH~16=e(`~)!Jofnf8x2&5nKoCBc^U-!}rr zQP^2DXth|TefQ*$B1MKOOL1xB!3ItF3sM#*uyre#V3|W=Xn-;5d(#3qYli`ttNY+LYW+V}}Gcd>V)^hGldUtx)Jken5 z^kMMFBrQw<4_LCvUF%M{)a5%YKI=Y7TFMRf79`yZ{a!N;Oq+u?$A*FQ4nvkdVDA4T zf6%|InF{sAMv^Kj%gl_XLn)=|PC{8!IX&f0yn)j$f`gUST~Y>y22lIWTZEEf2TtEP ze(U2B>mA!z)BF;l2GHWnn!kkBz4~ZD4~JfSO1U`NG82vgs|tBS%z?6s&24ipTWoTg z(k)xAx#AuH?xWf2yUtZ6zST&361&9IxrK{2b?6>T%h z(B>xSbBRqgKijt7|K0;=NQ$TcKPl53QRK)skw8=cuji910$ znVK@ew;MyY&6iDkYL0U>7Sha*o_2R|-mO6V;Ob!ZWV^gUod~w$&HrHF{;6&>`3R}k z!4|9x)~9Nvy}}q+E%cAuKmpB^aBDV@m)Q9 zrx~LU1-{uabNv~B5kLW54MbJcvwzfN%4--2_SNn=nLNVyyd!^c^ZNEdrtOKKBL^AY(*AOX_?IoH|J3r1P6VWG&v0nsA;QdC#E4ut1gNhbYVHv* z?JBWhvaG6bgzJ`>47U?edVG4fD04&>$E@;2)sEH&M&T*>fuirLl%Zjm0%#;%)WV$J zVI@zxOHn5!94jMTr0ve<4?3*nx^P(2*01$NOl<2>s&PM67&B`^Hme?ULk1Nso61 zuiq5mU#CkVNWV}GpoD&(?)z^D)t|6L8D1kC+WS79qs*jxLd{5ebrO^l71qO0_Vh}i zP#2qQbTQfD{A#?9c1qIQraej9a&mJIxSH|96OWd?2r!_(hUOF7r&b9n&Md5+6M7t; zsQRIUWn``h<&(JUeQAryM>Td|DmJ;cNwRu#1#5%lD=ib3a<;}U`#&ky`8!s8mB%Uo zH(Be2qY5SJpuu!;5~>l-Y>yMZDPj8tL)by~s2yKul{?p?s_*2ZsGivehh5;VLsoLW zlepC!=pT}DL-jnleM*?B?)hQKf-FMEme~g8`W_=i`14Sndk|)VhAgYl4m*kwwk6#o zC2xeBj*`XE0>x6waat6U*?Z3dS2U}WSb@Sy6o@tne0yqhZd{5(o~c98GwogdwLFbH z8@=R>7)UuH-=*ni*gj%X3l4_vQD*leVMp3;7NFTEK8ELbqC#^`HCg@&RwNzd%0L9m67)YaZ7Z_o6!tCsDA+@3CIj4~|P zvfX9#>rQ^%o%}mX&rjE)-*TK~jP-AY<`;*k?tpq?8u4s&!HfuB_-N}TUZ7_u7jgT$ zo!l+@HJzMbPL6^PsyFFt-r+IZm!!Qfar)(g;_tfQ3&(I7^%7NxB%;Wub7Vh!HM|c@ z=$Q^6Ai*#zu7I;-a6{Ih>I8XyvQON~y*OO0#5CZ+(&jlRXjEo$g^MUog}|!hiz#{z zV@i4cR)(YglG?L-Cmq{zd<3_X9P@iAgNccctlCJ@ATCprDH}u zfi_YuQO$aMS*%AA;KYMnxGndx^Rkrlw!E|??r9jl>b;9aH>ILw(PuTspV7b5YI`Bp zcOO~3CT5b=)4Tl~84@(#&e3mmp5oiSvH4QjxeMegVkWL34{>K0PmR%c`?^pB$=!P8 zJyqL$FOkp@PSR9lK7IgLQe<(r-Migp-@!btM$((AAMbqjJ4P(agQ}y5Y9%*JsQ&>n z{U!m0YzIKuU@%vqkEPB?@GT61E9%1t-IGWpI;+$xmNNSZ#LBp;QE`ql&W_H z$r`?Aj6SxDbH4IVMGk)xK$zU8cZ2hjipfPw)m2IqFR}z3ASvx!YH~1j4O3ThOMl& zJ)^6T{nLiE&{y1E%|zz10wRcl#zlDkc3zuexT5Z}TtW8bNPtuZR6b-U6;F+l3~Qkb z6I}Z!_tM$4n;9JeyqaW;>(kcZ;OqrfdZh8!u>l5PmWTs@Vmd-W%Tt|Cc>?+&;62D3X`Ks zooZ5?((2QX}q1*|h_TN$XT%phyM`nk52);x<~0#SskA=_wvZ~}OPe~YC5>m>0aN(KRmQ}8tzVF{J; zJaew=U7p^qmZz2zli8fR24sd3Tb&lK{eqI?Gi-QvJA}qQtYW9O8cT%OR9Y)mCv)grs_zSvWnemobn4jJ77%_)`=azBbf6#w>ar+0=R;$ zXeXAx7#N1IaRjF{KEhwb(xN6w;Ek^+*DYxuCGcL_amvXmRx%fA6ZRKruNqAzM}3RB k_w}!@9uPg%%%|H2w*9aC^5^X8&;NhM;NNi!fIs#9KYW7SbN~PV literal 0 HcmV?d00001 diff --git a/docs/getting_started.md b/docs/getting_started.md index e45c7e13..9fcc5b1b 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -184,7 +184,7 @@ We provide lots of useful tools under `tools/` directory. We provide a script adapted from [flops-counter.pytorch](https://github.com/sovrasov/flops-counter.pytorch) to compute the FLOPs and params of a given model. ```shell -python tools/get_flops.py ${CONFIG_FILE} [--shape ${INPUT_SHAPE}] +python tools/analysis_tools/get_flops.py ${CONFIG_FILE} [--shape ${INPUT_SHAPE}] ``` You will get the result like this. diff --git a/docs/index.rst b/docs/index.rst index af646641..3a29b7a9 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -38,6 +38,8 @@ You can switch between Chinese and English documentation in the lower-left corne tools/pytorch2torchscript.md tools/model_serving.md tools/visualization.md + tools/analysis.md + tools/miscellaneous.md .. toctree:: diff --git a/docs/tools/analysis.md b/docs/tools/analysis.md new file mode 100644 index 00000000..a127eee3 --- /dev/null +++ b/docs/tools/analysis.md @@ -0,0 +1,211 @@ +# Analysis + + + +- [Log Analysis](#log-analysis) + - [Plot Curves](#plot-curves) + - [Calculate Training Time](#calculate-training-time) +- [Result Analysis](#result-analysis) + - [Evaluate Results](#evaluate-results) + - [View Typical Results](#view-typical-results) +- [Model Complexity](#model-complexity) +- [FAQs](#faqs) + + + +## Log Analysis + +### Plot Curves + +`tools/analysis_tools/analyze_logs.py` plots curves of given keys according to the log files. + +

+ +```shell +python tools/analysis_tools/analyze_logs.py plot_curve \ + ${JSON_LOGS} \ + [--keys ${KEYS}] \ + [--title ${TITLE}] \ + [--legend ${LEGEND}] \ + [--backend ${BACKEND}] \ + [--style ${STYLE}] \ + [--out ${OUT_FILE}] \ + [--window-size ${WINDOW_SIZE}] +``` + +**Description of all arguments**: + +- `json_logs` : The paths of the log files, separate multiple files by spaces. +- `--keys` : The fields of the logs to analyze, separate multiple keys by spaces. Defaults to 'loss'. +- `--title` : The title of the figure. Defaults to use the filename. +- `--legend` : The names of legend, the number of which must be equal to `len(${JSON_LOGS}) * len(${KEYS})`. Defaults to use `"${JSON_LOG}-${KEYS}"`. +- `--backend` : The backend of matplotlib. Defaults to auto selected by matplotlib. +- `--style` : The style of the figure. Default to `whitegrid`. +- `--out` : The path of the output picture. If not set, the figure won't be saved. +- `--window-size`: The shape of the display window. The format should be `'W*H'`. Defaults to `'12*7'`. + +```{note} +The `--style` option depends on `seaborn` package, please install it before setting it. +``` + +Examples: + +- Plot the loss curve in training. + + ```shell + python tools/analysis_tools/analyze_logs.py plot_curve your_log_json --keys loss --legend loss + ``` + +- Plot the top-1 accuracy and top-5 accuracy curves, and save the figure to results.jpg. + + ```shell + python tools/analysis_tools/analyze_logs.py plot_curve your_log_json --keys accuracy_top-1 accuracy_top-5 --legend top1 top5 --out results.jpg + ``` + +- Compare the top-1 accuracy of two log files in the same figure. + + ```shell + python tools/analysis_tools/analyze_logs.py plot_curve log1.json log2.json --keys accuracy_top-1 --legend exp1 exp2 + ``` + +```{note} +The tool will automatically select to find keys in training logs or validation logs according to the keys. +Therefore, if you add a custom evaluation metric, please also add the key to `TEST_METRICS` in this tool. +``` + +### Calculate Training Time + +`tools/analysis_tools/analyze_logs.py` can also calculate the training time according to the log files. + +```shell +python tools/analysis_tools/analyze_logs.py cal_train_time \ + ${JSON_LOGS} + [--include-outliers] +``` + +**Description of all arguments**: + +- `json_logs` : The paths of the log files, separate multiple files by spaces. +- `--include-outliers` : If set, include the first iteration in each epoch (Sometimes the time of first iterations is longer). + +Example: + +```shell +python tools/analysis_tools/analyze_logs.py cal_train_time work_dirs/some_exp/20200422_153324.log.json +``` + +The output is expected to be like the below. + +```text +-----Analyze train time of work_dirs/some_exp/20200422_153324.log.json----- +slowest epoch 68, average time is 0.3818 +fastest epoch 1, average time is 0.3694 +time std over epochs is 0.0020 +average iter time: 0.3777 s/iter +``` + +## Result Analysis + +With the `--out` argument in `tools/train.py`, we can save the inference results of all samples as a file. +And with this result file, we can do further analysis. + +### Evaluate Results + +`tools/analysis_tools/eval_metric.py` can evaluate metrics again. + +```shell +python tools/analysis_tools/eval_metric.py \ + ${CONFIG} \ + ${RESULT} \ + [--metrics ${METRICS}] \ + [--cfg-options ${CFG_OPTIONS}] \ + [--metric-options ${METRIC_OPTIONS}] +``` + +Description of all arguments: + +- `config` : The path of the model config file. +- `result`: The Output result file in json/pickle format from `tools/test.py`. +- `--metrics` : Evaluation metrics, the acceptable values depend on the dataset. +- `--cfg-options`: If specified, the key-value pair config will be merged into the config file, for more details please refer to [Tutorial 1: Learn about Configs](../tutorials/config.md) +- `--metric-options`: If specified, the key-value pair arguments will be passed to the `metric_options` argument of dataset's `evaluate` function. + +```{note} +In `tools/test.py`, we support using `--out-items` option to select which kind of results will be saved. Please ensure the result file includes "class_scores" to use this tool. +``` + +**Examples**: + +```shell +python tools/analysis_tools/eval_metric.py configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py your_result.pkl --metrics accuracy --metric-options "topk=(1,5)" +``` + +### View Typical Results + +`tools/analysis_tools/analyze_results.py` can save the images with the highest scores in successful or failed prediction. + +```shell +python tools/analysis_tools/analyze_results.py \ + ${CONFIG} \ + ${RESULT} \ + [--out-dir ${OUT_DIR}] \ + [--topk ${TOPK}] \ + [--cfg-options ${CFG_OPTIONS}] +``` + +**Description of all arguments**: + +- `config` : The path of the model config file. +- `result`: Output result file in json/pickle format from `tools/test.py`. +- `--out_dir`: Directory to store output files. +- `--topk`: The number of images in successful or failed prediction with the highest `topk` scores to save. If not specified, it will be set to 20. +- `--cfg-options`: If specified, the key-value pair config will be merged into the config file, for more details please refer to [Tutorial 1: Learn about Configs](../tutorials/config.md) + +```{note} +In `tools/test.py`, we support using `--out-items` option to select which kind of results will be saved. Please ensure the result file includes "pred_score", "pred_label" and "pred_class" to use this tool. +``` + +**Examples**: + +```shell +python tools/analysis_tools/analyze_results.py \ + configs/resnet/resnet50_b32x8_imagenet.py \ + result.pkl \ + --out_dir results \ + --topk 50 +``` + +## Model Complexity + +### Get the FLOPs and params (experimental) + +We provide a script adapted from [flops-counter.pytorch](https://github.com/sovrasov/flops-counter.pytorch) to compute the FLOPs and params of a given model. + +```shell +python tools/analysis_tools/get_flops.py ${CONFIG_FILE} [--shape ${INPUT_SHAPE}] +``` + +Description of all arguments: + +- `config` : The path of the model config file. +- `--shape`: Input size, support single value or double value parameter, such as `--shape 256` or `--shape 224 256`. If not set, default to be `224 224`. + +You will get a result like this. + +```text +============================== +Input shape: (3, 224, 224) +Flops: 4.12 GFLOPs +Params: 25.56 M +============================== +``` + +```{warning} +This tool is still experimental and we do not guarantee that the number is correct. You may well use the result for simple comparisons, but double-check it before you adopt it in technical reports or papers. +- FLOPs are related to the input shape while parameters are not. The default input shape is (1, 3, 224, 224). +- Some operators are not counted into FLOPs like GN and custom operators. Refer to [`mmcv.cnn.get_model_complexity_info()`](https://github.com/open-mmlab/mmcv/blob/master/mmcv/cnn/utils/flops_counter.py) for details. +``` + +## FAQs + +- None diff --git a/docs/tools/miscellaneous.md b/docs/tools/miscellaneous.md new file mode 100644 index 00000000..0463b525 --- /dev/null +++ b/docs/tools/miscellaneous.md @@ -0,0 +1,59 @@ +# MISCELLANEOUS + + + +- [Print the entire config](#print-the-entire-config) +- [Verify Dataset](#verify-dataset) +- [FAQs](#faqs) + + + +## Print the entire config + +`tools/misc/print_config.py` prints the whole config verbatim, expanding all its imports. + +```shell +python tools/misc/print_config.py ${CONFIG} [--cfg-options ${CFG_OPTIONS}] +``` + +Description of all arguments: + +- `config` : The path of the model config file. +- `--cfg-options`: If specified, the key-value pair config will be merged into the config file, for more details please refer to [Tutorial 1: Learn about Configs](../tutorials/config.md) + +**Examples**: + +```shell +python tools/misc/print_config.py configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py +``` + +## Verify Dataset + +`tools/misc/verify_dataset.py` can verify dataset, check whether there are broken pictures in the given dataset. + +```shell +python tools/print_config.py \ + ${CONFIG} \ + [--out-path ${OUT-PATH}] \ + [--phase ${PHASE}] \ + [--num-process ${NUM-PROCESS}] + [--cfg-options ${CFG_OPTIONS}] +``` + +**Description of all arguments**: + +- `config` : The path of the model config file. +- `--out-path` : The path to save the verification result, if not set, defaults to 'brokenfiles.log'. +- `--phase` : Phase of dataset to verify, accept "train" "test" and "val", if not set, defaults to "train". +- `--num-process` : number of process to use, if not set, defaults to 1. +- `--cfg-options`: If specified, the key-value pair config will be merged into the config file, for more details please refer to [Tutorial 1: Learn about Configs](../tutorials/config.md) + +**Examples**: + +```shell +python tools/misc/verify_dataset.py configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py --out-path broken_imgs.log --phase val --num-process 8 +``` + +## FAQs + +- None diff --git a/docs/tutorials/config.md b/docs/tutorials/config.md index 4424e5e5..b4899f41 100644 --- a/docs/tutorials/config.md +++ b/docs/tutorials/config.md @@ -2,7 +2,7 @@ MMClassification mainly uses python files as configs. The design of our configuration file system integrates modularity and inheritance, facilitating users to conduct various experiments. All configuration files are placed in the `configs` folder, which mainly contains the primitive configuration folder of `_base_` and many algorithm folders such as `resnet`, `swin_transformer`, `vision_transformer`, etc. -If you wish to inspect the config file, you may run `python tools/analysis/print_config.py /PATH/TO/CONFIG` to see the complete config. +If you wish to inspect the config file, you may run `python tools/misc/print_config.py /PATH/TO/CONFIG` to see the complete config. diff --git a/docs_zh-CN/_static/image/tools/analysis/analyze_log.jpg b/docs_zh-CN/_static/image/tools/analysis/analyze_log.jpg new file mode 100644 index 0000000000000000000000000000000000000000..8eb1a27d6464d255b84b23a7460a5f622f51712f GIT binary patch literal 68146 zcmeFa2|Sd2zdwG7C@r!@#8i|mv{4(ldbF*aO~>e*o-0EiKsTonVi_yFI|q0XpWLEJsgY zpl8)GXE^4-cE&68?JmBH*;VY?9XS58w;UhuW<1Ejd5G)y2?0SNVF^ho>2or&mo6(^ zQNF68s&hkEPv5}M$l|u;9V=@a+xt$=4<5R>x_LkG@%4M^9}pJ)A|f*CWpqq@LSj;K z%DdF}Ik|cH1%*Y$CDk>xb@dG&8=E@2x_f&2KKBn`$Hu=-OioSD%;J|Z#i}_9^{w69>;$h+TTX@#|HNJKQyxcH?V(D<&C=8QBxdSd#Tw#~cF z)0+CkI~RxMR-4ze38G^m1JdjEcyuLo3I781d6ps zGaI{nZiT!l)eyk1tnVi9;8WI>dNH?yAvL|c+#kwt>26ni(hX)D%C_V`9P@ZKb4u&Y z+epFNv~H27$M3aT(|#gJ67@WXF}z|VOFUxToS=hSx`BguQ;v)f?$lJ547GeR+6d6s zC?y_W>*h$lotI4>)A-tOQX9d%Z}F8c51e>|G7LK`uY?zZRWz1P)Ap`;a zTB3(l1LY8bhiGm)yp!OHSF2^6GdqK;3-Pr;YZsQ4+%VqglDs6V5HfJl-mc@i*3j^o zCpa2WzlBo#tvM{y0&IMfbULoa11F;2n}HO#yPnHdY^PcU>lH?T ze!sy=1t<9f%8n@rowwm#v4eEHOOeaQ?Li7rkpLf)_IJh{nBnqiu2WcB=qay@DAOFO zm&&P5mU^J5r93pbq*+jLk_uFvuHbV?9v|CyLSdrxMValziPsNmBdxH-^4z>ht~^~f zp*Ee(a1YU@JcpLO%)7P5Lt7!uVaKYXIz4xL){Wp&6cugIhf!Edlr22D7J?Dv&Dw^LFye2!tClTBH;Ky)P85(4qM zC<_#TJXeexsfMyh`|)>?Y>tn1BTdt)YJCv&oTuQ+P zm7?=@$7@?p6q%Zy$O*e@s8x7bS6>FIMUq4CLTkwTC_+@=4yyW^le@*((fC??f8df3 zk_sHX{m?S-(@@^o>^(u*@_h{`PRt!sxpf~Z@Hhy`+b)6fF_X*2QGv<~E){v$xTss} zjV^f=wp;ac${zw|wXdg~)bBXnRrJw6BS~NG>-fR|AQ&I;Kjvz|-zk2GU4RfT*4NAV z$IR~H3os0_H8RvbrMB}HARg$B=@R*Zpck zw%DCuB;IB!@Ua_xlK{4SNh|`K@Iopuo`<YhZ3B;12e%ZXC~ zK3zN3+F@wlDqP*Mt=dSn#`SJ^Z4Nw)p0SD3>fl!^hTilFH2y#;Lv0C+kK%=(0u8ok zUlMe@g|DVpRu}D`*TtTn=%9qT<~XQ({*WTVgb^quzs2)Wd`vm007DKHSeAqgKv@cy2VJ#bmyRGy*O?7eCa%^%!+ld+Pc5n*t~`{PIR zs@DD1JD3)j{2{Bt(-?w66cq@4w|x-{1>^QY|99TA{}zOmK(VtQzo4<(M$tJxPv>KU zakz~kmy#w^|H|w7{_&zT)c4b?1evNXftqNe1HjC@RC=ptJ6dAhYGs6e_W4()@=m_9=8e!vd`ZQG#?LYV2ez zOBsHDf^)jffViK$T*{|VwfD188xDp8#=H49kLktS9^$+A^w0-~$28MU`b~W(oEGe< zEG*b~{caEHPT+0Ah*-ImMGApyq+!pA2nVbCmdGnJ( zM96#{omSUeYAe0Jmb zjj%Y{_usFAOD^S7EH9|Y(oi$-a6UDel@`LvwxC}19V7=) zf$4>cLsS6!5OhB9v#Vq8M_ey#1;`gsfvFgZZWqJjuO|2i(rY_HU2Y_tr*-rA9;Xt} zN#4!x>Q64fSja;N%`jgS!E}ZSkld1SQ49BpLN4*d>$pN+j7^r`*0U53{s-$u&LKhR zma$`Zj_OGSTb^800R-v79L%0|G0 zJuLU2d{LxO1>_+R8|XmgEmfikLOzqPa4WUE+&vlcD!7emHx3_p(TtgmR69G5)Hyew z52Z2Ii|bjD?adPD&+TIb;|jSCA;&k~xaco{L#3k-REXD*nt@{@f%0}^WXO$>3wd3; zaKlMEOTw+(Hr6?8muQ>WTQa&sP9j+2Q#9!eZVjIqYDKVqH(WkWwYj&Qe7;t z75R9vhWg=sAG-2i8H>y*9HauOc)GqweCztc{gdlm7bJaJ*H)7D2Kcyoq)nUcz+)DK z@RX6Ul=)~p?U;Sk=m@Wih<0auF@B>a<4~kX`3cI-B11d7rKH#WLu_g*c0#&ZN9FO9Y6N;0t#O6v;mwY;0X&o4xG zf%NpP@44{+iOT_k2};15(&(3uT-!)2+5%um^!Pqf__1;0;vtDcx+x_u8&4*hUYC~R zH4M?#;17cjk_5hq;{hu0Sr@e$EBuihTR~6IScenN#*y~n#1h^O%_#EZ!`(%KjjX>G zJKOMHG)=gy?KlJ>h%$5c_V6Uab2K>fF;|I#=cTVz$RVuJ4#Fa(% zx{F%18?ygnio(EC6ny4}FJ@kX3T!FgH;&lm^3fFkL4C7BBg(UJIC2?RNShHm&nJ|G>rRh9bQP)~8e!K)0k83q_bfKQt`#Jvh4u8>qmQq239A zO-tz~m#%$&IoHQf_q2g(WtbFJ9}=DV0=v^y&TaaJg;1#ytUX`pzS~H>#uGDcFd*|D z3-L?$pDvXBw5#Fsj(&j|Lo}|P3hc2OSMHWHy@4(z2V&ER?ITxw55gMGWim$B>!>2b zYcD8%!6O#0YvANkCuA}VRxUmnSZe04SdYiB7RDE$38q&NF*R6$;7qnv#Qf3ysNHXN zkM^UkM1hEep$p*?N-fN0?~qz!g<5< z#e2`}3*>Nr7~4#iQ63dgIa`np{kK?w0S5PiLhSuSOz4+gA9Ex_f5~Olno_ zQ}jecYE@%*N{a1s#Mj7-P2NByHvL4`Do}p@OJw5*`El7;7#~g3;GukS8;)|u&D9Q{ zfU94+)-Adq&x$PA8hQ~q;8QmD&Sw52Hgn;DNAkG>v}7Ff&84b!EGXZtuRb4EzCI0%`+a=diS!eOn1-%8!N(i2og^Z%YfL<9@ zrIQq$MKRDVr&JEJ;3Lskji)enq|IoHMoku8d@M6<%;W6E-i08J-bg0){5RP}PgeHk zDBTI?d^M?YP6ViWv=JakizY-Pct(6#diCz*%nrDnes(d4ru50ZrVBcY`<1Nb6j?e~ zLzt~u2Zn6L+EQgh@27pacKWNT3yTbZ@4jz@26fI?S_ZOj-V*twZ^%K^$NM!S>+A@! zK;$FIiYkVUNh*L@qDXi=W?E{Y0;mxZoTIKJs!mOWFpd-OcG`EPyLxY~?J$3AlTx_A z#BQF~RZ*OBg|uN;&NVhshRE@l=|}>Iz3gemP;~py`e%}^BzUK24Zcs2s!+FX-7<7m z?e6YTwI%ahOn;ap$67ZM92QbxwL4EiswYst7eT4+0*-9KV9alMqNbdw6yW#FtR z98oAzS_tPif_V+}V71$YEtEv$yXW zIc%y+9`=6POc);pLj8B%NRNm3E|u*YL;32XEIabvs7UzxkgH9`rDC81Ms)B%^)#Nr zNc{_w=KwM-L2f+c08$fIHhdIsRVB_z*fA+GRjAHUFJ0n3r0T`|WY%eLuBa@>;N2&$ z6>dvlz;HnNNnSau)`+yBZxnFG_G+B5?Z$y1S7zSJ_B8Tq@_iv7vUm<@l}V^MPY5W1 z)MmKXkL+`z0uafq+Af6si~Cb-8v{Y=IfPNODoebXQL;W#kER3bX% zB>^u+INw~J(Oaw*eXzjB$`#9gn@0Hqz&QW)w?MXS3$lG(;f?`7&5vuFux(xcmJ#@? zsXsPX!pVhM(`H;BJoIbbdXmv_$VhQ{u|WbX?KE(s^d)NI{Xqqq zn1(d8`AI86yR+mK#4<9KPK4hHRmW0+{ll=!6Ss@`{muOo8a|yG{NP$O+S^7FZDScB zRqznl$L8YQRGcCoID}-O?`3TsRqUBW*-dDSzE7gCg9*`@17Rdjt0Zz@jQF$g5W;J7 zDv;DFj@#t^0~YRRAJkgGIFQM2_)sV43O1W+)Sfl@2wiv~HH+Ci=ruYLJb-S?K-%VI zw~=yj8$RP92omG0S`hi3;tQ0mDypicCyHDWSFsfqLJ5QPhx*|i*v6>+bsibSF zAuojLJkQ=oJ!Z5=sF!P*By{qq&{#(8IR2J3^Vz>eoko)NPzw8*c_+-}s}fal!v8{Z}5DG1H8 zt}$}4wb`(BBpp|TRmB=pfxX1>(hvfP98c1KWUae0Q_SEhdh;kN&SO=*o>It5-5Pk; zVL@xq5vsf$$UeQQMGlU_1gpZvQne`i<6)051*hjm9YP4^>Yl={Q2tXu>wP%J+SNV8j+~SAsrIE0G zDIV$4C{21EDNk$P#E!x*LO6@S^AP~U%`}Ql{`}gnpmh9VRk;keYeYv1_*q82RDsMszkgsHpsxEKG zOl{l%efYWtD!`!zs=f_y%FsjlU?!deLNW#Hr7x|%b6O(c{3ylZ%jvF)5NmVJz#;t< zGQAj-Q1S^U?Jd!wUuAS9#r>Uauw?x!x||O^DTnR^vD|~uhM(sNX7~G{QP3B4f^L#~ zP&fw%FfMGTSFR87s$eC|*kIG32WMr2O9;mGZV3Opl@xWZLt^umE4Ngn6DE&+ImK=8 zET|lwn+_%Dmq19ASq$ME2-%KJgBI5PE$BThf?dR7vV+VCY$>|Av z-4_mibFtjtqS_Wg*1+HH$y*KPR&T%Si%v3n8o$>dxHi(F@z!-U)>+rMJvV*|GyMWR znS-G3vbH3QxZ^X3BM+8*NmqbURIh-KzG)00b3ZhG@mnz7ckk;#0#RQTuvx@?P)g{R z-U)L;A?4CAC@vqmK-0j}@=pQ5?^+5&r6D%dZ;?R%GVCGfM1JNSqykE!c0YsY{%ux) zJ$y`@*9B6|e28Q?t{|eObhsknUEMB7;D_|t-h_+$oo~lS>2<=~Z9GrBNY%}fWOQ&u z#H(Y9Pk@%xC}50fVK;7)xZ%kZu7l@KSrqV-@LvqYEaYT`B@Df0W)azme;KcZ_nk;t*Y{cd^qD9-FTldk%)B~8BR zdUprH0!fh46I8*-4w41#4Unt^-=Zv=pggu(RG`S8!WVsQ@l@^bvz(|Q=>C5iUNcJq5 zR%V!!N21@Fd_`>jX`#)Ftl`9okHI6XBo%`5Xej!S%Z5@XDBu@FwFRN8QtVN7#Pdk{ zvO{@v7jc^X%PEqweG!9bwB()k(1;~UL(O(WG(Q)NRk?SA5nJ{rH7 zY}_1MaaVZclh)m}IXAVY9@sdT6rMSX47IV%YF+Ka+Ms5Xxjdy({6j)<-c^MZK>?rc zc5?VN7%E@&v@o$UhYY$K{g}M#Szu&tCW4E6>bG<}GSmTeWJ`s=J!{MYs=81K+}umk z&M0X3;6y>u>&t@+4D;gzGel1-&90MK6ZxqU?f=e@+R@)^0 z2$4*LX$(So1fo$Ndf$M)+jDReZ?iaGeq{#J@c%r37zigDqe|c$*ep_bIRm9~ePm0> zn$|u*FmA_x;@bNi^b3Og-x~$TM~s5;k61L;>?kwous_vm)quL|YoaY?eP=cm=qPk9 zHuxUW|6?@abm&2lo#tKxTjxTN`HC}|eu$s{KCeHiz^Lt|oIpzBx|5lhN#{ExHaV7# zFJ|n$?76o(C&SE__i)dh_|nG2iuF#Cx{YSm(QIrE&!ES!Cn1(&>CTmb&uB-KrHcA|P^=srxKe|Y*~ozHLJAPvN8yD%@=)wM-?zeJ9!3iTlgs9m zq1vP*)Z4q7bCYX%DOV&&Q(b7GyYGFgRJ7{>iktn)Ge^#8iuIDb7OVt!6 zf>pwxi^au)9OE~#yEB8Bfv%>GR2zQ#%-Xd({GD)l^x|L&WGV9wMKdCT^yQ1b)rYRe zqf1HJurJ6|
+ +```shell +python tools/analysis_tools/analyze_logs.py plot_curve \ + ${JSON_LOGS} \ + [--keys ${KEYS}] \ + [--title ${TITLE}] \ + [--legend ${LEGEND}] \ + [--backend ${BACKEND}] \ + [--style ${STYLE}] \ + [--out ${OUT_FILE}] \ + [--window-size ${WINDOW_SIZE}] +``` + +所有参数的说明 + +- `json_logs` :模型配置文件的路径(可同时传入多个,使用空格分开)。 +- `--keys` :分析日志的关键字段,数量为 `len(${JSON_LOGS}) * len(${KEYS})` 默认为 'loss'。 +- `--title` :分析日志的图片名称,默认使用配置文件名, 默认为空。 +- `--legend` :图例名(可同时传入多个,使用空格分开,数目与 `${JSON_LOGS} * ${KEYS}` 数目一致)。默认使用 `"${JSON_LOG}-${KEYS}"`。 +- `--backend` :matplotlib 的绘图后端,默认由 matplotlib 自动选择。 +- `--style` :绘图配色风格,默认为 `whitegrid`。 +- `--out` :保存分析图片的路径,如不指定则不保存。 +- `--window-size`: 可视化窗口大小,如果没有指定,默认为 `12*7`。如果需要指定,需按照格式 `'W*H'`。 + +```{note} +`--style` 选项依赖于第三方库 `seaborn`,需要设置绘图风格请现安装该库。 +``` + +例如: + +- 绘制某日志文件对应的损失曲线图。 + + ```shell + python tools/analysis_tools/analyze_logs.py plot_curve your_log_json --keys loss --legend loss + ``` + +- 绘制某日志文件对应的 top-1 和 top-5 准确率曲线图,并将曲线图导出为 results.jpg 文件。 + + ```shell + python tools/analysis_tools/analyze_logs.py plot_curve your_log_json --keys accuracy_top-1 accuracy_top-5 --legend top1 top5 --out results.jpg + ``` + +- 在同一图像内绘制两份日志文件对应的 top-1 准确率曲线图。 + + ```shell + python tools/analysis_tools/analyze_logs.py plot_curve log1.json log2.json --keys accuracy_top-1 --legend run1 run2 + ``` + +```{note} +本工具会自动根据关键字段选择从日志的训练部分还是验证部分读取,因此如果你添加了 +自定义的验证指标,请把相对应的关键字段加入到本工具的 `TEST_METRICS` 变量中。 +``` + +### 统计训练时间 + +`tools/analysis_tools/analyze_logs.py` 也可以根据日志文件统计训练耗时。 + +```shell +python tools/analysis_tools/analyze_logs.py cal_train_time \ + ${JSON_LOGS} + [--include-outliers] +``` + +**所有参数的说明**: + +- `json_logs` :模型配置文件的路径(可同时传入多个,使用空格分开)。 +- `--include-outliers` :如果指定,将不会排除每个轮次中第一轮迭代的记录(有时第一轮迭代会耗时较长) + +**示例**: + +```shell +python tools/analysis_tools/analyze_logs.py cal_train_time work_dirs/some_exp/20200422_153324.log.json +``` + +预计输出结果如下所示: + +```text +-----Analyze train time of work_dirs/some_exp/20200422_153324.log.json----- +slowest epoch 68, average time is 0.3818 +fastest epoch 1, average time is 0.3694 +time std over epochs is 0.0020 +average iter time: 0.3777 s/iter +``` + +## 结果分析 + +利用 `tools/train.py` 的 `--out` 参数,我们可以将所有的样本的推理结果保存到输出 +文件中。利用这一文件,我们可以进行进一步的分析。 + +### 评估结果 + +`tools/analysis_tools/eval_metric.py` 可以用来再次计算评估结果。 + +```shell +python tools/analysis_tools/analyze_results.py \ + ${CONFIG} \ + ${RESULT} \ + [--metrics ${METRICS}] \ + [--cfg-options ${CFG_OPTIONS}] \ + [--metric-options ${METRIC_OPTIONS}] +``` + +**所有参数说明**: + +- `config` :配置文件的路径。 +- `result` : `tools/test.py` 的输出结果文件。 +- `metrics` : 评估的衡量指标,可接受的值取决于数据集类。 +- `--cfg-options`: 额外的配置选项,会被合入配置文件,参考[教程 1:如何编写配置文件](https://mmclassification.readthedocs.io/zh_CN/latest/tutorials/config.html)。 +- `--metric-options`: 如果指定了,这些选项将被传递给数据集 `evaluate` 函数的 `metric_options` 参数。 + +```{note} +在 `tools/test.py` 中,我们支持使用 `--out-items` 选项来选择保存哪些结果。为了使用本工具,请确保结果文件中包含 "class_scores"。 +``` + +**示例**: + +```shell +python tools/analysis_tools/analyze_results.py configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py ./result.pkl --metrics accuracy --metric-options "topk=(1,5)" +``` + +### 查看典型结果 + +`tools/analysis_tools/analyze_results.py` 可以保存预测成功/失败,同时得分最高的 k 个图像。 + +```shell +python tools/analysis_tools/analyze_results.py \ + ${CONFIG} \ + ${RESULT} \ + [--out-dir ${OUT_DIR}] \ + [--topk ${TOPK}] \ + [--cfg-options ${CFG_OPTIONS}] +``` + +**所有参数说明**: + +- `config` :配置文件的路径。 +- `result` : `tools/test.py` 的输出结果文件。 +- `--out_dir` :保存结果分析的文件夹路径。 +- `--topk` :分别保存多少张预测成功/失败的图像。如果不指定,默认为 `20`。 +- `--cfg-options`: 额外的配置选项,会被合入配置文件,参考[教程 1:如何编写配置文件](https://mmclassification.readthedocs.io/zh_CN/latest/tutorials/config.html)。 + +```{note} +在 `tools/test.py` 中,我们支持使用 `--out-items` 选项来选择保存哪些结果。为了使用本工具,请确保结果文件中包含 "pred_score"、"pred_label" 和 "pred_class"。 +``` + +**示例**: + +```shell +python tools/analysis_tools/analyze_results.py \ + configs/resnet/resnet50_xxxx.py \ + result.pkl \ + --out_dir results \ + --topk 50 +``` + +## 模型复杂度分析 + +### 计算 FLOPs 和参数量(试验性的) + +我们根据 [flops-counter.pytorch](https://github.com/sovrasov/flops-counter.pytorch) 提供了一个脚本用于计算给定模型的 FLOPs 和参数量。 + +```shell +python tools/analysis_tools/get_flops.py ${CONFIG_FILE} [--shape ${INPUT_SHAPE}] +``` + +**所有参数说明**: + +- `config` :配置文件的路径。 +- `--shape`: 输入尺寸,支持单值或者双值, 如: `--shape 256`、`--shape 224 256`。默认为`224 224`。 + +用户将获得如下结果: + +```text +============================== +Input shape: (3, 224, 224) +Flops: 4.12 GFLOPs +Params: 25.56 M +============================== +``` + +```{warning} +此工具仍处于试验阶段,我们不保证该数字正确无误。您最好将结果用于简单比较,但在技术报告或论文中采用该结果之前,请仔细检查。 +- FLOPs 与输入的尺寸有关,而参数量与输入尺寸无关。默认输入尺寸为 (1, 3, 224, 224) +- 一些运算不会被计入 FLOPs 的统计中,例如 GN 和自定义运算。详细信息请参考 [`mmcv.cnn.get_model_complexity_info()`](https://github.com/open-mmlab/mmcv/blob/master/mmcv/cnn/utils/flops_counter.py) +``` + +## 常见问题 + +- 无 diff --git a/docs_zh-CN/tools/miscellaneous.md b/docs_zh-CN/tools/miscellaneous.md new file mode 100644 index 00000000..a2cb6250 --- /dev/null +++ b/docs_zh-CN/tools/miscellaneous.md @@ -0,0 +1,59 @@ +# 其他工具 + + + +- [打印完整配置](#打印完整配置) +- [检查数据集](#检查数据集) +- [常见问题](#常见问题) + + + +## 打印完整配置 + +`tools/misc/print_config.py` 脚本会解析所有输入变量,并打印完整配置信息。 + +```shell +python tools/misc/print_config.py ${CONFIG} [--cfg-options ${CFG_OPTIONS}] +``` + +**所有参数说明**: + +- `config` :配置文件的路径。 +- `--cfg-options`: 额外的配置选项,会被合入配置文件,参考[教程 1:如何编写配置文件](https://mmclassification.readthedocs.io/zh_CN/latest/tutorials/config.html)。 + +**示例**: + +```shell +python tools/misc/print_config.py configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py +``` + +## 检查数据集 + +`tools/misc/verify_dataset.py` 脚本会检查数据集的所有图片,查看是否有已经损坏的图片。 + +```shell +python tools/print_config.py \ + ${CONFIG} \ + [--out-path ${OUT-PATH}] \ + [--phase ${PHASE}] \ + [--num-process ${NUM-PROCESS}] + [--cfg-options ${CFG_OPTIONS}] +``` + +**所有参数说明**: + +- `config` : 配置文件的路径。 +- `--out-path` : 输出结果路径,默认为 'brokenfiles.log'。 +- `--phase` : 检查哪个阶段的数据集,可用值为 "train" 、"test" 或者 "val", 默认为 "train"。 +- `--num-process` : 指定的进程数,默认为1。 +- `--cfg-options`: 额外的配置选项,会被合入配置文件,参考[教程 1:如何编写配置文件](https://mmclassification.readthedocs.io/zh_CN/latest/tutorials/config.html)。 + +**示例**: + +```shell +python tools/misc/verify_dataset.py configs/t2t_vit/t2t-vit-t-14_8xb64_in1k.py --out-path broken_imgs.log --phase val --num-process 8 +``` + +## 常见问题 + +- 无 diff --git a/tools/analysis_tools/eval_metric.py b/tools/analysis_tools/eval_metric.py index 070136a6..1c95dbc0 100644 --- a/tools/analysis_tools/eval_metric.py +++ b/tools/analysis_tools/eval_metric.py @@ -29,7 +29,7 @@ def parse_args(): 'Note that the quotation marks are necessary and that no white space ' 'is allowed.') parser.add_argument( - '--eval-options', + '--metric-options', nargs='+', action=DictAction, help='custom options for evaluation, the key-value pair in xxx=yyy ' @@ -56,14 +56,14 @@ def main(): dataset = build_dataset(cfg.data.test) pred_score = outputs['class_scores'] - kwargs = {} if args.eval_options is None else args.eval_options eval_kwargs = cfg.get('evaluation', {}).copy() # hard-code way to remove EvalHook args for key in [ 'interval', 'tmpdir', 'start', 'gpu_collect', 'save_best', 'rule' ]: eval_kwargs.pop(key, None) - eval_kwargs.update(dict(metric=args.metrics, **kwargs)) + eval_kwargs.update( + dict(metric=args.metrics, metric_options=args.metric_options)) print(dataset.evaluate(pred_score, **eval_kwargs))

+QtDunL^!o}~Li3Tm=p7HN@`eQq={z2!F*QqIaM4Hvq8N9*64j@{ zafG7DweT*5j1HI!Ox*zW*u-rzYERc5jU~{NXx@Kh7Ls>8=YbIcQ7y~p#A-VGkSm;w zOne5q@S#v)DAs^`5mRgx*TZy0{35m4#x4$0k4+L0*plll(8 zqUTXiG14;;_t}hW8Sb;!1`uWY^?84P7?7v^_vfwfQGwXN&GsD~|9>@e3sjEZyXD-w zSt$pvynh(7No%k{1(FMOOMe|w+|Ky$Uq+?;%a9%&4AA&xRJ^}Ag*4WQzmBToPj9BU zjfHfRpgua7U@OofJ9A+{&~4gA&v z2v=SFOfb=<%68bf851)5QGUvf>@szH>D7wfGN@FaJ#hVyXf!lySR$uTl_6eau+2RZPS=(LL40Cej{yqgbuO=105^F%+1AD`Y8%jZpG zw;1-2Zah|x(YvWI6AugVJgIoAG0x>;-}{DR1*tAIofP z-Ql3uc?-i@lpezALaeF)vehru^ByhCR)JZTIZ9a;p%#_6CMFQ;nyd>Z8ziA5KgtEr zOv%F$;3}|UrJze&3N|iKEX<3a*!PIl;ua zDRz&~TEWt!&X_Q>(8uZ+efFWp2`{uaQjByc^Ebyv(vt_&I#kxL+#`&qG)WsFCGmN2 zt5hIBZ%iHxJt~Vx-=bLbI9X=`hpDu4^_p?W7%?(b+T4Bn1!@G-T8j8ih*3J9k`nP6 zl%W)LX!#1|!rG@2*#<_;!fZHv5nT1UeeUalus0PPE|OkTH}Q4tmg^u~VnhFY7Mf@F z4q1d08eQOYTAAw2%%?@eMD0d7sel(ncW~Pt?;uBOP2+X#D1upa+8T?uD4ogq2DTuA zuopc#o4=6|z`KVqpvd{Ht8#X;#?NIMMeIiC@GyDQUe;mG22qSEcxyCQ;^vMs7 zL!=$Yyo55a4{ZLbxcoHVd@K9|?|2WpPt!0b{ z3`|WdHrk##mzJ&}2z1LiH1ljswqM#QTu2$*r;AMc2(*{GIy`3$Z8U2cM&FCTztaH) z2@#eqt}9({cZ!j+F%-J4D9SY}V~J054UtK%`GYZf#BePT%(=jK^*O&ayYl%2K)Z9> zRP?!4{t=}}^HUpF8>T!_xWus-yB3V=y$spEdc*|vk#`V7K4S!Y`4`hyzo$KGe~1I5 zSOiEXgKsQBmg%#XsDQeC+Qa`Dc13hsOgH$eMD}vj3LTg}W!^yYby}nXtDE}(#P8co zmY-uacUAa3jXm+6GfM_k;7@XL*vDaosA_~iinD@FIgWI))6O|?LArX}x?6LN~P|8Y8+?{t=B{3FGk;1b?}4uo7JXC{NeMIb?LUEfZi zx%X%u;xW!8N;OcsEZalOU9Cl@<=Op|q`6Obl_Tv>;SBb5rbTG2X3Z<(qQ(Sjpner6 z@X~cHEF)L(;yt?q#>$cP8-dF2kai`afzENYM z>=3TlmkVN}Dz~Eax;hOKHFdj(FYY?BM)x2Ug*76{fmKbcpnFnv7e-PSsD*9!?6&2Z$l{VMoZjVxW{gw?K#zZXfUmdvzMReS8pHh9?gdX!lP5>tPDS zDJKf1+8O7-%90qJbc|l2IZ^%ldzyW9LdCeiS+s zWL8gOv_-InH~bF1_AcK*dZ&r+5ewjXTKI_dY6kj;`BMKdK6d532ASq>3`(uu3OVFK z+}g^FQCl5~>B7T9HT`dl_|A|m_8PBdBCns_?Hv$#Eb%7VQ1vviI0n-+bp^6Ju&%M* zM@Bs}G$;v7RizLP=KLBPEk)Og7?iTVrxsd;vXCi8X-KEUwUT*NYy4J*kAUk)_uY)#% z8JPp6w!+J9S6FFJUOTR>t@-5Pi!69~e3*Ixk$*lsJjZAayCAX;uIB#7)zM3AGJm#&vf@{&k>t3kJ>HtJ6-ZQnWXB$?<) z0-ff>D`{95)1We$7k*_%O)%78;-%3|&VN zxRce_(_l}`4xim9dZp}Zs6pD-olTlv?{CrY4dSB+2V+VO6jIbzm@LA(2x-@0*vD#1 zua^kO;5{&aB#wDvVnC3)6mz!oYWh)2=u@^0f&*?$LR`Qr&r(xHH03e?az)@LmeySe z><>?uKz2*)I)Ny6uuN7)fPtY`Ejz~OmS-!Jrf|1d1dqX|nZ=l-%JR|GvK;3dhWDP1m%DwlqRxLq!5Ye_cywC8qreNd zQ5*UU8CXt}SL_*U%k2Z*nAcC$(jf9NL9pGa9243ITzCvh0y5AuK~i=4S~gSVd0G?x zwM_)EhcIXc?cdUvCt`VM8dz=Jy0cB|nbI`}-{&QeH~gk;)nNS^m2NN6-0}O@$4y?z zNa3%3!L}PS^p?xa!EA%V zAm%C_?*C~9w$qboT62#bHx}WSq2+dxOJ0QVT3lcl1iOATOXb2htlN{{stTVr<_}Zr z;Ex)0U9;`U(rQh7ow#9FFw*iZvASKYgFGY(gupUlgd?4DIjedZt>+Z2U44olmSZBt zbU5h!Yd5bM)1>VwChI%m+a(`lrhI3}dWYC({I?skZ$Zd+=2fy#3ApuxiT%Nl*>#-f z`~w@`*h!OK;Qv6!%o}li*w4XFaVJ7RyYWd1wCp*a5RxM9_=cB+adR8JjdqJzg#B{! z2{iZF>|hoC&@h!H)1aS1VGREV=IdWp!WO z@?q5J$aCd;>JPT%&8%sDh&=u?p80+aU(Ej6yUJb?o_-YbZ!yaE=(e%;&qGDPuH&~0 zg}&nj{3fQqEgkrQ0y+=>237uBEzzG)#L>_HjO~otvG>o?8vwm4sd#SeqHpkt{PA+G z7EQ~6xf=ne5=QRt)zgKyzvb?U30Sf5wBzc0YgU?nP6o<9@-;eF>E06Y67o#xPNabQ zJD0Falni;vgfB2Yw1%w2Y4M&n`ojsvkYLyW1I5&L0*@sMWDZ8|V;@cxTy-^&I7pcd z6j}{ccWslY*4?-ll_A!i3zM0l!z@e^Pq^(7tn1py)=Pja4MAT>zCx|fu2?ULaMz;_ zAR)M|Cb>5DIjuo%zkZv$PdiOap&LhiXk3lSFSOxwq~mzxB{J>0A3o8{YpgQ~vdL15 z?#&||!lhK3F_wp$6b5))GAvRrbX(7m9uPkP8CqfbqH(odcz=_iOVDs(@%6`GI342{ z6t)^98NO^UQaa7(tE)VcYS;~H=4=w(YI0L@O%mvLK06PV)vf3-yf2SslFpBs=HUSxUI`3?t_LStb0OWX`y?3~8rAL!80g&jOro)| z`WJp*%; z_Z9JK?PD8xi_z%liaVf(^2BlR^dV?9KhXj~o4gPhTf`$M3t6DaK#S<$m;9WrD$`95 zoQyhKSQIvBnZ3H$X#21WSy_X1InjcRY>l==q=!MbB zi>8Cj2kk681&iJQaRFL0w2;0^YdQxS~RuK=7fNUs`0}(@fJU z%7|ECpJ8uiUk>>sa<v-{I%R@>2`Z4A2qSMvBLXuLmG=J;(oIe#Z1`fI5p|IeJgKWZ5)UE1nB zA-Mfs;!jB`4?3#2g_RF$MMb(AV2=eeoVohxyzb#JxNN?SW|D9Oe*$M(!K#Y*+}2!@ zur9nOSk}Q_z5jSAs&8%Zy3N}{|4gyt2IER67Z)DIIrGOpE(*Rhp5^B(gjOQxS_uT^ zt4~U*^M_mRd3bjCX3YVfN>KsJEAqN_eAwW=+EO&(T7}GX`xlNQGiIzs=90~d????ekH+l;S3Z%fqmfhjgSKe9#@H4(+I1h|6E|1H1L3nK+B&g31*e957 znkMX7d{O*0uS{n{KSj{;b%+`|6@-%?NYH z(E*5(s$>moZndtJ0q%z%AI#qgzaKkDGa$Q99Q&|Ffw@ zA`b$FeRU$+kxeg5N}5zu?KbX*IU***`UTVY82d9VOdU&dv}QO9Igj^W5mgcoN2DP% z0h&i3WoT7H{@)UZ{srhT-!X3fYiLM&xA7k%eyCRd8{z01>FA$c-)~IGVuSDZ$Rp{$ zPp!&NxwQpgvh=4Xp$z^?!uknW=)2kZ$>+`Z3h#76LbZ{fUzx(VhjPuszQ`8#%<{x4 zJJFWF*s%I2&6Yi<=J|vBdei}1Em&+9(vqBul*U8rxS(FpyICv*bd5BI#VOdnv2=R8g;_wZ2AIhJ0O4L4fhq_~&H^Cjl|W+W3kWcM^e8r*D8v~k#eU$@XB>;K{oxBKz6U5Mhz8s zsz{fOI}vE*I<6or6yKQv0{Y#2dRLg=9eM1yOW_@D&L4Z|bh-w?5cUoAcK?QYTY{)J zyE5ry7KnN)OoOQR84&djhy!U{VBTLJgyVTvQKUoPI-!3W&pN*3lc$BiqG<^sSGwgd zq)yyTh|<)^9XLYllitfc;g2faNW3e&cH9+j@cx{vScZP`fbF0gf8?V2blA|kN`8k= zb#J}ijnoX}s*0#F;&xb1Vc{zqg)DB8<_BCw0>sBjLD&OPtAE-xvV~cnt$To!&S-N^5TZ#RA!@=Dr6+kZngHLT@Oyb~Ey6 z>!pztN(hFdJZC_p1$%oY#y15eJs|n!)|{4KYsKpzj=NIq=V^2qngVt_i@<gakys%E|5)&crZ&rDS^dY!)!N>HcOZfBTLvZq%)t1SgBcilEfoF+>OU%Kf2eCe zTlF71MkDcknWR341)jFjf1IqkMzTuUb-VL!r;Iwu-7r!utn%>a^Fxku_PqJI|8vIo zFFWynv}4R8|}#i#YDGu z1{)<1uh^#8eo$&)9BqK!uGm6;&@Uj#Ob^Ulf3IV}Qysq3FPij2>>x_|{XTrRz4JG~ z6Mgf zc6A@PAVbI7P1CRS*E;>GL;o8DXot->R)7oa3j?Tjsd15_XL1mMfSA;gs|o~%ty|GTB3H#_|-!5tR811 zGwkRoQ@h)Bdexw(IeQ)Geete0LqsPqd|92M60Mtc@rmpEMkUugj8uqHeA#u>U?J8l zO{joifrYTU^+_bJEh-n(6-2>3T zRfX0f4K7L-CU>c3L4yb$bipBUxkqL&zJDk%ZKGks?&io@&t32!SU(k5zuXNVcnHyD z<{+MDgoKZmX6VT@9 z?MF&GM#XEv0>eqPuTIBy1SuGY6T<>X*Jy~>hiAthdZF&L`EQLCZlyB$nRiAsSbXD4 zEEpY{M`K{$^b8)qUlJG;$atUz70xM0t|lEqQlHEgIS|E*e_p@ zum|ruMHbg2=OIx%9kUWc95s03tegh1Xb}tYl*N)M=J)is*4k{^XpqIsF3S6Bif@%j zjxWls#5?K^Um&pK<6&P+C+q^fFaKpi@&w|K&$Y8v2=t>lWf$PNz-dEuUu07Tyvidg7FcWXqR3=N~JvA_i0!6a$GTNe2jNqahrm z^Mncf;!;gbJ&f5IE@Z!o+5M%>A;m^YN-xk34nMrXtaQ8vl^HQ%J-#zV_y8Rp@i-fe z4}LFg-1%Kp;AQJCl|oZd5Yh^)DO+(H^uEK@0VHfolHM(kg9K{$rgq2qy^y8 z*Fbg!SG^y0_=zghChgz@kkPDW?D$(rf*;%2J<&h~VoQ4ptNs4lh))jw=nQytD=|z$?E!Ki!{rzTWyv?~#)gODVRDq{M!g)tlduI!g zMZX*l=%2FfD7Htezf7p|FMWv&05pE-4(~60F%qQtdv|^vRH}6x-LLDb_3NS}#s60~ zWA2h8K^eyNeT05JL&nUsP8Oyy)nS%-aeZW8u>S4pqBSkzf5brkF|A?BkjRlzSc$uy zbd5nfNv_@h0=Qo8_9Lr}aQmOp^!8=_E3V8x2K1wx?B(9o^{t3O3f%GrAU*qA5yP({ zlP)5Cwc^-p=T)q(lN!sX27pOwDP}?vJwRNCcjT+&NfL^aXZ3rRa>kl#=1!zYXprmG zBqAa({d|f&8N%#y1*ug{L*b~XISa1OL5e&AdbZMwoy5$wO_SAM_tUQ(Tkaz2P7kCs zk7_)Iv69YDE$%AY?VOxO=I_~=?y@lya0NfLU@_MA#YKPggb&twBx55Osh8V)fC^-p zDjYXtDjd*6B;h? zE(Lj!;Ga0t*GDTAqjR0Vyco1edL4@~Fv1S5NsakUJa;`tx*jZ$sx^8cPAR;MbF!#e z-#06LzRgYfE_XWfKnZxR17v3fEAnVz~opq^^(kn~1!6z}a8N>?uH zoN{iJHeqs-sAPy5)g~X~?c&iNef)Sy07S(eHLQg;ph0Bp0LgqA$wiJXQ-4=a1!mV! zoNTzYsBDpYT&r$)TNe^YNj3BAt~j?@hQK#m#RaE1cadn~`1^$AyxrgV>ypnDWQWeh z#%3r%jGsV#uAg4=R3?0~dJ&WQ>Px@Nkv!M*DP>yIIa)ETQL=GtPYKwlu*{PGm za($W3S5?;Yqg!%3SdOy~yXas?Ks;@Su}5GfWoJj8wL^%AYS*zs6SrHEI|>BTXUFim z3C%JErKW?69iH@gA6^=c^e}b^=+!qR92S*r z)rHGx`@D`B8d2h;0&ePtPAaMzSNo@tqV4%}3u-!q)NuvL^n6iQ^JiV^FD6zO6pz$% z9$sD9BOliIj8>)a&7;`X01*@3gPiiRNe71)3d<6QCIlu2D`G@hI`|Ho3A;2cCaByE zGZsv;u5;zj$js7Py9*-ovoeo=qh$F{%MWZjMeEBpv?y+0)9y%$YA|CYIAAx+^zpQH1NQ#O#-N7mmV|c~`Y6ymJaT)TNBy1<@iu91 zG+hB=ocra=J2LYW&bbxkBl92e{p`*PmxPWyyO6=Olky3|ND>*_g>=X?8Ybk{wjZt= zLvarCTwG@TQdQ>q;1LygFmw!NyDIyzQm3;M{bp0Y(MV&dFfQzqFx;i4>C4jO68i?M zd%1?FV>sv6NU+3-jA{5M$?uRqDi(S`ad~#K<0?;65W915T{FnCW~S2A^ZCjuN`$(;7NI$Du-7Z+p?`= z_WNPqi19GhxJf3XE$vHh=4KZf#0R+v3j$1W=?mW<-VqUDa{X9|YT_%k6s75iuuVSO zb0$(eh1YjWm(G%nOHoo_UFl8xe_mXw1hD^(Hu=Zl9$`Os;D7hr(zXYkNA~Pr9b!C$&_0E@up>|7{AcU*rx6VPyU%fPO->`;qI{C&&2mn#9pRF&=Fcv&7F*=iBT6`CjRbw!xm+lO zjq<$l^)Pe5v+?x~a>Qlk3O{^b4dqDeJwGU{dTvGN0)e2J4^QFklb@` zfav?9W>UsRolYIn7L1zhZ9!kqnAHiv z3vg~VN^dAf2hO!e;jlt(N=X~d5xl7e!zT&BxQ$9pa!P}lTt}V(9(`7?=U9P>+vFAY zYNmd^$}KEWxef+#8LZDmS2!=RYwRmt1GAYqQtOz_Jp%EuaIKJLVekHJv z$=j3v(%4-3f^T!x!JN4>pEfI%h{rz0O$B8$%7y_a-UefAQGv_5$HGRi*PnP=_rw?W zNNzURRTcL46Ens9*DjfT2u-}o=fC59knp?89g*nXQ>z$n4YOrim}5@JaH4qHEn6zU zmHa0Dwdck|@#5NL@ZZLn>T6Eb!SXihL~o$WgWdWx7{EfN^#TG(5QLp%3F3~>G53e# zb|&a@axPxZv;?#()_J6h)*7f)YoX zQbeTJpooBo2uK&AA~ix(dX0+o5(NbXBmx2=B`Cc{x`1>c5=sbIXn}+pNQnQZ&YU|i zo*B=%p8wu^jTZUP=v<%eXyEd9(p0L!(S)NgPAl{rx3TyYEG|T^#7My`-iBTkELM z`yN6#@)C;zwXq0+}wpfKMP*9xR=`7)5iuA^YyzEQ`yiT?OpCEO{^$7Fx^a zHvuX;Wl|%bBoyf&9-D0?E;7hnpKMtc&97f8cKOKCrU12lX`ju(wzB}2b`&!V@ZIZg zr_1OM&~!+HX%%rEvhg*VY+QHyYq2*(Zbs`(XUayG^XQiI^Aa#LU@Kv*fReY-?XAm^ z$GUloW9+$K*LEa4%ut_vhtAm7>VFe;)Gs z`7XS4&MN_)sdZ7vdfjB`WvLiFl(LBI&A?C&Y4{>_6kya-X)=X!uPP5wQQ;A&*8?ee z0KB?3e18lb`s^HYvnb2D03s=--Yhf<3`#jOgdjfyl)c~67I3uvKiSgyd^@Mj=p#;L zzf3D$aSQ8lS^LNKCN?+C4r@0rdog_A7Pv4MF+2+7j@Ax2xr2b=+nT7Y>$UG4yZ_-0 z<~rRNzeV|Z)7~A4 zm*+QiFv2u00?Cb9z;L_P9Qek4?h`2TGtpx0ELP*aXAGu^sW8%gWzaNXIE2yzCgsh8 zmq~zY_oL6QyXOGj<-d1JF7d(jdRrZ2TYa7(9Qu!AykU_{L=HdgaL#5JzQ}H>mSCR* zm{nMJUD+)eCl}Q@N?*}8H%v9pDmZ{ofzO_9PIHttn$XtlNn0-PcHBC%DSQNOan2U>TQ{`Xtd{ z?s|^}O}x0%%@RlN!+_Ah{qeS@1y)(F#DBbke|XT)C`#|RKw)vTGNQng#T!bo$MZO1 zBmxkZXF?T2Anrb|qz7dsr)y>x4}n%r>J_?Yk{N0{A4jPunHlK^>JwUCf`Ix0=P>I= zZeRxAmcit#2%Jm$1G0pQz zp%d~+U&_s7%h=7AONB+>zhilD;ch^iR6ry!X8wbY1m8DGe&~T`Eye+OL9E@rjOC(y zT~Y?e^6EKbv$ytS?|mpOTEJ~l@?f^}X5vmpgR5J#H{MV(|6vEP(r?n29}3rrZ)&KW z;%Qg^1UmZV@SXAF^4}&Xe}F@>7)RCd6^xoFhwI#hyg=@dapre3KzG_PCKqfT^Xq`m_*?i|wGnbuV@KfcNxMHXCXN zoVnHZ;kAR^DY=Ll=W#C=(dXrC4@*Eb*D6M}9%~>tBu=b649$$=_7skK}UMWx4_+%>M#E`&#enA5&~x2ZIYmA4v?O zL*M856-$kEoA;r#KR|^bG{4@ml@V3>PoRvq6j)6qqL5~n!7s5CJEZPgJ6_b2PmG_F z$~>i@;l`B}9=7o4F=kIvtS)zp%;R?j1_u6jD)0mbbXP&Y6R%EDmsrdd*`o0sqGQjJ z)o6jHfFvN5yGBIPSVz6>UYde}_F!I0*rAZ$nC##B`adKZc&94l>qXHgMZx@(8io#s zshpW-&#*T>(&Uk8^$CRCuEqHwZsMuhw1%akk*HpDqJ>{Q3)yZ4YNWb|D$t zX=RM!#@Hh{_Q$=1zDTQ2JnFK!J&I=GlpbXD`ci>K(^(JPSkBu(<;^_8L=+nEGMK58 z2e8@}M0=MtEnmj_CD+8(L76TLM$YVy}UdefHQe&~XNc+Fxi# zd*7Q-GSQUO#=FUz#guJEwmE0JO!PXuz^PA@WtFUjkQZ+w>wzOJAp%9e-yG0CDoE-q zrlbL8VNd3cI+-k%Q3Q9VsRj(UTI-XO(wsjiO!;n4J?}_A9q>#?=7sbTxu>xw>W^b~ zLmZH-wxiXk7f+03KdzdQ+oPQ3mo4xu&x5G;i+&W z<|Q}>l+%F^>@ZS(*#g#M1mYP#`9U#zNrLl|Dir^C_yl zt67!@Kh7dE;i|vT*EoEee;oMxid`7rT$G4%iOWX4=QDimI`bAVmNVX9QmJq!j8M{v)oBO{M>eYppUYnQc$Fc&LP+kAx4f5{CUPx`y z&?9Ts`B*b&TflSfz)(dt79Yt3X*UMDT08~rmAAbgLQ6$@5V48$Wx>sW)n0yaN|DVnD3$qOjv%s-B^o97xeyqq@YyiM9g&fc^Sv^j(rFmzGLb{d z77k#v+slN=a|cC?63k5d-sLxnw-`yW37lOy?J;ddxT>Vd%qHAg?n?bg3&=-MX5j zya7N2htkr|SYsuEo{70WlEKcXEu|}Lic_fwx!a6dNZ)@<(vsd(CY$DrJA)?Ij@W#- zFS~n5tCSoO$vNOR;w8IGU^%{9L*O#|tz+|u9iGryeXDVR>Fv;6G?O`Bi`5KY%cl)^#Ay=x{$q0=Le9+!aAYUFu!kl? z5hi63HkmLGu(!nK$7d|ulb?dRhOCO0GD`0hbQf1dHI&7Rh*(R?TzvK3e$iNGS{wTD z`uK3(n~b(pFAt}t)mbO>eSiayYyI?F$C`hUXz#zdZ2Zn>eLdmhkIa?d=mkHHKJXpI zK-OMf{-BC1{<5h4gTl@dWB`gR*`w?*lG?tAjBq{>iFsQX#H-A5VR06m39R_+R0S5f z;i}+g#X~)p5)ALGRKDpt|D1L@y&_w0M#@(i5-TH2Ik_Du{A-sPygFnQf|@&(h}Z*I zq>mpYK1~T*&=dRC82J6O?>*+uDq+IFT=XNJZ9q|r)r3OG&bRie`{H#hE$Wj5I_=(w zFYG&J0qZl@WNt0B3Ev@DGIUNmq{l2)H=(n(t#YbM$9-O>ShK20pr4ouW1@tdec@2F zsAEhWcYD!O)MdG#z4D&(J?6@4;rz@10BEX^icQ@O~hg`ZHIq3?j9ga1k-W&&M8V0m8 zJk5D6cfO6Hc=E#vX@@;m<}4w#pGzO8vMOUCEUMt~6}R_Tn*T(Se(d7ZN~_B+4(Y1X zQ}&CVB8a^M6)D*m(jou5EDljwB*U|FG39{KiS^2>4-i7C@5K>R3;{gVGwNi+yK0q& z#x*`R8-0G!x5+Jl-db*mA;A4RaUQfF%w>!IY`+ zpJsWL$}J8012lb|9<0Y~1RFK`6G*BVI6LRk!S;+Tl+t0KnvCbRMtoeo|GqnB8M%{_ zhB6Omw%LB_4Llh{4?l_9{_LL77J5M)T9?RNkp)@NXmlQ7O1ke3nT6Eq8Qd#>hB&K9 zy!SRGY-`=whNodio~45PIrZxZnLhL5IKRWwaq+rVtGPIfi7p}2^SaRMIXbmDsm=Hp z2V8qjG~9{Zj$s7S2VL|f%c3&O&eO!Y#h!E$*{~}A6g`zY23woxc{f=4h3r-M6CVc{#+)o&)JQEf&`rN^+ZIap> zN57W_zaJjUz<9{=cssQiD5Eusc+@W5jj(OOuzH4d*P*~wLROxnaf9qKp%}09T8DJp z>x={Mi7Cs@!i>{Rhc5DMh}3`4_~vQKi=2=w%UqA5mJ@L)#yah~-UGNOzNmew8dWpK z_$&{2i>0FJA=z9a^`u(EwYEd{*}AvZWBEYtbRC)Z`rlz=ZCv+!)3=<#9lyz*E&EP- z)sgwNGE3t>6xaa$ekJ&O->)}`|DP|fAzF(WPQ~x65~5kHMk$+-@(IGVqa%-?b?oD@ zQua{XF7uq8fC=N2Wpj(5b4FA!Zd;r77z2A4Xo*L34yr+ldkHx0y`=-|<}FXpghgJG zT}`-bRXJI%4!+4#%|bVsQq?EjfAfj9ScgSFZ%M38m2Gi6XTM;4^gj2P1r{26aBGJu zn5ItAuy#=zgR;;iVRSGl(S(advap+Vq$H@vvdMby7Q(QaHty~LSA8`;>^iR>fJYk| z-NYEg=9!aZt81ZW6Lge8f$J(Q^%{e4h5*Yo14NA(8qq1!i&#uW}SLDVw3a zoB(tCexTao`^wJ^e$j)oPa4xVbv_O!&~p@p?mK5m8=7yvvlaV+H-nK8y&vodnsA%^ z1e&Vcxnw8|wk?Q0{|QtED9%rPtYR=l*!b=wImer9!Pk`eUiJyMug=DY56SgT0cAN- zR`h~#6{0M`@XHQKFx}@j86GvV$K1r;(6y&%p2xZ(w~c$Br8s4eV9(t8B{Z8n+mK54 zp_;^z?UL?c=AU6!GskHP!a309IG|dMTLF;5TK9K?NwF~eysjIX3J@!R6B_3P-^H(a ztTj;s5@PEj_BA2)7p-ypniKm`ZMnq9A!npv+X^kAFUuaKwEG?~g9wqaV7X*O(V6J=2#t|O3$BGKBFvBVWYce>W1w`ijb7&pCCGl>HGw?KDG`=u@+8G{b= zSVR3*t}OBMedMELClEarZ>^?WK8IlN%zCNH@Q{Idwf;OHJZDM>G=}|MRO@aaHbWUZ#{Ap7Z ztJ%aNWcq=OLawX?^+Ntc0Pd6rND9!cq)Xh zl}607cqReV=mHc%U2sY=b+3CHAhCJLkBF=IY@-w#N;UJ|Hk5^WN7FK6>=54^IxX%*0ItORa@8P6exJgCtHCWCW8D(%utA7GI=IzyN$ zwo0d3VM^MuuOH>>5BN8LH(pW0P`pyCC{iZj2~LO_^Lw#z*H9U+AwhSW1T zYD8{`;V9xE!y^bQ|SC>h*%cs4IuBJq>thTQv>0)6w1m}8e*&>A*uVD9f5G4C@Q|g?c}q{-JG)IuSC3>Q4)5C8` zb}s#)Ex?2ZAfv47v-LCbE0yYJOMe-sZ;7^?|k!mO_Su>;cwxZWowMjuhS9C7EUJwxI5{Zl}O z{4e*A{%>429ilz+t!De_;zju#fM&o4IK=$O)b}3p+_1?lh&?_i4zfepOE%TXN&J1< zLTp;9RRU~KcBOYjDcqvmOHK8Br!I%{lUqkiE;OG4L{OmRW++7nRV$j2(4dHHz=s0D zJd{;YPco%c2z|M1wIFt}8>g|!N36HW={A>~WCG&Fg9IX*l&Hdx3T#;TC1x|d4YTtX ziqmlTqm$FW__-mTPT+bz;v@=nsF_0}A7($CXF2L%b9XRg!^z{%o;`ay1=Mx;(WM(D zAr5R7!@}V?=jPlV512rWX1^pqJbXY1beHa_yuSBkgqm1*gMUzAEQr_-+V! zg_zZxHe+Xma&*PwMWR&{tEP1|JTmoHJ2Q5DN#?x#TQa8v843aWa+2`SfND+MUf_7g zI%w|q3}2hX-6c?2HY?}E)>O>ef6|nb?F>0}mzqF8)b)v_#T-f@c+m+Nd_|SZelGF6 zqlb5dVp19Z^z&nEg^wzNAKy(Y*&aj9zX8kTR(uhbh>L)lDLK3=i~-b8O@_Ruq^{m_ zi5}W$TgAu6Q}C=H*|87 zE#WozfGQA0DNQfKQ&h_}X$1~3?twe=jZh6y+*0QUv`)uBt~{6DdyomVPs1?tRyvW# z+5fr|ls*{M((RcF5e&cx9LG-z~jh~Um3hF4NL06}S zs~5VXihKMB%t{%XV%s_YFfl}Im=QL?6;V+S+rtm3=O2-7kp&!R!FyyTSy(fwF}GH9 ztDcXBx7Z{db1R=xWX@tsSzM00^CW0?*g)XXy+GWvSs`>a;9JNeTm$26!S)<>L$ksR zvgv~4GwB?I6EQb$SlHKH+b?Jxc;qgcNkL%AFinqTv+;|{g$SSJs!p2Cix(A9Oofh@ zgvL7Di=yM;2bbaq`vwP7B&yK?Wp*`|eA7CSR(`NL#h#btWX(|Yo$ZkZlg@IxnwzLpL;Ye008sVEqYnrk^@-4)GY^ z+)N~0FhS9rDCJq6vKB1CkR9`{WZ?TeBQ0I`$-1j*Wb52Koa}CLuQ2$kFt@|rC)#uJ z{G~lIx=JUy54+`9Eh@ouwD^3oLK8(acn;aR3qJ<2=@ohfPZg$EQi3&mAodszhn(H{ z13Trkt2m`Q;rcHy1sQ&Sg?c|l)+}$JcE`sac;1WAmv%> zKyIc`?q%ypfWfS9>BM28aw;ZAzs;HW2?WL3q$nOT$T%5)n=_h@Aq)`*D?f9Nw%pYC zqiS}`A953aNKAf_#Qv2B_U83eil1)UIhaA2wF!a3!f%njI<^e{W=Z}Je>I7u52E(c z>`1)BdsncvkG5NsVhab2`_;Q;#JOrb%CL*P=VJjb1k7efzBXrtjzIZwR)O%nC(sW| zKps3xjqNx6caxIPqj+|j11a`VvM0hI<_xwl3pmE&fQ;P`|7IUpXp-8U=IrJj&gvUq zvbMrKVRb{iIeWwGXk(gIxd4w_%&7zFM>Ws+^r=;N>VCprAxg42e!{93R%*BUT~z zq-kqcPMv{H_N$1N_qp5joz>)tF-Su-W+1PmwVGNm1b9G^be9Mpt;`W*WLLT4-1hA= z-~pgSb3Qt#AVwx5z?JmM?V;?mRu8X$@~P#fBT15Pue&tbttONC>4BD9pFoTN$UQ6L zWoC>qjq?U*0$TLojECb>{J8s&#w2Lwmo$yNuNGizY@z|kJf|?UNK*uBmB=KNNqL{w zD0;7hqIf#WHmf8zUpZQr|A;9eOr}s=Ecd-W|5>xXTQ{`PR=XZ)KdrZ{y!-yl)eJ_3 za>1-CA2AMbh>)`2V1(^Hg(g0!QXeG86l9_T;;4^i@ZL5!oWEp|!9UHjaQj5$rfZJ@ zS&0!h$lD}}2Hcn$CfnI35qwn#C@k8~dujS`82n;Nd#*yviLT)to#sknJJM~V&C>5> z_LpfKD>;$v2d1izXs$AqNzhqr0Ib+?qW;to{bVk;eZ>42d@oJ|BJ9Lh@j$EjdhCHy zZ9c+fVH7`!?D#p89*1xJ@tTXlb;uP?S166I<#Ts6xF-Ojt+fx|Al60f3m)`;?VT;a z0alRT=o2XK*E7dRHoC$WkRd91!nLLL3BA_d(|ehT>vbtk-47)YQ_j!8y*H7ZzkX-| zrVXYD9GDFiAarpC<2JG;Va|8kcB zWMV(x`IBAvJL2;VJ=*;xWH7Q&SdV#J$offND8J(Co5z>ql(gXa@1i{}U$AC~Ns>hr zs0tuDvH*wX;YBF@A*x~&KD^Of(6%X<8M-DET!Qm7)&jpc>6Gmjrc;d(bod%up* z@yiuY-7;|10+jSJCLrQVYVwE7h3`ALy#XlBnAtP)U@AjDVk`~t*8fi8*SE@hD5L99 z&o!r7M-faD{yWJmwGVl^jmb}{4woc#TUHg72Uy&3@|lD^wbLdFir#T^X}k~rmcsE?IqJ_z|Dz) zW&P=QUFq(DFZzM+C%o;@+lqMN>q^e~2o*imNpzu)mB)dXc^7hVC8!|GIGBxoz{-u4 za;xnPcuBfkTM{@xXBD9(gt6NWd1L~|jrV5g)q?psKgnxIdjsh5831f{W+J0+drPAV zgd;xFF9N;~ zeN;pG0MzRC%(r^`JPM2Q^DT$jBNk&ti*kjBAu|hs9PcIPq7@&->9OO zWzu2o*K`s&D}c&cN~081X|@mhQQC^#i`Ue#N>qx*~p!#n-`(f6N>s;pk7nFAT(m_99 z%g_tn`J5)-@y_K&nap$H4lf-wZcKh-J1MDo4oIpppPW>!a+;t0_z48g4QL!XoJV53 z>4xTWeBu{(qRSaRef=7>o5O9{p^!$GB__i42ZlVUo*09BW=7#CZF z^(osB!}80BJ&t8|K^4zw`i@*3xOU1`frrQ!@gsz=Q$?1yZWrA}@xejr2Cvt)#XWEQ=AUJOwdEmn6$(wd#tQkzPsa6s(Wh z1~nsYhQ}NxY3A8kz4tXM5)eM{1?3fv)Z@1uH0_^My^9+IEcGrs{zv%g{ShqQ@H+tc z#{tJQfai2)2Y8XE8#Dic2XHkFLCkIlpx2sv&|}pBz(rpJ^f})Dty+NJ?}rPN3^N?b zsKwv10==m^z$%KLkh2I&|7hBI{HfUV^jJvPcBQgsw_a};6Af8?Oc(a$ zqZ}hGLiYK(TiB4Mcx6~~n!aQk1B#jkKXIorHCz!60txF}uFxbZG`;Kt755|o4B zsW{=P(9siXCl}`t*U_GC&U|~?YcIx_fex=cDC%j`FphxMz#tmQZPc z|9#otm^Y1>j&;zFe#5EP9{!xUT{_TBd_d?|JqB)pF#HFiik zKA*=)DfKCd9+>WFq4u2A77BmY=g@o29CzY;(W_LokW%HugABs)bBIU;*}&I>qB9J- zerHlKF7)syxRLpOcz>>6U_*|QIyNb{4dy>|cZ|NaQyh49X4)kZ z?|1$k9(}kfqX>ZG0QCWY$2kIcoCbi$33RTa05~`(2Y`b&831sw?)68ufPi8BQ}DW9 z14^C#$G@~8V_jW?k}Od$@Smerl0esqTJ8L!TCNUE_va3~-@w?v+CSaY)TYU>7oca_ z(LQ79S)Va=4h${kD?*F_pkIE9|9Xmifo<1Kh(FAu zpD*ctUkLt=5bB>=k`FQzd?sG5T}uD36?`H1|9t1~Zk68?AE0}d5AmQ^Z(c#nI}JLU zaN1apj*b$w+3Vu99yrL_YVVBv#h}Wjt%HlWXVbnu1%+3`){juY;s+#8W$ZDO13>aL z+Ws@|pWgmCd8%$44ru%i0Xr}9=S%-=2-XSm*KgXra#2@&edEQ1=IY{gG8PMOZ0CzS za0d?ycW5e6-crq9vgxwj{;$;KA22(<&07XYI&8{C?O_GZ!SoA&d}vk|LO|1EDjDf7 z7x62H?18$MzjGM)N4|dsE87`761jV()?cVIM`oWQeQ>xc4^wQGToM|@qrh5iVz6zq z*8TwFvBx`kx)(35#%HVU)*~HlT=XJIt8Od1*WxxjRv~CxcCO7-(zbxrOUy$g@qU|{ zO4c(HCa^+s7|CKr(X960o7W}%NOAB&_k88yR&gTnl73*6{OAUKwXw$_k=1{R5topZy0_IoN|LUlfS+m*piEc#S056}d38(1-}k2g1iXE@I*V9qiK(>FoL}@5 zt8TBH<-=UDDQq15XiY}zgAOvKf?PM2^F8$S{Q#7HNn1(OU&`O(b%mBOWHdA z0s;KVB>l-GrKUgk=K*e7N*#zP7C>;H5_>})iJMuKBBX~Cs<9@NNCM25?|Gzio0;Y) zhk;3^Ni4k+KTemTXcLm#`{}}({Aj8y&N$5j^%&+#(?T2MHsdag<M|c9o<#MI<#Wji8bB^RdQpyNikz6^$O2RA4GfOQLYST|5%ZcCr%zn{VDO~* zo!F_IU*y%*_PU}-28bZzOCM|E8>^uqy^vkW*s1w&`0XVg-53})-_kG4MfZ?0Ni6>Tn#FiSrrQ^ z7DZ7Wa>P(B?hkuUueM+M zQ{%l9izi=qOA4l&erfD74MJ)tyf&=M<5{)ev4elycq*@+IoY&2(qF%9<5k7$+Dngj zfWrWfsMQQr4*R$5eHXC{SG>K9pPK`T4B_V#Y^F(=qZM~=p+vvb$zsSZ8CxeXA@<3L zHFNqqs%`S6Bamwm;O8EDmOc3sR}ky|6!%L787EZ$7gazxa<~3=Z<>Qg0*!w&{WDSr z?0`uCB~(N7v(?CM1P!UV?w7AUO8=hMzAIe+z72f?%#+wLsXIzMHQ=k(6ObEY+bSQ_ z?Ghl^#z7zD1`01Q%%k}-oE!;e7_!y#LFGy_#L#l6b8ta%%6~|?-bcZ;} z$savW(3_ULdFvzyBH5&DZTE9=`&VrM>(RWm^I#!z6t!dWtbJYuW|xe^&7N!yfjcL05MS?8%v7eWB zQIzppm8U9-hvR6c{gr$#V--V`_0z^r4W;bc-8_t}M)Q(ez3;nQ7vyj+MsU$jp(d_f zkV!q{aSu;Fjiuv&zu?-VZLx~b$=9t#WMQcvv_~L1Q&IzOUnCIvf&6hrYS7+T& z?4AKe;;RIY?4bdM2!XYFTaw!{9+YGR?=kxaZ}$PX>s+4LrkGnlt{*YAZJskbM%hJoo?vpri|~;vuU( zl1a*P8Yap+_QtVtMDrFEpM1DSPo(DfaKNhrl}urx<_nhIbI+Dy;4@D2^mB(aS&nnW zc53@INpt{!*sC|+;)wqK*^fesb^{lGpIYc54`P+S^e12RU&Q$SjaB=%ui1K`U{>FK zq?niS8FlCop3>)?1+bH)Zw?1m#2h^dDc|eZ&d;W{HOvffn$+Sf2CC+uhtZTnwrOM& z0ywC`gw(Ql{mnaaTqIp^vT7FzeN$N{`xD6aRAict=i%JKZ4qp|oLX0&c=5BI9_N3e zrl6Gd7Go=c{J0oQ+PoAZ1Kr#TRO(bCb1vQ-X=iTtj?>hAlOxb408`v-Hew^$viGKm z>-A)jC-=dEp2CDrpn75mdKb3SZA7{Z%aKLSU!Sqc)Q%~)u0l( zSW+xYI(B}?qn8nGtv0Kv!!xTK>40xr7lIB^+gHh1=eOpOCQfxH0Aihj)5L{=M9ch! zer@_UcEwvuU`F@7zS_>Hc?ICm);K9wEkzUIW9~YcMg~=1l^#>3APC|~65@+Ha;uMQ z3ux%rq6o;P4~LDYc9J~kVTK5T0|cPxrV_!7fZ|1tzB>l+gI)m<$8!1CK3e%Fb*-JTo35IF^1xl66CsPRLyE2qpl0oSp}ScY5q5qyhTISJzM`P|G~|y*AdjM z3)PDXbx2KWmVHq7$cAXi%wimr&enu#(aYBjYI-oWTohRg_Sv(Xr>1iB=lK3@eE*2K zj{AnOdNx-HMJaD|M8_52QMLi=pjp*66yYq%Ia)>{WXtxO8OJ`>0*2C8$X@U~8~1HD zPV>p$knHBzFz#x?x`+(G%o`*C1!n*-ow5qHrzipF`5_Fc0LTa+I~d^_*VaAjT=y#= z_xYdA{2H%6#KsA@{m&Z6K&czwegcj4mkpq$3Qz3c7;MszTy>;H)0e&6B4hmg!j=nX)w`~dmv zXR#&g{qq;^v5sl*okz^IB1Fs&Zcr`(jH~16=e(`~)!Jofnf8x2&5nKoCBc^U-!}rr zQP^2DXth|TefQ*$B1MKOOL1xB!3ItF3sM#*uyre#V3|W=Xn-;5d(#3qYli`ttNY+LYW+V}}Gcd>V)^hGldUtx)Jken5 z^kMMFBrQw<4_LCvUF%M{)a5%YKI=Y7TFMRf79`yZ{a!N;Oq+u?$A*FQ4nvkdVDA4T zf6%|InF{sAMv^Kj%gl_XLn)=|PC{8!IX&f0yn)j$f`gUST~Y>y22lIWTZEEf2TtEP ze(U2B>mA!z)BF;l2GHWnn!kkBz4~ZD4~JfSO1U`NG82vgs|tBS%z?6s&24ipTWoTg z(k)xAx#AuH?xWf2yUtZ6zST&361&9IxrK{2b?6>T%h z(B>xSbBRqgKijt7|K0;=NQ$TcKPl53QRK)skw8=cuji910$ znVK@ew;MyY&6iDkYL0U>7Sha*o_2R|-mO6V;Ob!ZWV^gUod~w$&HrHF{;6&>`3R}k z!4|9x)~9Nvy}}q+E%cAuKmpB^aBDV@m)Q9 zrx~LU1-{uabNv~B5kLW54MbJcvwzfN%4--2_SNn=nLNVyyd!^c^ZNEdrtOKKBL^AY(*AOX_?IoH|J3r1P6VWG&v0nsA;QdC#E4ut1gNhbYVHv* z?JBWhvaG6bgzJ`>47U?edVG4fD04&>$E@;2)sEH&M&T*>fuirLl%Zjm0%#;%)WV$J zVI@zxOHn5!94jMTr0ve<4?3*nx^P(2*01$NOl<2>s&PM67&B`^Hme?ULk1Nso61 zuiq5mU#CkVNWV}GpoD&(?)z^D)t|6L8D1kC+WS79qs*jxLd{5ebrO^l71qO0_Vh}i zP#2qQbTQfD{A#?9c1qIQraej9a&mJIxSH|96OWd?2r!_(hUOF7r&b9n&Md5+6M7t; zsQRIUWn``h<&(JUeQAryM>Td|DmJ;cNwRu#1#5%lD=ib3a<;}U`#&ky`8!s8mB%Uo zH(Be2qY5SJpuu!;5~>l-Y>yMZDPj8tL)by~s2yKul{?p?s_*2ZsGivehh5;VLsoLW zlepC!=pT}DL-jnleM*?B?)hQKf-FMEme~g8`W_=i`14Sndk|)VhAgYl4m*kwwk6#o zC2xeBj*`XE0>x6waat6U*?Z3dS2U}WSb@Sy6o@tne0yqhZd{5(o~c98GwogdwLFbH z8@=R>7)UuH-=*ni*gj%X3l4_vQD*leVMp3;7NFTEK8ELbqC#^`HCg@&RwNzd%0L9m67)YaZ7Z_o6!tCsDA+@3CIj4~|P zvfX9#>rQ^%o%}mX&rjE)-*TK~jP-AY<`;*k?tpq?8u4s&!HfuB_-N}TUZ7_u7jgT$ zo!l+@HJzMbPL6^PsyFFt-r+IZm!!Qfar)(g;_tfQ3&(I7^%7NxB%;Wub7Vh!HM|c@ z=$Q^6Ai*#zu7I;-a6{Ih>I8XyvQON~y*OO0#5CZ+(&jlRXjEo$g^MUog}|!hiz#{z zV@i4cR)(YglG?L-Cmq{zd<3_X9P@iAgNccctlCJ@ATCprDH}u zfi_YuQO$aMS*%AA;KYMnxGndx^Rkrlw!E|??r9jl>b;9aH>ILw(PuTspV7b5YI`Bp zcOO~3CT5b=)4Tl~84@(#&e3mmp5oiSvH4QjxeMegVkWL34{>K0PmR%c`?^pB$=!P8 zJyqL$FOkp@PSR9lK7IgLQe<(r-Migp-@!btM$((AAMbqjJ4P(agQ}y5Y9%*JsQ&>n z{U!m0YzIKuU@%vqkEPB?@GT61E9%1t-IGWpI;+$xmNNSZ#LBp;QE`ql&W_H z$r`?Aj6SxDbH4IVMGk)xK$zU8cZ2hjipfPw)m2IqFR}z3ASvx!YH~1j4O3ThOMl& zJ)^6T{nLiE&{y1E%|zz10wRcl#zlDkc3zuexT5Z}TtW8bNPtuZR6b-U6;F+l3~Qkb z6I}Z!_tM$4n;9JeyqaW;>(kcZ;OqrfdZh8!u>l5PmWTs@Vmd-W%Tt|Cc>?+&;62D3X`Ks zooZ5?((2QX}q1*|h_TN$XT%phyM`nk52);x<~0#SskA=_wvZ~}OPe~YC5>m>0aN(KRmQ}8tzVF{J; zJaew=U7p^qmZz2zli8fR24sd3Tb&lK{eqI?Gi-QvJA}qQtYW9O8cT%OR9Y)mCv)grs_zSvWnemobn4jJ77%_)`=azBbf6#w>ar+0=R;$ zXeXAx7#N1IaRjF{KEhwb(xN6w;Ek^+*DYxuCGcL_amvXmRx%fA6ZRKruNqAzM}3RB k_w}!@9uPg%%%|H2w*9aC^5^X8&;NhM;NNi!fIs#9KYW7SbN~PV literal 0 HcmV?d00001 diff --git a/docs_zh-CN/getting_started.md b/docs_zh-CN/getting_started.md index 15188696..5edfab7f 100644 --- a/docs_zh-CN/getting_started.md +++ b/docs_zh-CN/getting_started.md @@ -180,7 +180,7 @@ CUDA_VISIBLE_DEVICES=4,5,6,7 GPUS=4 ./tools/slurm_train.sh ${PARTITION} ${JOB_NA 我们根据 [flops-counter.pytorch](https://github.com/sovrasov/flops-counter.pytorch) 提供了一个脚本用于计算给定模型的 FLOPs 和参数量 ```shell -python tools/get_flops.py ${CONFIG_FILE} [--shape ${INPUT_SHAPE}] +python tools/analysis_tools/get_flops.py ${CONFIG_FILE} [--shape ${INPUT_SHAPE}] ``` 用户将获得如下结果: diff --git a/docs_zh-CN/index.rst b/docs_zh-CN/index.rst index 1aa36cee..b76c3274 100644 --- a/docs_zh-CN/index.rst +++ b/docs_zh-CN/index.rst @@ -38,6 +38,8 @@ You can switch between Chinese and English documentation in the lower-left corne tools/pytorch2torchscript.md tools/model_serving.md tools/visualization.md + tools/analysis.md + tools/miscellaneous.md .. toctree:: diff --git a/docs_zh-CN/tools/analysis.md b/docs_zh-CN/tools/analysis.md new file mode 100644 index 00000000..be10eb0f --- /dev/null +++ b/docs_zh-CN/tools/analysis.md @@ -0,0 +1,211 @@ +# 分析 + + + +- [日志分析](#日志分析) + - [绘制曲线图](#绘制曲线图) + - [统计训练时间](#统计训练时间) +- [结果分析](#结果分析) + - [评估结果](#查看典型结果) + - [查看典型结果](#查看典型结果) +- [模型复杂度分析](#模型复杂度分析) +- [常见问题](#常见问题) + + + +## 日志分析 + +### 绘制曲线图 + +指定一个训练日志文件,可通过 `tools/analysis_tools/analyze_logs.py` 脚本绘制指定键值的变化曲线 + +