<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:DengXian;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
text-indent:21.0pt;
font-size:10.5pt;
font-family:DengXian;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:DengXian;
color:windowtext;}
span.author-a-uz72zz70zqmz78ztz82zqz79zz88zl6i4n
{mso-style-name:author-a-uz72zz70zqmz78ztz82zqz79zz88zl6i4n;}
span.author-a-z89zz71zz86zz88zv3gcz65zz76z8z80zuz75zz82zj
{mso-style-name:author-a-z89zz71zz86zz88zv3gcz65zz76z8z80zuz75zz82zj;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:DengXian;}
/* Page Definitions */
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:75516980;
mso-list-type:hybrid;
mso-list-template-ids:1144939930 -550606682 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:18.0pt;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-text:"%2\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:42.0pt;
text-indent:-21.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:63.0pt;
text-indent:-21.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:84.0pt;
text-indent:-21.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-text:"%5\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:105.0pt;
text-indent:-21.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:126.0pt;
text-indent:-21.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:147.0pt;
text-indent:-21.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-text:"%8\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:168.0pt;
text-indent:-21.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:189.0pt;
text-indent:-21.0pt;}
@list l1
{mso-list-id:555436272;
mso-list-type:hybrid;
mso-list-template-ids:-533176954 -550606682 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l1:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:18.0pt;
text-indent:-18.0pt;}
@list l1:level2
{mso-level-number-format:alpha-lower;
mso-level-text:"%2\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:42.0pt;
text-indent:-21.0pt;}
@list l1:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:63.0pt;
text-indent:-21.0pt;}
@list l1:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:84.0pt;
text-indent:-21.0pt;}
@list l1:level5
{mso-level-number-format:alpha-lower;
mso-level-text:"%5\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:105.0pt;
text-indent:-21.0pt;}
@list l1:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:126.0pt;
text-indent:-21.0pt;}
@list l1:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:147.0pt;
text-indent:-21.0pt;}
@list l1:level8
{mso-level-number-format:alpha-lower;
mso-level-text:"%8\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:168.0pt;
text-indent:-21.0pt;}
@list l1:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:189.0pt;
text-indent:-21.0pt;}
@list l2
{mso-list-id:846555390;
mso-list-type:hybrid;
mso-list-template-ids:-230376312 -1643629730 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l2:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:30.0pt;
text-indent:-18.0pt;}
@list l2:level2
{mso-level-number-format:alpha-lower;
mso-level-text:"%2\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:54.0pt;
text-indent:-21.0pt;}
@list l2:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:75.0pt;
text-indent:-21.0pt;}
@list l2:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:96.0pt;
text-indent:-21.0pt;}
@list l2:level5
{mso-level-number-format:alpha-lower;
mso-level-text:"%5\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:117.0pt;
text-indent:-21.0pt;}
@list l2:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:138.0pt;
text-indent:-21.0pt;}
@list l2:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:159.0pt;
text-indent:-21.0pt;}
@list l2:level8
{mso-level-number-format:alpha-lower;
mso-level-text:"%8\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:180.0pt;
text-indent:-21.0pt;}
@list l2:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:201.0pt;
text-indent:-21.0pt;}
@list l3
{mso-list-id:1413510336;
mso-list-type:hybrid;
mso-list-template-ids:1458990858 -550606682 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l3:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:18.0pt;
text-indent:-18.0pt;}
@list l3:level2
{mso-level-number-format:alpha-lower;
mso-level-text:"%2\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:42.0pt;
text-indent:-21.0pt;}
@list l3:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:63.0pt;
text-indent:-21.0pt;}
@list l3:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:84.0pt;
text-indent:-21.0pt;}
@list l3:level5
{mso-level-number-format:alpha-lower;
mso-level-text:"%5\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:105.0pt;
text-indent:-21.0pt;}
@list l3:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:126.0pt;
text-indent:-21.0pt;}
@list l3:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:147.0pt;
text-indent:-21.0pt;}
@list l3:level8
{mso-level-number-format:alpha-lower;
mso-level-text:"%8\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:168.0pt;
text-indent:-21.0pt;}
@list l3:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:189.0pt;
text-indent:-21.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=ZH-CN link="#0563C1" vlink="#954F72" style='text-justify-trim:punctuation'><div class=WordSection1><p class=MsoNormal><span lang=EN-US>Hi, Kata-team,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>To address the objective of how to shape up Kata performance metrics testing and execute it well last week, and we happen to have an open source hackathon event in Shenzhen this week, gathering most of Kata developers/user delegates together, here is a brief note of the initial discussion and also proposal of the meeting set up as a follow-up. If there is any comments or feedback or suggestions, please kindly share here.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Two main areas are majorly addressed on the discussion,<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span lang=EN-US><span style='mso-list:Ignore'>1.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-US>Performance metrics testing environment<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span lang=EN-US><span style='mso-list:Ignore'>2.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-US>Performance metrics testing requirement<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>And some of the initial conclusion in these two areas are below,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Base Guideline<o:p></o:p></span></p><p class=MsoListParagraph align=left style='margin-left:30.0pt;text-align:left;text-indent:-18.0pt;mso-list:l2 level1 lfo4'><![if !supportLists]><span lang=EN-US><span style='mso-list:Ignore'>1.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-US>To measure performance of each major/minor change and prevent regression, light weight testing for each PR, full functional test on daily or weekly basis.<o:p></o:p></span></p><p class=MsoListParagraph align=left style='margin-left:30.0pt;text-align:left;text-indent:-18.0pt;mso-list:l2 level1 lfo4'><![if !supportLists]><span lang=EN-US><span style='mso-list:Ignore'>2.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-US>To measure Kata performance, comparing with default container runtime (e.g. runc).<o:p></o:p></span></p><p class=MsoNormal align=left style='text-align:left'><span lang=EN-US>Testing Environment<o:p></o:p></span></p><p class=MsoListParagraph align=left style='margin-left:30.0pt;text-align:left;text-indent:-18.0pt;mso-list:l2 level1 lfo4'><![if !supportLists]><span lang=EN-US><span style='mso-list:Ignore'>3.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-US>Physical server nodes required, virtualized environment or nested virtualization is not accepted (as it is dynamically allocated and cannot be used as baseline)<o:p></o:p></span></p><p class=MsoListParagraph align=left style='margin-left:30.0pt;text-align:left;text-indent:-18.0pt;mso-list:l2 level1 lfo4'><![if !supportLists]><span lang=EN-US><span style='mso-list:Ignore'>4.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-US>Several testing environment options, Openlab, Alibaba server. <a href="https://openlabtesting.org/">https://openlabtesting.org/</a><o:p></o:p></span></p><p class=MsoListParagraph align=left style='margin-left:30.0pt;text-align:left;text-indent:-18.0pt;mso-list:l2 level1 lfo4'><![if !supportLists]><span lang=EN-US><span style='mso-list:Ignore'>5.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-US>It may not require large scale hardware resource environment, typically like 2 server nodes should be efficient.<o:p></o:p></span></p><p class=MsoNormal align=left style='text-align:left'><span lang=EN-US>Testing Requirement<o:p></o:p></span></p><p class=MsoListParagraph align=left style='margin-left:30.0pt;text-align:left;text-indent:-18.0pt;mso-list:l2 level1 lfo4'><![if !supportLists]><span lang=EN-US><span style='mso-list:Ignore'>6.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-US>Base metrics testing tools are: <a href="https://github.com/kata-containers/tests/tree/master/metrics">https://github.com/kata-containers/tests/tree/master/metrics</a><o:p></o:p></span></p><p class=MsoListParagraph align=left style='margin-left:30.0pt;text-align:left;text-indent:-18.0pt;mso-list:l2 level1 lfo4'><![if !supportLists]><span lang=EN-US><span style='mso-list:Ignore'>7.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-US>Network testing - latency testing, pps testing, throughput testing are critical.<o:p></o:p></span></p><p class=MsoListParagraph align=left style='margin-left:30.0pt;text-align:left;text-indent:-18.0pt;mso-list:l2 level1 lfo4'><![if !supportLists]><span lang=EN-US><span style='mso-list:Ignore'>8.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-US>Also needs to measure CPU utilization when reaching best network performance<o:p></o:p></span></p><p class=MsoListParagraph align=left style='margin-left:30.0pt;text-align:left;text-indent:-18.0pt;mso-list:l2 level1 lfo4'><![if !supportLists]><span lang=EN-US><span style='mso-list:Ignore'>9.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-US>Performance comparing with runc, 10% overhead tolerance maximally.<o:p></o:p></span></p><p class=MsoListParagraph align=left style='margin-left:30.0pt;text-align:left;text-indent:-18.0pt;mso-list:l2 level1 lfo4'><![if !supportLists]><span lang=EN-US><span style='mso-list:Ignore'>10.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-US>I/O latency, throughput, CPU utilization when reaching best I/O performance<o:p></o:p></span></p><p class=MsoListParagraph align=left style='margin-left:30.0pt;text-align:left;text-indent:-18.0pt;mso-list:l2 level1 lfo4'><![if !supportLists]><span lang=EN-US><span style='mso-list:Ignore'>11.<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-US>Typical container workload performance measurement, one proposal is to fetch top 10 list from famous docker registry, and to measure how these workload performed in Kata containers. For example, how long it will take to complete a tensorflow job or finish a kernel building.<o:p></o:p></span></p><p class=MsoNormal align=left style='text-align:left'><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>To further follow up the discussion, and push the effort. A biweekly meeting was discussed as well to be set up for people to join and share progress/feedback. Please kindly share feedback if biweekly works or not for you. It could be a zoom call or IRC meeting.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US>Horace<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></div></body></html>