metrics CI tracking proxy/agent/shim PRs
Hi all, Thanks to the generosity of Packet.net and the CNCF, we have gained some hardware resource to enable metrics CI tracking. I've configured and hooked up an instance to track PRs on the proxy/agent/shim repos. You may/will start to notice an extra 'jenkins-metrics-ubuntu-16-04' item appear in the CI checks list for those repos. This should be *non-blocking* whilst we settle the system - but, be aware, even non-blocking CI failure items stop the big merge button going green - it will be grey, and still allow you to merge, if a non-blocking item fails.
As you may now need to check metrics CI failures, the 'quick' route is to go to the Jenkins console log output of the relevant build (via the link off the github status), and scroll to near the bottom of the console. You should find (given the CI built and ran OK), a summary table that looks something like this:
Report Summary: +-----+----------------------+-------+--------+--------+-------+--------+--------+-------+------+-----+ | P/F | NAME | FLR | MEAN | CEIL | GAP | MIN | MAX | RNG | COV | ITS | +-----+----------------------+-------+--------+--------+-------+--------+--------+-------+------+-----+ | P | boot-times | 95.0% | 98.4% | 105.0% | 10.0% | 94.1% | 103.7% | 10.2% | 2.3% | 20 | | P | memory-footprint | 95.0% | 100.4% | 105.0% | 10.0% | 100.4% | 100.4% | 0.0% | 0.0% | 1 | | P | memory-footprint-ksm | 95.0% | 97.0% | 105.0% | 10.0% | 97.0% | 97.0% | 0.0% | 0.0% | 1 | +-----+----------------------+-------+--------+--------+-------+--------+--------+-------+------+-----+ Fails: 0, Passes 3
Check the P/F column - anything with a '*F*' will be a failure case, and cause the CI to fail. And, yes, improving the documentation about how to set up, configure, and read the results is on 'the list' :-) You are basically looking for the MEAN to fall between the FLR(floor) and CEIL(celing) of expected results.
Once we are happy the checks are stable we will enable them across the other repos (runtime, tests etc.), and make them blocking.
Graham
--------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
Thanks! This is good news, thanks for the heads up. +1 to improve documentation, this will help new reviewers.
- Carlos
-----Original Message----- From: Whaley, Graham [mailto:graham.whaley@intel.com] Sent: Tuesday, September 11, 2018 10:44 AM To: kata-dev@lists.katacontainers.io Subject: [kata-dev] metrics CI tracking proxy/agent/shim PRs
Hi all, Thanks to the generosity of Packet.net and the CNCF, we have gained some hardware resource to enable metrics CI tracking. I've configured and hooked up an instance to track PRs on the proxy/agent/shim repos. You may/will start to notice an extra 'jenkins-metrics-ubuntu-16-04' item appear in the CI checks list for those repos. This should be *non-blocking* whilst we settle the system - but, be aware, even non-blocking CI failure items stop the big merge button going green - it will be grey, and still allow you to merge, if a non-blocking item fails.
As you may now need to check metrics CI failures, the 'quick' route is to go to the Jenkins console log output of the relevant build (via the link off the github status), and scroll to near the bottom of the console. You should find (given the CI built and ran OK), a summary table that looks something like this:
Report Summary: +-----+----------------------+-------+--------+--------+-------+--------+--------+-------+------+-----+ | P/F | NAME | FLR | MEAN | CEIL | GAP | MIN | MAX | RNG | COV | ITS | +-----+----------------------+-------+--------+--------+-------+--------+--------+-------+------+-----+ | P | boot-times | 95.0% | 98.4% | 105.0% | 10.0% | 94.1% | 103.7% | 10.2% | 2.3% | 20 | | P | memory-footprint | 95.0% | 100.4% | 105.0% | 10.0% | 100.4% | 100.4% | 0.0% | 0.0% | 1 | | P | memory-footprint-ksm | 95.0% | 97.0% | 105.0% | 10.0% | 97.0% | 97.0% | 0.0% | 0.0% | 1 | +-----+----------------------+-------+--------+--------+-------+--------+--------+-------+------+-----+ Fails: 0, Passes 3
Check the P/F column - anything with a '*F*' will be a failure case, and cause the CI to fail. And, yes, improving the documentation about how to set up, configure, and read the results is on 'the list' :-) You are basically looking for the MEAN to fall between the FLR(floor) and CEIL(celing) of expected results.
Once we are happy the checks are stable we will enable them across the other repos (runtime, tests etc.), and make them blocking.
Graham
--------------------------------------------------------------------- Intel Corporation (UK) Limited Registered No. 1134945 (England) Registered Office: Pipers Way, Swindon SN3 1RJ VAT No: 860 2173 47
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.
_______________________________________________ kata-dev mailing list kata-dev@lists.katacontainers.io http://lists.katacontainers.io/cgi-bin/mailman/listinfo/kata-dev
participants (2)
-
Venegas Munoz, Jose Carlos
-
Whaley, Graham