<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" id="owaParaStyle">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body fpstyle="1" ocsi="0">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Thanks for sharing this Ben. This is definitely a tool that we should add to our future investigations in order to address Kata CI scalability concerns (Zuul being already on this
list).<br>
<br>
Sebastien<br>
<div style="font-family: Times New Roman; color: #000000; font-size: 16px">
<hr tabindex="-1">
<div id="divRpF873945" style="direction: ltr;"><font size="2" face="Tahoma" color="#000000"><b>From:</b> Benjamin Elder [bentheelder@google.com]<br>
<b>Sent:</b> Thursday, March 01, 2018 7:27 PM<br>
<b>To:</b> Tim Allclair<br>
<b>Cc:</b> Cole Wagner; Sen Lu; kata-dev@lists.katacontainers.io<br>
<b>Subject:</b> Re: [kata-dev] Sharing our Kubernetes CI experience<br>
</font><br>
</div>
<div></div>
<div>
<div dir="ltr">Hello Kata Community,
<div><br>
My understanding is that initially Prow [1] <span style="color:rgb(34,34,34); font-family:sans-serif; font-size:13px; font-style:normal; font-weight:400; letter-spacing:normal; text-align:start; text-indent:0px; text-transform:none; white-space:normal; word-spacing:0px; background-color:rgb(255,255,255); float:none; display:inline">grew
out of a need for more reliable GitHub integration</span> just handled triggering Jenkins jobs; </div>
<div>eventually though someone realized that most of our tasks ran in docker containers and that Kubernetes Clusters are a great way to run containers, so we started scheduling some tests as Kubernetes Pods instead.
<div>Since then we've also realized that we were wasting a lot of time dealing with Jenkins cluster outages, nodes running out of disk space, etc. and that we didn't run into those sort of issues very often with Kubernetes.</div>
<div>We now run all of our builds / tests for Kubernetes (10k+ jobs a day [2]) and generally only have outages when say, GitHub is unreachable or we've broken something in Prow, we rarely actually need to do any maintenance tasks on the cluster itself.</div>
<div>Prow has also expanded to handle a lot of our other project automation tasks [3] in a scale-able manner with plugins that handle GitHub events (via webhook) through the same tooling used for triggering presubmits.</div>
</div>
<div><br>
</div>
<div>There's definitely some bias in the preference for building on Kubernetes, but I think this system has generally worked really well and we've been able to make Prow do a lot for us. A few other communities (OpenShift, Istio...) are also using Prow and
seem to be doing pretty well with it. </div>
<div><br>
</div>
<div>[1] <a href="https://github.com/kubernetes/test-infra/tree/master/prow" target="_blank">
https://github.com/kubernetes/test-infra/tree/master/prow</a></div>
<div>[2] <a href="http://velodrome.k8s.io/dashboard/db/bigquery-metrics?orgId=1" target="_blank">http://velodrome.k8s.io/dashboard/db/bigquery-metrics?orgId=1</a></div>
<div>[3] <a href="https://prow.k8s.io/plugins" target="_blank">https://prow.k8s.io/plugins</a></div>
</div>
<br>
<br>
<div class="gmail_quote">
<div dir="ltr">On Thu, Mar 1, 2018 at 6:11 PM Tim Allclair <<a href="mailto:tallclair@google.com" target="_blank">tallclair@google.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex; border-left:1px #ccc solid; padding-left:1ex">
<div dir="ltr">
<div class="gmail_default" style="font-size:small">​Several folks from our Kubernetes test-infra team have agreed to share some of their experience with running Kubernetes CI, which was originally based on jenkins but now uses a custom built tool called
<a href="https://github.com/kubernetes/test-infra/blob/master/prow/README.md" target="_blank">
prow</a>.</div>
<div class="gmail_default" style="font-size:small"><br>
</div>
<div class="gmail_default" style="font-size:small">@Kata community - Feel free to respond with questions.<br>
</div>
<div class="gmail_default" style="font-size:small">@Ben, Sen & Cole - Thanks for sharing! Would you like to give a brief summary of the issues we had with Jenkins, and why we decided to build prow?<br>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</body>
</html>