[kata-dev] Sharing our Kubernetes CI experience

Boeuf, Sebastien sebastien.boeuf at intel.com
Fri Mar 2 18:14:01 UTC 2018

Thanks for sharing this Ben. This is definitely a tool that we should add to our future investigations in order to address Kata CI scalability concerns (Zuul being already on this list).

From: Benjamin Elder [bentheelder at google.com]
Sent: Thursday, March 01, 2018 7:27 PM
To: Tim Allclair
Cc: Cole Wagner; Sen Lu; kata-dev at lists.katacontainers.io
Subject: Re: [kata-dev] Sharing our Kubernetes CI experience

Hello Kata Community,

My understanding is that initially Prow [1] grew out of a need for more reliable GitHub integration just handled triggering Jenkins jobs;
eventually though someone realized that most of our tasks ran in docker containers and that Kubernetes Clusters are a great way to run containers, so we started scheduling some tests as Kubernetes Pods instead.
Since then we've also realized that we were wasting a lot of time dealing with Jenkins cluster outages, nodes running out of disk space, etc. and that we didn't run into those sort of issues very often with Kubernetes.
We now run all of our builds / tests for Kubernetes (10k+ jobs a day [2]) and generally only have outages when say, GitHub is unreachable or we've broken something in Prow, we rarely actually need to do any maintenance tasks on the cluster itself.
Prow has also expanded to handle a lot of our other project automation tasks [3] in a scale-able manner with plugins that handle GitHub events (via webhook) through the same tooling used for triggering presubmits.

There's definitely some bias in the preference for building on Kubernetes, but I think this system has generally worked really well and we've been able to make Prow do a lot for us. A few other communities (OpenShift, Istio...) are also using Prow and seem to be doing pretty well with it.

[1] https://github.com/kubernetes/test-infra/tree/master/prow
[2] http://velodrome.k8s.io/dashboard/db/bigquery-metrics?orgId=1
[3] https://prow.k8s.io/plugins

On Thu, Mar 1, 2018 at 6:11 PM Tim Allclair <tallclair at google.com<mailto:tallclair at google.com>> wrote:
​Several folks from our Kubernetes test-infra team have agreed to share some of their experience with running Kubernetes CI, which was originally based on jenkins but now uses a custom built tool called prow<https://github.com/kubernetes/test-infra/blob/master/prow/README.md>.

@Kata community - Feel free to respond with questions.
@Ben, Sen & Cole - Thanks for sharing! Would you like to give a brief summary of the issues we had with Jenkins, and why we decided to build prow?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.katacontainers.io/pipermail/kata-dev/attachments/20180302/0e396830/attachment.html>

More information about the kata-dev mailing list