Thanks for sharing this Ben. This is definitely a tool that we should add to our future investigations in order to address Kata CI scalability concerns (Zuul being already on this list). Sebastien ________________________________ From: Benjamin Elder [bentheelder@google.com] Sent: Thursday, March 01, 2018 7:27 PM To: Tim Allclair Cc: Cole Wagner; Sen Lu; kata-dev@lists.katacontainers.io Subject: Re: [kata-dev] Sharing our Kubernetes CI experience Hello Kata Community, My understanding is that initially Prow [1] grew out of a need for more reliable GitHub integration just handled triggering Jenkins jobs; eventually though someone realized that most of our tasks ran in docker containers and that Kubernetes Clusters are a great way to run containers, so we started scheduling some tests as Kubernetes Pods instead. Since then we've also realized that we were wasting a lot of time dealing with Jenkins cluster outages, nodes running out of disk space, etc. and that we didn't run into those sort of issues very often with Kubernetes. We now run all of our builds / tests for Kubernetes (10k+ jobs a day [2]) and generally only have outages when say, GitHub is unreachable or we've broken something in Prow, we rarely actually need to do any maintenance tasks on the cluster itself. Prow has also expanded to handle a lot of our other project automation tasks [3] in a scale-able manner with plugins that handle GitHub events (via webhook) through the same tooling used for triggering presubmits. There's definitely some bias in the preference for building on Kubernetes, but I think this system has generally worked really well and we've been able to make Prow do a lot for us. A few other communities (OpenShift, Istio...) are also using Prow and seem to be doing pretty well with it. [1] https://github.com/kubernetes/test-infra/tree/master/prow [2] http://velodrome.k8s.io/dashboard/db/bigquery-metrics?orgId=1 [3] https://prow.k8s.io/plugins On Thu, Mar 1, 2018 at 6:11 PM Tim Allclair <tallclair@google.com<mailto:tallclair@google.com>> wrote: ​Several folks from our Kubernetes test-infra team have agreed to share some of their experience with running Kubernetes CI, which was originally based on jenkins but now uses a custom built tool called prow<https://github.com/kubernetes/test-infra/blob/master/prow/README.md>. @Kata community - Feel free to respond with questions. @Ben, Sen & Cole - Thanks for sharing! Would you like to give a brief summary of the issues we had with Jenkins, and why we decided to build prow?