eventually though someone realized that most of our tasks ran in docker containers and that Kubernetes Clusters are a great way to run containers, so we started scheduling some tests as Kubernetes Pods instead.
Since then we've also realized that we were wasting a lot of time dealing with Jenkins cluster outages, nodes running out of disk space, etc. and that we didn't run into those sort of issues very often with Kubernetes.
We now run all of our builds / tests for Kubernetes (10k+ jobs a day [2]) and generally only have outages when say, GitHub is unreachable or we've broken something in Prow, we rarely actually need to do any maintenance tasks on the cluster itself.
Prow has also expanded to handle a lot of our other project automation tasks [3] in a scale-able manner with plugins that handle GitHub events (via webhook) through the same tooling used for triggering presubmits.
There's definitely some bias in the preference for building on Kubernetes, but I think this system has generally worked really well and we've been able to make Prow do a lot for us. A few other communities (OpenShift, Istio...) are also using Prow and seem to be doing pretty well with it.