Since its inception, the OpenStack project has used Jenkins to perform its testing and artifact building. When OpenStack was two git repos, we had one Jenkins master, a few slaves, and we configured all of our jobs manually in the web interface. It was easy for a new project like OpenStack to set up and maintain. Over the years, we have grown significantly, with over 1,200 git repos and 8,000 jobs spread across 8 Jenkins masters and 800 dynamic slave nodes. Long before we got to this point, we could not manage all of those jobs by hand, so we wrote Jenkins Job Builder, one of our more widely used projects, so that we could automatically generate those 8,000 jobs from templated YAML.
We also wrote Zuul.
Zuul is a system to drive project automation. It directs our testing, running tens of thousands of jobs each day, responding to events from our code review system and stacking potential changes to be tested together.
We are working on a new version of Zuul (version 3) with some major changes: we want to make it easier to run jobs in multi-node environments, easier to manage large numbers of jobs and job variations, support in-tree job configuration, and the ability to define jobs using Ansible.
With Zuul in charge of deciding which jobs to run, and when and where to run them, we use very few advanced features of Jenkins at this point. While we are still working on Zuul v3, we are at a point where we can start to use some of the work we have done already to switch to running our jobs entirely with Zuul.
As of today, we have turned off our last Jenkins master and all of our automation is being run by Zuul. It's been a great ride, and OpenStack wouldn't be where it is today without Jenkins. Now we're looking forward to focusing on Zuul v3 and exploring the full potential of project automation.
Subscribe to Amo Probos