Wikimedia Release Engineering Team/Runbooks
This is a list of runbooks for the Wikimedia Release Engineering Team, covering step-by-step lists of what to do when things need doing, especially when things go wrong.
Gerrit
GitLab
Continuous integration
Configuration
- Add/modify CI for a new/existing repo (Zuul)
- Adding a new release pipeline for MediaWiki to CI (Zuul)
- Add/modify a new type of CI job (Jenkins Job Builder)
- Add/modify a new docker environment for CI jobs (Dockerfiles)
- Creating and deploying a new Quibble release (Dockerfiles + JJB for the critical CI workflow)
- Update doc.wikimedia.org static content (docroot)
- Replay a gerrit CI event into Zuul to re-trigger jobs
Infrastructure
- Clear part of Jenkins, when jobs are deadlocked ("waiting on executors") / Jenkins stuck
- Restart zuul (and drop all running jobs!)
- Agent remote call failed
- Upgrade Jenkins
- Work requests waiting in Zuul is CRITICAL -- this is usually a long chain of patchsets
- If you caught it fast enough you can do: Continuous_integration/Zuul#Very_high_queue_of_merger:merge_functions
- Otherwise your options are: wait or restart zuul
- Adding a new Jenkins agent
- Deploy doc.wikimedia.org changes
- Switch primary host for doc.wikimedia.org
- Switch primary host for integration.wikimedia.org