[00:05:34] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10observability, 06SRE, 13Patch-For-Review: Export zuul metrics to Prometheus - https://phabricator.wikimedia.org/T233089#10320722 (10colewhite) All dashboards in the [[ https://grafana-rw.wikimedia.org/dashboards/f/NHnAVr54k/rel... [00:07:16] Hey folks! Wanted to let you know the RelEng dashboards have Prometheus versions available for your review (https://phabricator.wikimedia.org/T233089#10320722). Please let us know if these replacements are suitable and we will begin cleaning up the old ones. Thanks! [07:08:26] 10Release-Engineering-Team (Doing 😎), 13Patch-For-Review, 05Release, 05Train Deployments: 1.44.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T375662#10321265 (10SomeRandomDeveloper) Is MultimediaViewer also updated with the train? If so, is it possible to get the fix for T379835 on the tr... [08:36:55] 06Release-Engineering-Team, 10MediaWiki-Core-Tests, 10Wikidata, 10wmde-wikidata-tech, and 4 others: Create a daily job running core + extension PHPUnit tests serially - https://phabricator.wikimedia.org/T372618#10321496 (10ArthurTaylor) a:03ArthurTaylor [08:52:38] FIRING: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [09:02:38] RESOLVED: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [09:28:35] Yippee, build fixed! [09:28:36] Project beta-scap-sync-world build #180591: 09FIXED in 2 min 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/180591/ [09:51:20] (03open) 10jnuche: jenkins-rel: updating plugin versions to address vulnerabilities [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/98 (https://phabricator.wikimedia.org/T379763) [09:56:38] (03update) 10jnuche: jenkins-rel: updating plugin versions to address vulnerabilities [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/98 (https://phabricator.wikimedia.org/T379763) [09:56:58] (03update) 10jnuche: jenkins-rel: updating plugin versions to address vulnerabilities [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/98 (https://phabricator.wikimedia.org/T379763) [09:57:27] (03merge) 10jnuche: jenkins-rel: updating plugin versions to address vulnerabilities [repos/releng/jenkins-deploy] - 10https://gitlab.wikimedia.org/repos/releng/jenkins-deploy/-/merge_requests/98 (https://phabricator.wikimedia.org/T379763) [10:07:14] 10GitLab (Infrastructure), 06collaboration-services: GitLab OpenSSL 3 upgrade in 17.7 - https://phabricator.wikimedia.org/T379598#10321821 (10MoritzMuehlenhoff) I had a closer look and I can confirm that the use of openssl is self-contained within the gitlab monorepo package: - The openssl binary as called by... [10:16:00] 10Continuous-Integration-Infrastructure, 07Jenkins, 06Release-Engineering-Team: Cleanup some plugins from the releases Jenkins - https://phabricator.wikimedia.org/T328687#10321831 (10jnuche) 05Open→03Resolved [10:48:14] (03open) 10jelto: add wikidata-query-builder to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/100 (https://phabricator.wikimedia.org/T350793) [10:49:07] (03update) 10jelto: add wikidata-query-builder to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/100 (https://phabricator.wikimedia.org/T350793) [10:49:56] (03approved) 10eoghan: add wikidata-query-builder to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/100 (https://phabricator.wikimedia.org/T350793) (owner: 10jelto) [10:50:49] (03update) 10jelto: add wikidata-query-builder to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/100 (https://phabricator.wikimedia.org/T350793) [10:52:40] (03merge) 10jelto: add wikidata-query-builder to trusted runners [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/100 (https://phabricator.wikimedia.org/T350793) [12:31:44] 06Release-Engineering-Team, 06Data Products, 10Dumps-Generation, 10MW-on-K8s, and 2 others: Migrate current-generation dumps to run from our containerized images - https://phabricator.wikimedia.org/T352650#10322183 (10BTullis) >>! In T352650#10276308, @MatthewVernon wrote: > I suspect I've missed something... [12:34:17] 10GitLab (Pipeline Services Migration🐤), 06collaboration-services, 10Wikidata, 10Wikidata Query UI, and 3 others: move commons-query.wikimedia.org and query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10322212 (10Jelto) I have mirrored the `query-builder` Gerrit repository to [Gi... [12:34:59] 10GitLab (Pipeline Services Migration🐤), 06collaboration-services, 10Wikidata, 10Wikidata Query UI, and 3 others: move commons-query.wikimedia.org and query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10322214 (10Jelto) [12:35:21] 10GitLab (Pipeline Services Migration🐤), 06collaboration-services, 10Wikidata, 10Wikidata Query UI, and 3 others: move commons-query.wikimedia.org and query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10322215 (10Jelto) [12:44:35] FIRING: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [12:46:33] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [12:54:01] 06Release-Engineering-Team, 06Data Products, 06Data-Platform-SRE, 10Dumps-Generation, and 3 others: Migrate current-generation dumps to run from our containerized images - https://phabricator.wikimedia.org/T352650#10322255 (10BTullis) I'll add the #epic tag and move this back to the main #data-platform-sre... [12:59:36] RESOLVED: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [13:01:33] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [13:42:44] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 06Content-Transform-Team-WIP, 10Cloud-VPS (Debian Buster Deprecation): Rebuild or delete deployment-docker-proton01 - https://phabricator.wikimedia.org/T369916#10322441 (10hashar) 05Resolved→03Open [[ https://openstack-browser.toolforge.org/server/d... [13:48:00] 10Beta-Cluster-Infrastructure, 10MediaWiki-Core-HTTP-Cache, 10MediaWiki-Core-Revision-backend, 06MW-Interfaces-Team: Caching on beta.metawiki playing up - https://phabricator.wikimedia.org/T379841#10322453 (10Tgr) They look identical to me. In production (if the caching happens in Varnish) caching bugs mig... [13:55:08] (03open) 10lwatson: web: Convert components to Composition API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/570 (https://phabricator.wikimedia.org/T379832) [14:00:17] 10Beta-Cluster-Infrastructure, 10MediaWiki-Core-HTTP-Cache, 10MediaWiki-Core-Revision-backend, 06MW-Interfaces-Team: Caching on beta.metawiki playing up - https://phabricator.wikimedia.org/T379841#10322529 (10Krinkle) I can't reproduce this issue. 1. Logged-in: https://meta.wikimedia.beta.wmflabs.org/w/in... [14:02:55] (03update) 10lwatson: web: Convert components to Composition API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/570 (https://phabricator.wikimedia.org/T379832) [14:03:45] (03update) 10lwatson: web: Convert components to Composition API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/570 (https://phabricator.wikimedia.org/T379832) [14:19:05] 10Beta-Cluster-Infrastructure, 06cloud-services-team, 06Content-Transform-Team-WIP, 10Cloud-VPS (Debian Buster Deprecation): Rebuild or delete deployment-docker-proton01 - https://phabricator.wikimedia.org/T369916#10322602 (10fnegri) p:05Triage→03Low If it's not deleted by tomorrow, it will be deleted... [14:33:27] (03update) 10lwatson: Draft: web: Convert components to Composition API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/570 (https://phabricator.wikimedia.org/T379832) [14:56:07] (03merge) 10lwatson: releases: Bump Codex to 1.16.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/43 (owner: 10annet) [15:16:13] 10Release-Engineering-Team (Doing 😎), 13Patch-For-Review, 05Release, 05Train Deployments: 1.44.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T375662#10322786 (10Ladsgroup) Yeah, it gets updated with all other extensions. You can see the list of changes in wmf.3 in https://www.mediawiki.or... [15:17:45] 10Release-Engineering-Team (Doing 😎), 13Patch-For-Review, 05Release, 05Train Deployments: 1.44.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T375662#10322789 (10Ladsgroup) But that revert will go to wmf.4 so it should be backported to wmf.3 [15:24:57] 10Continuous-Integration-Infrastructure, 10Quibble: Add support for multi-DB testing to MediaWiki - https://phabricator.wikimedia.org/T379925 (10Urbanecm) 03NEW [15:25:18] 10Continuous-Integration-Infrastructure, 10Quibble: Add support for multi-DB testing to MediaWiki - https://phabricator.wikimedia.org/T379925#10322833 (10Urbanecm) [15:28:39] 10Continuous-Integration-Infrastructure, 10Quibble: Add support for multi-DB testing to MediaWiki - https://phabricator.wikimedia.org/T379925#10322880 (10Urbanecm) [15:48:45] (03PS2) 10Jforrester: Docker: [sury-php] Replatform to bookworm [integration/config] - 10https://gerrit.wikimedia.org/r/1090959 [16:04:26] (03update) 10lwatson: Draft: web: Convert components to Composition API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/570 (https://phabricator.wikimedia.org/T379832) [16:14:41] 10GitLab (Pipeline Services Migration🐤), 06collaboration-services: move os-reports.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T350794#10323046 (10Arnoldokoth) We settled on what a deployment for this could look like. An initContainer that shares an emptyDir volume with the serving containe... [16:20:49] 10Release-Engineering-Team (Doing 😎), 13Patch-For-Review, 05Release, 05Train Deployments: 1.44.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T375662#10323062 (10Ladsgroup) >>! In T375662#10322789, @Ladsgroup wrote: > But that revert will go to wmf.4 so it should be backported to wmf.3 {{... [16:21:33] (03Abandoned) 10Umherirrender: Review access change [extensions] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1082872 (owner: 10Blake Hale) [16:23:25] (03update) 10dancy: Reduce the cardinality of scap timers/statsd metrics (phase 1) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/569 (https://phabricator.wikimedia.org/T377883) [16:23:30] (03update) 10dancy: Reduce the cardinality of scap timers/statsd metrics (phase 1) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/569 (https://phabricator.wikimedia.org/T377883) [16:26:21] Is there someone I can ask to reload zuul? [16:26:33] https://gerrit.wikimedia.org/r/c/integration/config/+/1088649 doesn't seem to have been applied [16:27:17] Dreamy_Jazz: I can do it. [16:27:30] Thanks! [16:28:29] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/c/integration/config/+/1088649 [16:28:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:28:44] Done. [16:38:23] (03CR) 10Jforrester: [C:03+2] Zuul [mediawiki/extensions/EmailNotifications] Disable selenium [integration/config] - 10https://gerrit.wikimedia.org/r/1088595 (owner: 10Zoranzoki21) [16:39:56] (03Merged) 10jenkins-bot: Zuul [mediawiki/extensions/EmailNotifications] Disable selenium [integration/config] - 10https://gerrit.wikimedia.org/r/1088595 (owner: 10Zoranzoki21) [16:40:06] !log Zuul: [mediawiki/extensions/EmailNotifications] Disable selenium testing [16:40:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:23:51] (03update) 10lwatson: Draft: web: Convert components to Composition API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/570 (https://phabricator.wikimedia.org/T379832) [17:25:00] (03update) 10lwatson: Draft: web: Convert components to Composition API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/570 (https://phabricator.wikimedia.org/T379832) [18:25:16] (03update) 10lwatson: Draft: web: Convert components to Composition API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/570 (https://phabricator.wikimedia.org/T379832) [18:33:10] (03open) 10dancy: initialize-mediawiki-staging: misc fixes [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/113 [18:33:13] (03update) 10dancy: initialize-mediawiki-staging: misc fixes [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/113 [18:34:09] (03merge) 10dancy: initialize-mediawiki-staging: misc fixes [repos/releng/train-dev] - 10https://gitlab.wikimedia.org/repos/releng/train-dev/-/merge_requests/113 [18:43:09] I notice that the (only?) pool counter server in beta, poolcounter06, is shut off and listed for deletion soon. [18:43:24] https://openstack-browser.toolforge.org/project/deployment-prep [18:43:30] https://openstack-browser.toolforge.org/server/deployment-poolcounter06.deployment-prep.eqiad1.wikimedia.cloud [18:44:16] This seems like a problem :). I'm guessing that's causing warnings in beta Logstash (if not, it should be!) [18:44:40] https://codesearch.wmcloud.org/search/?q=poolcounter06&files=&excludeFiles=&repos= [18:50:36] (03update) 10lwatson: Draft: web: Convert components to Composition API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/570 (https://phabricator.wikimedia.org/T379832) [18:51:35] (03update) 10lwatson: Draft: web: Convert components to Composition API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/570 (https://phabricator.wikimedia.org/T379832) [18:52:59] (03update) 10lwatson: web: Convert components to Composition API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/570 (https://phabricator.wikimedia.org/T379832) [20:14:38] (03update) 10lwatson: web: Convert components to Composition API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/570 (https://phabricator.wikimedia.org/T379832) [20:16:07] (03update) 10lwatson: web: Convert components to Composition API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/570 (https://phabricator.wikimedia.org/T379832) [20:32:09] (03approved) 10thcipriani: Reduce the cardinality of scap timers/statsd metrics (phase 1) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/569 (https://phabricator.wikimedia.org/T377883) (owner: 10dancy) [20:32:35] (03merge) 10thcipriani: Reduce the cardinality of scap timers/statsd metrics (phase 1) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/569 (https://phabricator.wikimedia.org/T377883) (owner: 10dancy) [20:45:21] (03open) 10dancy: tests/scap/test_log.py: Add test for log.Timer() with name=None [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/571 [20:45:23] (03update) 10dancy: tests/scap/test_log.py: Add test for log.Timer() with name=None [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/571 [20:48:16] (03open) 10dancy: Reduce the cardinality of scap timers/statsd metrics (phase 2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/572 [20:48:19] (03update) 10dancy: Reduce the cardinality of scap timers/statsd metrics (phase 2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/572 [20:50:47] (03update) 10dancy: tests/scap/test_log.py: Add test for log.Timer() with name=None [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/571 [20:53:34] (03update) 10dancy: Reduce the cardinality of scap timers/statsd metrics (phase 2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/572 [21:33:10] 10Beta-Cluster-Infrastructure, 10Citoid: citoid service doesn't work on beta - https://phabricator.wikimedia.org/T346624#10324501 (10Mvolz) 05Open→03Resolved [21:33:18] 10Beta-Cluster-Infrastructure, 10Citoid: citoid service doesn't work on beta - https://phabricator.wikimedia.org/T346624#10324503 (10Mvolz) It's working now (though outdated.) [21:37:06] !log Installed scap 4.123.0 in deployment-prep. [21:37:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:39:48] Project beta-scap-sync-world build #180661: 04FAILURE in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/180661/ [21:41:33] !log Reverting scap to 4.101.1 in deployment-prep. [21:41:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:47:33] Yippee, build fixed! [21:47:33] Project beta-scap-sync-world build #180662: 09FIXED in 3 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/180662/ [22:05:06] hi! I'm trying to figure what I'm doing wrong at https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CodeMirror/+/1090587 [22:05:19] I got BetaFeatures added as a dependency of CodeMirror https://gerrit.wikimedia.org/r/c/integration/config/+/1090589 [22:05:40] then there's the changes in .phan/config.php https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CodeMirror/+/1090587/6/.phan/config.php [22:05:53] with this, Phan passes on my local, but CI still complains. Any ideas? [22:06:07] https://integration.wikimedia.org/ci/job/mwext-php74-phan/55072/console [22:06:47] (I ask here as this seems like a CI issue) [22:08:39] I have experienced caching issues with this repo in CI several times, and I wonder if maybe that is what is going on here? The build output doesn't suggest this, but all the pieces to make Phan happy are there… it's almost like CI isn't including the change to config.php [22:24:40] 10Release-Engineering-Team (Doing 😎), 13Patch-For-Review, 05Release, 05Train Deployments: 1.44.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T375662#10324619 (10brennen) 05Open→03Resolved Currently stable on all wikis. [22:37:47] 10Release-Engineering-Team (Doing 😎), 07OKR-Work: [WE6.2.6] Create design document for Group -1 deployment - https://phabricator.wikimedia.org/T379683#10324662 (10bd808) [22:41:56] musikanimal: there's a separate .yaml file for phan dependencies [22:42:28] dependencies.yaml is used for phpunit, phan_dependencies.yaml for phan [22:43:08] ah! that must be it! thank you MatmaRex :) [22:45:18] (03PS1) 10MusikAnimal: Zuul: add BetaFeatures as CodeMirror phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1091390 (https://phabricator.wikimedia.org/T376735) [22:45:48] (03PS2) 10MusikAnimal: Zuul: add BetaFeatures as CodeMirror phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1091390 (https://phabricator.wikimedia.org/T376735) [23:04:15] (03update) 10lwatson: web: Convert components to Composition API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/570 (https://phabricator.wikimedia.org/T379832) [23:04:17] (03update) 10lwatson: web: Convert components to Composition API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/570 (https://phabricator.wikimedia.org/T379832) [23:16:44] (03approved) 10egardner: web: Convert components to Composition API [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/570 (https://phabricator.wikimedia.org/T379832) (owner: 10lwatson) [23:24:46] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T375663#10324758 (10Jdlrobson) [23:29:14] Project beta-scap-sync-world build #180672: 04FAILURE in 3 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/180672/ [23:39:26] Yippee, build fixed! [23:39:26] Project beta-scap-sync-world build #180673: 09FIXED in 2 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/180673/ [23:48:57] PROBLEM - Work requests waiting in Zuul Gearman server on contint1002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [23:53:57] RECOVERY - Work requests waiting in Zuul Gearman server on contint1002 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10