[01:29:01] <wikibugs>	 (03update) 10kineticpelagic: Add missing rule to RULE_REFERENCE.md [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/18 (owner: 10kbach)
[03:19:13] <wmf-insecte>	 Project mediawiki-core-doxygen build #21915: 04FAILURE in 1 min 11 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21915/
[04:33:31] <wmf-insecte>	 Yippee, build fixed!
[04:33:32] <wmf-insecte>	 Project mediawiki-core-doxygen build #21916: 09FIXED in 15 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21916/
[04:33:46] <wikibugs>	 06Project-Admins, 06Release-Engineering-Team (Doing 😎): Add pwangai in acl*Project-Admins to create Milestones in Test Platform - https://phabricator.wikimedia.org/T433515#12170019 (10pwangai) Noted with thanks :)
[05:29:20] <wikibugs>	 (03PS1) 10Phedenskog: Measure specific test commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/1319382 (https://phabricator.wikimedia.org/T432966)
[05:32:33] <wikibugs>	 (03CR) 10Phedenskog: "There's one issue with this that we need to know: It will generate a couple of millions more rows in the database over time, without it we" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1319382 (https://phabricator.wikimedia.org/T432966) (owner: 10Phedenskog)
[08:10:03] <wmf-insecte>	 maintenance-disconnect-full-disks build 828688 integration-agent-docker-1072 (/: 28%, /srv: 95%, /var/lib/docker: 32%): OFFLINE due to disk space
[08:11:47] <wikibugs>	 (03CR) 10Jelto: "@hashar@free.fr yes please proceed with the image build 😊" [integration/config] - 10https://gerrit.wikimedia.org/r/1318680 (https://phabricator.wikimedia.org/T427403) (owner: 10Jelto)
[08:15:03] <wmf-insecte>	 maintenance-disconnect-full-disks build 828689 integration-agent-docker-1072 (/: 28%, /srv: 74%, /var/lib/docker: 32%): RECOVERY disk space OK
[08:15:03] <wmcs-alerts>	 FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates
[08:15:09] <wikibugs>	 10Beta-Cluster-Infrastructure: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://phabricator.wikimedia.org/T433573 (10wmcs-alerts) 03NEW
[08:25:03] <wmcs-alerts>	 RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates
[08:27:55] <wikibugs>	 10Phabricator-Bot-Requests, 06Discovery-Search, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Requesting Phab bot for Machine Learning/Data Engineering team - https://phabricator.wikimedia.org/T433543#12170353 (10Aklapper) See https://www.mediawiki.org/wiki/Phabricator/Bots
[08:53:51] <wikibugs>	 10GitLab (CI & Job Runners), 06collaboration-services: Puppet CA certificate for WMCS puppet server gitlab-runners-puppetserver-01 is about to expire - https://phabricator.wikimedia.org/T433581 (10Jelto) 03NEW
[09:03:03] <wmcs-alerts>	 FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-kafka-jumbo-9 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:03:16] <wikibugs>	 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-kafka-jumbo-9 on project deployment-prep - https://phabricator.wikimedia.org/T433582 (10wmcs-alerts) 03NEW
[09:13:03] <wmcs-alerts>	 FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance deployment-kafka-jumbo-5 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:19:15] <wikibugs>	 (03update) 10jforrester: fundraising: Fix broken Depends reference [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/112 (owner: 10krinkle)
[09:19:23] <wikibugs>	 (03merge) 10jforrester: fundraising: Fix broken Depends reference [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/112 (owner: 10krinkle)
[09:23:03] <wmcs-alerts>	 FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance deployment-kafka-jumbo-5 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:28:03] <wmcs-alerts>	 FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance deployment-kafka-jumbo-5 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[09:30:16] <wikibugs>	 (03CR) 10Hashar: [C:03+2] helm-linter: install helm319 along helm317 [integration/config] - 10https://gerrit.wikimedia.org/r/1318680 (https://phabricator.wikimedia.org/T427403) (owner: 10Jelto)
[09:32:54] <wikibugs>	 (03Merged) 10jenkins-bot: helm-linter: install helm319 along helm317 [integration/config] - 10https://gerrit.wikimedia.org/r/1318680 (https://phabricator.wikimedia.org/T427403) (owner: 10Jelto)
[09:38:50] <wikibugs>	 (03PS2) 10Hashar: jjb: [helm-lint] Use version 0.8.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1319058 (https://phabricator.wikimedia.org/T427403) (owner: 10Jelto)
[09:55:02] <wikibugs>	 (03CR) 10Hashar: [C:03+2] "I amended the commit message first line to match previous entries and added some small changelog about the image." [integration/config] - 10https://gerrit.wikimedia.org/r/1319058 (https://phabricator.wikimedia.org/T427403) (owner: 10Jelto)
[09:56:55] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: [helm-lint] Use version 0.8.3 [integration/config] - 10https://gerrit.wikimedia.org/r/1319058 (https://phabricator.wikimedia.org/T427403) (owner: 10Jelto)
[10:06:36] <hashar>	 jayme: jelto: I have pushed the helm-lint update :)
[10:11:26] <jelto>	 great thanks a lot!
[10:22:12] <wikibugs>	 (03PS1) 10Phedenskog: Measure specific test commands [integration/quibble] - 10https://gerrit.wikimedia.org/r/1319382 (https://phabricator.wikimedia.org/T432966)
[10:26:52] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Re-build integration-cumin.integration.eqiad1.wikimedia.cloud on something newer than bullseye - https://phabricator.wikimedia.org/T433592 (10Jdforrester-WMF) 03NEW
[11:33:03] <wmcs-alerts>	 FIRING: [5x] PuppetAgentNoResources: No Puppet resources found on instance deployment-kafka-jumbo-5 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[11:47:56] <MichaelG_WMF>	 It seems  integration-agent-docker-1085 is out of disk space
[11:49:00] <MichaelG_WMF>	 In https://integration.wikimedia.org/ci/job/quibble-with-gated-extensions-vendor-mysql-php83/49315/console there is `00:01:52.071 gzip: log/mw-debug-cli.log.gz: No space left on device`
[11:50:03] <MichaelG_WMF>	 (not sure whom to ping, hashar  maybe?)
[12:55:09] <kevinbazira>	 hashar: o/ CI is failing with message: "You need helm3.11 to run this task."
[12:55:09] <kevinbazira>	 details: https://integration.wikimedia.org/ci/job/helm-lint/34424/console
[12:55:26] <hashar>	 MichaelG_WMF: I'll clean it
[12:56:12] <MichaelG_WMF>	 Thank you! 🙏
[12:56:21] <hashar>	 kevinbazira: jayme and jelto crafted an update of the helm-lint image which drops helm 3.11 and added 3.19 (3.17 is kept)
[13:00:00] <hashar>	 and the change in the Rakefile to adjust to that is https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1318569
[13:00:09] <hashar>	 which means all deployment-charts are blocked for now
[13:00:36] <hashar>	 I guess we should rollback the Jenkins job to use the previous version of the image (which has 3.11/3.17) until that change is ready
[13:02:27] <hashar>	 kevinbazira: do you need to deploy? Cause I can just rollback the job to the previous version of image and that will unblock you
[13:03:35] <kevinbazira>	 yes, we need to deploy but can't get passed CI :)
[13:05:58] <wikibugs>	 10Diffusion, 10Phabricator, 06Release-Engineering-Team, 07Performance Issue: Diffusion code view: Per-file change info is slow to load despite caching (diffusion.lastmodifiedquery) - https://phabricator.wikimedia.org/T403215#12171252 (10Aklapper) Summarizing my understanding to confuse myself less: * `Diff...
[13:06:31] <jelto>	 I think it's related to this line https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/Rakefile#47 in the rakefile
[13:07:17] <jelto>	 which we would update in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1318569 but a unrelated chart problem blocked this change
[13:10:51] <jelto>	 hashar: I'll merge the change which updates the rakefile, so the check for 3.11 should be dropped
[13:18:03] <wmcs-alerts>	 FIRING: [5x] PuppetAgentNoResources: No Puppet resources found on instance deployment-kafka-jumbo-5 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[13:18:57] <hashar>	 jelto: or feel free to send an another change that just drop the helm3.11 check
[13:19:07] <jelto>	 give me 5 more minutes
[13:19:10] <hashar>	 that gives you time to complete the change that adds 3.19 support https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1318569
[13:19:16] <wmf-insecte>	 Project mediawiki-core-doxygen build #21925: 04FAILURE in 1 min 14 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21925/
[13:26:34] <jelto>	 hashar, kevinbazira: I rebased your change and there is a +2 from jenkins: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1319476
[13:28:03] <wmcs-alerts>	 FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates
[13:34:14] <hashar>	 thanks jelto !
[13:35:51] <wikibugs>	 (03PS1) 10Jforrester: WIP: Docker: [php85] Migrate to Wikimedia-provide binary, cascaded [integration/config] - 10https://gerrit.wikimedia.org/r/1319485 (https://phabricator.wikimedia.org/T433254)
[13:37:44] <wikibugs>	 (03CR) 10CI reject: [V:04-1] WIP: Docker: [php85] Migrate to Wikimedia-provide binary, cascaded [integration/config] - 10https://gerrit.wikimedia.org/r/1319485 (https://phabricator.wikimedia.org/T433254) (owner: 10Jforrester)
[13:38:03] <wmcs-alerts>	 RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates  - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates
[13:39:03] <wmcs-alerts>	 FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-kafka-jumbo-9 in project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[13:39:11] <wikibugs>	 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-kafka-jumbo-9 in project deployment-prep - https://phabricator.wikimedia.org/T433611 (10wmcs-alerts) 03NEW
[13:45:51] <kevinbazira>	 thanks jelto and hashar!
[13:49:03] <wmcs-alerts>	 RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-kafka-jumbo-9 in project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[13:55:58] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/1314574 (https://phabricator.wikimedia.org/T432925) (owner: 10Vaughn Walters)
[13:56:10] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/1316021 (https://phabricator.wikimedia.org/T433069) (owner: 10Vaughn Walters)
[13:56:20] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/1316615 (https://phabricator.wikimedia.org/T433083) (owner: 10Vaughn Walters)
[13:58:03] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: [catalyst-daily-AbuseFilter] Add AbuseFilter job [integration/config] - 10https://gerrit.wikimedia.org/r/1314574 (https://phabricator.wikimedia.org/T432925) (owner: 10Vaughn Walters)
[13:58:12] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: [catalyst-daily-AchievementBadges] Add AchievementBadges job [integration/config] - 10https://gerrit.wikimedia.org/r/1316021 (https://phabricator.wikimedia.org/T433069) (owner: 10Vaughn Walters)
[13:58:21] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: [catalyst-daily-MinervaNeue] Add MinervaNeue job [integration/config] - 10https://gerrit.wikimedia.org/r/1316615 (https://phabricator.wikimedia.org/T433083) (owner: 10Vaughn Walters)
[14:04:57] <wikibugs>	 (03approved) 10aghirelli: Add missing rule to RULE_REFERENCE.md [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/18 (owner: 10kbach)
[14:05:03] <wikibugs>	 (03merge) 10aghirelli: Add missing rule to RULE_REFERENCE.md [repos/ci-tools/wikimedia-spectral-ruleset] - 10https://gitlab.wikimedia.org/repos/ci-tools/wikimedia-spectral-ruleset/-/merge_requests/18 (owner: 10kbach)
[14:12:33] <wmcs-alerts>	 FIRING: [5x] PuppetAgentNoResources: No Puppet resources found on instance deployment-kafka-jumbo-5 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[14:12:40] <wikibugs>	 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-kafka-jumbo-8 on project deployment-prep - https://phabricator.wikimedia.org/T433613 (10wmcs-alerts) 03NEW
[14:17:54] <wikibugs>	 (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/Pride] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1319498
[14:17:55] <wikibugs>	 (03CR) 10QChris: [V:03+2 C:03+2] Allow “Gerrit Managers” to import history [extensions/Pride] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1319498 (owner: 10QChris)
[14:17:58] <wikibugs>	 (03PS1) 10QChris: Import done. Revoke import grants [extensions/Pride] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1319499
[14:17:58] <wikibugs>	 (03CR) 10QChris: [V:03+2 C:03+2] Import done. Revoke import grants [extensions/Pride] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/1319499 (owner: 10QChris)
[14:25:31] <wikibugs>	 (03PS4) 10Vaughn Walters: jjb: [catalyst-daily-PageTriage] Add PageTriage job [integration/config] - 10https://gerrit.wikimedia.org/r/1318803 (https://phabricator.wikimedia.org/T433085)
[14:25:41] <wikibugs>	 (03PS2) 10Vaughn Walters: jjb: [catalyst-daily-TemplateData] Add TemplateData job [integration/config] - 10https://gerrit.wikimedia.org/r/1319173 (https://phabricator.wikimedia.org/T433091)
[14:26:54] <wikibugs>	 (03CR) 10Hashar: [C:03+2] jjb: [catalyst-daily-PageTriage] Add PageTriage job [integration/config] - 10https://gerrit.wikimedia.org/r/1318803 (https://phabricator.wikimedia.org/T433085) (owner: 10Vaughn Walters)
[14:26:55] <wikibugs>	 (03CR) 10Hashar: [C:03+2] jjb: [catalyst-daily-TemplateData] Add TemplateData job [integration/config] - 10https://gerrit.wikimedia.org/r/1319173 (https://phabricator.wikimedia.org/T433091) (owner: 10Vaughn Walters)
[14:28:03] <wmcs-alerts>	 FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-kafka-main-6 in project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[14:28:15] <wikibugs>	 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-kafka-main-6 in project deployment-prep - https://phabricator.wikimedia.org/T433614 (10wmcs-alerts) 03NEW
[14:28:57] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: [catalyst-daily-PageTriage] Add PageTriage job [integration/config] - 10https://gerrit.wikimedia.org/r/1318803 (https://phabricator.wikimedia.org/T433085) (owner: 10Vaughn Walters)
[14:29:00] <wikibugs>	 (03Merged) 10jenkins-bot: jjb: [catalyst-daily-TemplateData] Add TemplateData job [integration/config] - 10https://gerrit.wikimedia.org/r/1319173 (https://phabricator.wikimedia.org/T433091) (owner: 10Vaughn Walters)
[14:29:16] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Traffic: Puppet agent failure detected on instance deployment-urldownloader04 in project deployment-prep - https://phabricator.wikimedia.org/T433510#12171609 (10ssingh) Sorry about that -- I had no idea we had urldownloaders in beta and that the above change would break tho...
[14:34:08] <wmf-insecte>	 Yippee, build fixed!
[14:34:08] <wmf-insecte>	 Project mediawiki-core-doxygen build #21926: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/21926/
[14:38:03] <wmcs-alerts>	 RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance deployment-kafka-main-6 in project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun
[14:57:33] <wmcs-alerts>	 RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-kafka-jumbo-8 on project deployment-prep   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[15:15:28] <wikibugs>	 10Continuous-Integration-Config, 10CSSJanus, 06MediaWiki-Core-Platform-Team, 07Test-Coverage: cssjanus.js has two coverage reports on https://doc.wikimedia.org/cover/ - https://phabricator.wikimedia.org/T424419#12171795 (10Umherirrender) 05Resolved→03Open There are still two coverage reports  https://d...
[15:37:05] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-extensions-ModeratorToolkit, 06Moderator-Tools-Team: Set up continuous integration on the ModeratorToolkit extension - https://phabricator.wikimedia.org/T430991#12171894 (10MusikAnimal)
[15:43:44] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-extensions-ModeratorToolkit, 06Moderator-Tools-Team (Kanban): Set up continuous integration on the ModeratorToolkit extension - https://phabricator.wikimedia.org/T430991#12171912 (10MusikAnimal) a:03MusikAnimal
[15:43:57] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-extensions-ModeratorToolkit, 06Moderator-Tools-Team (Kanban): Set up continuous integration on the ModeratorToolkit extension - https://phabricator.wikimedia.org/T430991#12171914 (10MusikAnimal)
[16:25:33] <wikibugs>	 (03PS1) 10Hslater: Zuul: [mediawiki/extensions/BlueSpiceProDistributionConnector] Add dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1319514
[16:42:37] <taavi>	 hmm, is the https://libraryupgrader.toolforge.org/logs2/2554994 failure caused by our stylelint config disabling rules that somehow just do not exist or is there some other explanation for this that I can't see?
[16:46:32] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 10Castor, 07Spike, 06Test Platform (Nairobi 29): Create a plan on how to make Castor save jobs async - https://phabricator.wikimedia.org/T425411#12172187 (10Mhurd) a:03Mhurd
[16:46:43] <wikibugs>	 (03PS1) 10Sohom Datta: Add Ignacio Rodríguez [integration/config] - 10https://gerrit.wikimedia.org/r/1319520
[16:48:37] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add Ignacio Rodríguez [integration/config] - 10https://gerrit.wikimedia.org/r/1319520 (owner: 10Sohom Datta)
[16:52:59] <wikibugs>	 (03PS2) 10Sohom Datta: Add Ignacio Rodríguez [integration/config] - 10https://gerrit.wikimedia.org/r/1319520
[17:05:30] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/extensions/MarkAsHelpful] Mark as archived (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1281881 (https://phabricator.wikimedia.org/T425234) (owner: 10Lewis Cawte)
[17:05:31] <wikibugs>	 (03PS1) 10Jforrester: Zuul: Correct name of archived extension MarkAsHelpful [integration/config] - 10https://gerrit.wikimedia.org/r/1319526 (https://phabricator.wikimedia.org/T425234)
[17:05:57] <wikibugs>	 (03CR) 10Jforrester: [C:03+2] Zuul: Correct name of archived extension MarkAsHelpful [integration/config] - 10https://gerrit.wikimedia.org/r/1319526 (https://phabricator.wikimedia.org/T425234) (owner: 10Jforrester)
[17:08:12] <wikibugs>	 (03Merged) 10jenkins-bot: Zuul: Correct name of archived extension MarkAsHelpful [integration/config] - 10https://gerrit.wikimedia.org/r/1319526 (https://phabricator.wikimedia.org/T425234) (owner: 10Jforrester)
[17:28:55] <James_F>	 !log jforrester@doc1004:~$ sudo -u doc-uploader rm -rf /srv/doc/cover/mediawiki-libs-node-cssjanus/ # T424419
[17:28:57] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:28:57] <stashbot>	 T424419: cssjanus.js has two coverage reports on https://doc.wikimedia.org/cover/ - https://phabricator.wikimedia.org/T424419
[17:29:33] <wikibugs>	 10Continuous-Integration-Config, 10CSSJanus, 06MediaWiki-Core-Platform-Team, 07Test-Coverage: cssjanus.js has two coverage reports on https://doc.wikimedia.org/cover/ - https://phabricator.wikimedia.org/T424419#12172325 (10Jdforrester-WMF) 05Open→03Resolved I've re-deleted it, let's hope it doesn't...
[19:13:31] <wikibugs>	 (03PS1) 10Ottomata: Zuul, jjb: Add EditCheck Headless image pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1319542 (https://phabricator.wikimedia.org/T433514)
[19:14:59] <wikibugs>	 (03PS2) 10Ottomata: Zuul, jjb: Add EditCheck Headless image pipeline jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1319542 (https://phabricator.wikimedia.org/T433514)
[19:27:47] <wikibugs>	 (03PS1) 10Tpt: Zuul: Add Cite to ProofreadPage dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1319544
[19:28:44] <wikibugs>	 (03PS2) 10Tpt: Zuul: Add Cite to ProofreadPage dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1319544
[19:33:34] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Zuul: Add Cite to ProofreadPage dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1319544 (owner: 10Tpt)
[19:34:03] <wikibugs>	 (03PS6) 10Vaughn Walters: jjb: [catalyst-daily-CentralNotice] Add CentralNotice job [integration/config] - 10https://gerrit.wikimedia.org/r/1312592 (https://phabricator.wikimedia.org/T427006)
[19:34:38] <wikibugs>	 (03CR) 10Vaughn Walters: "This is ready for review. Dependencies have been merged and tests are green." [integration/config] - 10https://gerrit.wikimedia.org/r/1312592 (https://phabricator.wikimedia.org/T427006) (owner: 10Vaughn Walters)
[19:38:37] <wikibugs>	 (03PS3) 10Tpt: Zuul: Add Cite to ProofreadPage dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1319544
[20:00:03] <wmf-insecte>	 maintenance-disconnect-full-disks build 828830 integration-agent-docker-1092 (/: 28%, /srv: 100%, /var/lib/docker: 33%): OFFLINE due to disk space
[20:00:35] <wikibugs>	 06Release-Engineering-Team, 06SRE, 10SRE-Access-Requests: Requesting access to Jenkins server for PWangai-WMF - https://phabricator.wikimedia.org/T433563#12172753 (10Dzahn)
[20:05:05] <wmf-insecte>	 maintenance-disconnect-full-disks build 828831 integration-agent-docker-1092 (/: 28%, /srv: 73%, /var/lib/docker: 33%): RECOVERY disk space OK
[20:16:51] <wikibugs>	 06Release-Engineering-Team, 06SRE, 10SRE-Access-Requests: Requesting access to Jenkins server for PWangai-WMF - https://phabricator.wikimedia.org/T433563#12172791 (10Dzahn) Hi releng team,  so the actual request here is "collect castor metrics".  Given our recent split of the CI server roles let's confirm it...
[20:33:03] <wmcs-alerts>	 FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[20:33:15] <wikibugs>	 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cache-text08 is down - https://phabricator.wikimedia.org/T433641 (10wmcs-alerts) 03NEW
[20:38:59] <wikibugs>	 (03open) 10damilare: Add redis to list of dependencies of fundraising ml trixie [repos/releng/dev-images] - 10https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/114 (https://phabricator.wikimedia.org/T430711)
[20:43:03] <wmcs-alerts>	 RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown
[20:53:24] <wikibugs>	 06Release-Engineering-Team, 06SRE, 10SRE-Access-Requests: Requesting access to Jenkins server for PWangai-WMF - https://phabricator.wikimedia.org/T433563#12172940 (10Dzahn) 05Open→03In progress
[20:53:41] <wikibugs>	 (03open) 10dancy: main.py: Handle empty baremetal target lists in a couple of places [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1232
[21:05:39] <wikibugs>	 (03merge) 10dancy: main.py: Handle empty baremetal target lists in a couple of places [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1232
[21:06:44] <wikibugs>	 (03update) 10dancy: Release 4.276.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1233
[21:06:45] <wikibugs>	 (03open) 10dancy: Release 4.276.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1233
[21:09:46] <wikibugs>	 (03merge) 10dancy: Release 4.276.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1233
[21:24:46] <wikibugs>	 (03open) 10dancy: Drop delay_messageblobstore_purge [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1234 (https://phabricator.wikimedia.org/T263872)
[21:25:31] <wikibugs>	 (03update) 10dancy: Drop delay_messageblobstore_purge [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1234 (https://phabricator.wikimedia.org/T263872)
[21:27:58] <wikibugs>	 (03merge) 10dancy: Drop delay_messageblobstore_purge [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1234 (https://phabricator.wikimedia.org/T263872)
[21:29:09] <wikibugs>	 (03open) 10dancy: Release 4.276.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1235
[21:30:34] <wikibugs>	 (03merge) 10dancy: Release 4.276.1 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1235
[21:36:29] <wikibugs>	 06Release-Engineering-Team, 06SRE, 10SRE-Access-Requests: Requesting access to Jenkins server for PWangai-WMF - https://phabricator.wikimedia.org/T433563#12173008 (10pwangai) I would say this may not be a temporary one-off. Right now as a team we are formulating a KR revolving around castor improvements, and...
[21:52:13] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Observability-Logging: scap on deployment-deploy04 failing to reach logging-logstash-04.logging.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T432254#12173031 (10colewhite) 05Open→03Resolved I've not seen any more OOMs since lowering Xmx on the logstash hos...
[21:57:12] <wikibugs>	 (03update) 10dancy: mwscript: Treat any script stderr as unexpected output [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1226 (https://phabricator.wikimedia.org/T433104)
[21:57:38] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Observability-Logging: scap on deployment-deploy04 failing to reach logging-logstash-04.logging.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T432254#12173038 (10dancy) Thanks @colewhite!
[21:58:02] <wikibugs>	 (03update) 10dancy: mwscript: Treat any script stderr as unexpected output [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1226 (https://phabricator.wikimedia.org/T433104)
[22:00:35] <wikibugs>	 (03merge) 10dancy: mwscript: Treat any script stderr as unexpected output [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1226 (https://phabricator.wikimedia.org/T433104)
[22:02:21] <wikibugs>	 (03update) 10dancy: Release 4.277.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1236
[22:02:22] <wikibugs>	 (03open) 10dancy: Release 4.277.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1236
[22:03:53] <wikibugs>	 (03merge) 10dancy: Release 4.277.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1236
[23:26:42] <wikibugs>	 10Beta-Cluster-Infrastructure: Last Puppet run was over 24 hours ago on instance deployment-docker-mobileapps02 in project deployment-prep - https://phabricator.wikimedia.org/T433496#12173159 (10bd808) 05Open→03Resolved a:03bd808 I kept this open so I could also cleanup the remaining huge syslog file a...