[02:28:09] (03CR) 10Arlolra: [C: 03+2] Config for running diffs with core-integrated Parsoid [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/933461 (owner: 10Subramanya Sastry) [02:28:14] (03CR) 10Arlolra: [C: 03+2] Reorg configs and helper files [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/933466 (owner: 10Subramanya Sastry) [02:28:54] (03Merged) 10jenkins-bot: Config for running diffs with core-integrated Parsoid [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/933461 (owner: 10Subramanya Sastry) [02:28:56] (03Merged) 10jenkins-bot: Reorg configs and helper files [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/933466 (owner: 10Subramanya Sastry) [02:29:00] (03Merged) 10jenkins-bot: Remove stale version code [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/933601 (owner: 10Subramanya Sastry) [04:52:59] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 2 others: Re-start Wikibase test coverage reporting - https://phabricator.wikimedia.org/T288396 (10tstarling) The way extension coverage works was not obvious to me and took a... [05:06:07] 10Continuous-Integration-Config, 10MediaWiki-extensions-WikibaseClient, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 2 others: Re-start Wikibase test coverage reporting - https://phabricator.wikimedia.org/T288396 (10tstarling) I don't know why mwext-phpunit-coverage is a shell script instead... [08:28:07] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests): Rebuild WMCS integration instances to larger flavor - https://phabricator.wikimedia.org/T340070 (10hashar) [09:45:24] 10Project-Admins: Create project tag for "IP blocking" - https://phabricator.wikimedia.org/T340487 (10kostajh) >>! In T340487#8965851, @Peachey88 wrote: > Could the name be reconsidered?, At the moment it reads as anything to do with IP blocks, which would lead to confusion with #mediawiki-blocks I don't have... [09:56:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests): Rebuild WMCS integration instances to larger flavor - https://phabricator.wikimedia.org/T340070 (10hashar) >>! In T340070#8977464, @hashar wrote: > I have changed the flavor for [[ https://horizon.wikimedia.org... [10:21:33] 10Project-Admins: Create project tag for HotCat - https://phabricator.wikimedia.org/T340711 (10Unite_together) Well, I asked @TheDJ and learned that there's actually no "team" as a whole and it has been unmaintained for quite long... [10:57:08] Do "gated extensions" listed in parameter_functions.py actually get cloned and tested when a patch is merged? I'm having trouble finding an example job where this actually happens. [10:57:46] awight: yes. That is applied to the jobs named `wmf-quibble*` [10:58:18] awight: typically https://integration.wikimedia.org/ci/job/wmf-quibble-vendor-mysql-php74-docker/ [10:58:21] I am off for lunch [10:58:50] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T340243 (10TheDJ) It seems I introduced a video playback bug with {T340816}. I'll schedule a back port of the fix for... [11:01:11] hashar: Thanks, that's quite clearly working then :-D [11:25:01] (03PS1) 10JMeybohm: helm-linter: Switch to envoy 1.23.10 [integration/config] - 10https://gerrit.wikimedia.org/r/934509 (https://phabricator.wikimedia.org/T300324) [11:25:04] (03PS1) 10JMeybohm: Update to new helm-linter image with envoy 1.23.10 [integration/config] - 10https://gerrit.wikimedia.org/r/934510 [11:31:42] 10Continuous-Integration-Config, 10Growth-Team, 10PageTriage, 10Moderator-Tools-Team (Kanban): Get Jest to run in PageTriage continuous integration - https://phabricator.wikimedia.org/T335542 (10Samwalton9) 05In progress→03Resolved a:03Samwalton9 [11:53:45] 10GitLab (CI & Job Runners), 10Data-Engineering, 10Performance Issue: Improve speed of Gitlab CI - https://phabricator.wikimedia.org/T311111 (10hashar) I am adding back #data-engineering since the slowness comes from how the test environments have been defined. See my analysis from last year above: T311111#8... [12:20:10] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [wikimedia/fundraising/cv] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/934526 [12:20:14] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [wikimedia/fundraising/cv] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/934526 (owner: 10QChris) [12:20:20] (03PS1) 10QChris: Import done. Revoke import grants [wikimedia/fundraising/cv] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/934527 [12:20:26] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [wikimedia/fundraising/cv] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/934527 (owner: 10QChris) [12:24:32] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [wikimedia/fundraising/coworker] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/934528 [12:24:36] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [wikimedia/fundraising/coworker] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/934528 (owner: 10QChris) [12:24:42] (03PS1) 10QChris: Import done. Revoke import grants [wikimedia/fundraising/coworker] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/934529 [12:24:48] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [wikimedia/fundraising/coworker] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/934529 (owner: 10QChris) [12:40:43] 10GitLab: [gitlab runner] Jobs failing on runner gitlab-runner-cbdb8c847-42dxt with Post "https://10.245.0.1:443/api/v1/namespaces/gitlab-runner/secrets": EOF - https://phabricator.wikimedia.org/T340841 (10dcaro) [12:55:21] 10GitLab (CI & Job Runners), 10Data-Engineering, 10Performance Issue: Improve speed of Gitlab CI - https://phabricator.wikimedia.org/T311111 (10xcollazo) I think @Antoine_Quhen took care of this via custom docker images. Right Antoine? [13:16:15] 10MediaWiki-Releasing, 10Documentation, 10MW-1.40-release: Write and send release announcement for 1.40.0 - https://phabricator.wikimedia.org/T329090 (10Jdforrester-WMF) [14:33:29] 10Project-Admins: Create project tag for HotCat - https://phabricator.wikimedia.org/T340711 (10Aklapper) I see... If someone plans to actively maintain it and want to use Phabricator as a central place for planning work and tracking tasks instead of on-wiki places, they are free to request a project tag of course! [15:00:03] 10phabricator maintenance bot: Make it possible to add reviewers automatically to patches uploaded by Maintenance bot - https://phabricator.wikimedia.org/T340796 (10Dzahn) If we add "langlist.tmpl" to the regex of at least one user then we should have the same effect without having to write new code. Basically t... [16:03:43] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review, 10Wikimedia-Incident: Warn against potentially deploying a config patch NOT specified - https://phabricator.wikimedia.org/T338168 (10dancy) 05Open→03Resolved p:05Triage→03Medium [16:06:21] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review, 10Wikimedia-Incident: Warn against potentially deploying a config patch NOT specified - https://phabricator.wikimedia.org/T338168 (10dancy) [16:31:05] 10Beta-Cluster-Infrastructure, 10Data Engineering and Event Platform Team, 10Data-Engineering, 10Event-Platform, 10MW-1.41-notes (1.41.0-wmf.12; 2023-06-06): cirrusSearchCheckerJob JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322491 (10JArguello-WMF) [16:31:57] 10Continuous-Integration-Config, 10Release-Engineering-Team (Radar), 10ChangeProp, 10Data Engineering and Event Platform Team, and 6 others: Run EventBus tests in MediaWiki core CI - https://phabricator.wikimedia.org/T257583 (10JArguello-WMF) [16:34:21] 10Release-Engineering-Team (Radar), 10Data Engineering and Event Platform Team, 10Data-Engineering, 10Event-Platform: Stop using puppet + git pull for auto deployment of schema repos - https://phabricator.wikimedia.org/T274901 (10JArguello-WMF) [16:57:34] 10MediaWiki-Releasing, 10MW-1.40-release, 10Release: Write release announcement for 1.40.0 - https://phabricator.wikimedia.org/T340877 (10Reedy) [16:57:46] (03PS1) 10Jforrester: Zuul: Drop REL1_38 pipeline, branch is EOL [integration/config] - 10https://gerrit.wikimedia.org/r/934589 (https://phabricator.wikimedia.org/T330174) [16:57:48] (03PS1) 10Jforrester: jjb: Drop REL1_38 MediaWiki jobs [integration/config] - 10https://gerrit.wikimedia.org/r/934590 (https://phabricator.wikimedia.org/T330174) [16:59:36] (03CR) 10Jforrester: [C: 03+2] Zuul: Drop REL1_38 pipeline, branch is EOL [integration/config] - 10https://gerrit.wikimedia.org/r/934589 (https://phabricator.wikimedia.org/T330174) (owner: 10Jforrester) [17:01:03] 10MediaWiki-Releasing, 10Documentation, 10MW-1.40-release: Write and send release announcement for 1.40.0 - https://phabricator.wikimedia.org/T329090 (10Reedy) [17:01:07] 10MediaWiki-Releasing, 10MW-1.40-release, 10Release: Write release announcement for 1.40.0 - https://phabricator.wikimedia.org/T340877 (10Reedy) [17:01:13] 10MediaWiki-Releasing, 10Documentation, 10MW-1.40-release: Write and send release announcement for 1.40.0 - https://phabricator.wikimedia.org/T329090 (10Reedy) [17:01:33] (03Merged) 10jenkins-bot: Zuul: Drop REL1_38 pipeline, branch is EOL [integration/config] - 10https://gerrit.wikimedia.org/r/934589 (https://phabricator.wikimedia.org/T330174) (owner: 10Jforrester) [17:01:35] upgrading gitab on replica/passive host [17:01:55] 10MediaWiki-Releasing, 10Documentation, 10MW-1.40-release: Write and send release announcement for 1.40.0 - https://phabricator.wikimedia.org/T329090 (10Reedy) [17:03:24] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/934590 (https://phabricator.wikimedia.org/T330174) (owner: 10Jforrester) [17:04:45] 10MediaWiki-Releasing, 10Documentation, 10MW-1.40-release: Write and send release announcement for 1.40.0 - https://phabricator.wikimedia.org/T329090 (10Reedy) 05In progress→03Resolved [17:04:47] 10MediaWiki-Releasing, 10MW-1.40-release, 10Release: Release MW 1.40.0 - https://phabricator.wikimedia.org/T329075 (10Reedy) [17:04:53] 10MediaWiki-Releasing, 10Documentation, 10MW-1.40-notes, 10MW-1.40-release, 10MW-1.41-notes (1.41.0-wmf.4; 2023-04-10): Finish up RELEASE-NOTES for 1.40.0 - https://phabricator.wikimedia.org/T329082 (10Reedy) 05Open→03Resolved [17:04:57] 10MediaWiki-Releasing, 10MW-1.40-release, 10Release: Release MW 1.40.0 - https://phabricator.wikimedia.org/T329075 (10Reedy) [17:05:29] (03Merged) 10jenkins-bot: jjb: Drop REL1_38 MediaWiki jobs [integration/config] - 10https://gerrit.wikimedia.org/r/934590 (https://phabricator.wikimedia.org/T330174) (owner: 10Jforrester) [17:08:11] 10MediaWiki-Releasing, 10MW-1.40-release, 10Release: Release MW 1.40.0 - https://phabricator.wikimedia.org/T329075 (10Reedy) 05Open→03Resolved [17:44:14] 10GitLab (CI & Job Runners), 10Data Engineering and Event Platform Team: Allow a shared, protected runner for the data-engineering group in GitLab - https://phabricator.wikimedia.org/T295045 (10JArguello-WMF) [17:44:50] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10API Platform, 10Anti-Harassment, and 18 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10JArguello-WMF) [17:49:05] (03PS1) 10Subramanya Sastry: Shrink custom CSS needed now that CSS have been updated on wikis [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/934596 [17:53:20] (03CR) 10Subramanya Sastry: Shrink custom CSS needed now that CSS have been updated on wikis (033 comments) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/934596 (owner: 10Subramanya Sastry) [18:00:02] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T340243 (10brennen) > It seems I introduced a video playback bug with T340816: Unable to load media player on mobile d... [18:24:31] (03PS2) 10Jforrester: Docker: [helm-linter] Switch to envoy 1.23.10 [integration/config] - 10https://gerrit.wikimedia.org/r/934509 (https://phabricator.wikimedia.org/T300324) (owner: 10JMeybohm) [18:27:52] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T340243 (10brennen) 05Open→03Resolved Deployed fix for T340816. Otherwise looks stable on all wikis. [18:27:55] (03CR) 10Jforrester: [C: 03+2] Docker: [helm-linter] Switch to envoy 1.23.10 [integration/config] - 10https://gerrit.wikimedia.org/r/934509 (https://phabricator.wikimedia.org/T300324) (owner: 10JMeybohm) [18:29:33] (03Merged) 10jenkins-bot: Docker: [helm-linter] Switch to envoy 1.23.10 [integration/config] - 10https://gerrit.wikimedia.org/r/934509 (https://phabricator.wikimedia.org/T300324) (owner: 10JMeybohm) [18:33:39] !log Docker: [helm-linter] Switch to envoy 1.23.10 for T300324 [18:33:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:33:42] T300324: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 [18:38:17] (03PS2) 10Jforrester: jjb: [helm-lint] Update to new helm-linter image with envoy 1.23.10 [integration/config] - 10https://gerrit.wikimedia.org/r/934510 (https://phabricator.wikimedia.org/T300324) (owner: 10JMeybohm) [18:40:20] (03CR) 10Jforrester: [C: 03+2] "Deployed and a test job ran successfully." [integration/config] - 10https://gerrit.wikimedia.org/r/934510 (https://phabricator.wikimedia.org/T300324) (owner: 10JMeybohm) [18:42:03] (03Merged) 10jenkins-bot: jjb: [helm-lint] Update to new helm-linter image with envoy 1.23.10 [integration/config] - 10https://gerrit.wikimedia.org/r/934510 (https://phabricator.wikimedia.org/T300324) (owner: 10JMeybohm) [18:47:36] (03PS2) 10Jforrester: Docker: [node14] Update node from 14.17.5 to 14.21.3 [integration/config] - 10https://gerrit.wikimedia.org/r/902104 (https://phabricator.wikimedia.org/T332387) (owner: 10Kosta Harlan) [18:48:01] (03CR) 10Jforrester: [C: 03+2] Docker: [node14] Update node from 14.17.5 to 14.21.3 [integration/config] - 10https://gerrit.wikimedia.org/r/902104 (https://phabricator.wikimedia.org/T332387) (owner: 10Kosta Harlan) [18:49:07] (03Merged) 10jenkins-bot: Docker: [node14] Update node from 14.17.5 to 14.21.3 [integration/config] - 10https://gerrit.wikimedia.org/r/902104 (https://phabricator.wikimedia.org/T332387) (owner: 10Kosta Harlan) [18:49:29] !log Docker: [node14] Update node from 14.17.5 to 14.21.3 for T332387 [18:49:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:49:32] T332387: Quibble: Update nodeJS to at least v14.18.0 - https://phabricator.wikimedia.org/T332387 [19:02:06] thank you, James_F! [19:02:40] kostajh: Thank me in an hour when it's done. :-) [19:14:57] (03PS1) 10Jforrester: jjb: [node16*] Update to images with node 16.19.1 [integration/config] - 10https://gerrit.wikimedia.org/r/934603 [19:14:59] (03PS1) 10Jforrester: jjb: [selenium-daily-*] Update to images with node 14.21.3 [integration/config] - 10https://gerrit.wikimedia.org/r/934604 (https://phabricator.wikimedia.org/T332387) [19:15:01] (03PS1) 10Jforrester: jjb: [quibble*] Update to images with node 14.21.3 [integration/config] - 10https://gerrit.wikimedia.org/r/934605 (https://phabricator.wikimedia.org/T332387) [19:15:23] (03CR) 10Jforrester: [C: 03+2] "Deployed. Whoops, should have done this in February alongside f226465ecb600bc177dea281d30d1127d1d3372c!" [integration/config] - 10https://gerrit.wikimedia.org/r/934603 (owner: 10Jforrester) [19:16:15] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/934604 (https://phabricator.wikimedia.org/T332387) (owner: 10Jforrester) [19:16:31] (03Merged) 10jenkins-bot: jjb: [node16*] Update to images with node 16.19.1 [integration/config] - 10https://gerrit.wikimedia.org/r/934603 (owner: 10Jforrester) [19:17:28] (03Merged) 10jenkins-bot: jjb: [selenium-daily-*] Update to images with node 14.21.3 [integration/config] - 10https://gerrit.wikimedia.org/r/934604 (https://phabricator.wikimedia.org/T332387) (owner: 10Jforrester) [19:54:09] upgrading gitlab version now on prod server [19:54:10] 10GitLab, 10Release-Engineering-Team (Priority Backlog 📥): WMCS GitLab runners running frequently running out of disk space - https://phabricator.wikimedia.org/T340887 (10dduvall) [19:54:17] after replicas had no issues [19:55:38] mutante: thx [19:56:09] please so pushes to gitlab right now if possible [19:56:13] it's making the backup now [19:56:19] s/so/no [20:07:53] 10GitLab (CI & Job Runners), 10commit-message-validator, 10Patch-For-Review, 10User-bd808: Update commit-message-validator to work nicely with GitLab repos - https://phabricator.wikimedia.org/T339307 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/repos/ci-tools/commit-message-validator/-/merge... [20:08:09] 10GitLab (CI & Job Runners), 10commit-message-validator, 10Patch-For-Review, 10User-bd808: Update commit-message-validator to work nicely with GitLab repos - https://phabricator.wikimedia.org/T339307 (10CodeReviewBot) [20:24:18] (03CR) 10Jforrester: [C: 03+2] "Deployed. All seems well so far." [integration/config] - 10https://gerrit.wikimedia.org/r/934605 (https://phabricator.wikimedia.org/T332387) (owner: 10Jforrester) [20:25:42] (03CR) 10Jforrester: [C: 03+2] "Let's do this." [integration/config] - 10https://gerrit.wikimedia.org/r/931687 (owner: 10Jforrester) [20:25:59] 10GitLab, 10Release-Engineering-Team (Priority Backlog 📥): WMCS GitLab runners running frequently running out of disk space - https://phabricator.wikimedia.org/T340887 (10dduvall) 05Open→03In progress p:05Triage→03Medium a:03dduvall After poking around a bit on `runner-1029.gitlab-runners.eqiad1.wiki... [20:26:03] (03Merged) 10jenkins-bot: jjb: [quibble*] Update to images with node 14.21.3 [integration/config] - 10https://gerrit.wikimedia.org/r/934605 (https://phabricator.wikimedia.org/T332387) (owner: 10Jforrester) [20:27:10] (03CR) 10Jforrester: "[This is still blocked on T256626.]" [integration/config] - 10https://gerrit.wikimedia.org/r/894125 (owner: 10Jforrester) [20:27:22] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Flow] Drop from extension gate [integration/config] - 10https://gerrit.wikimedia.org/r/931687 (owner: 10Jforrester) [20:27:39] !log Zuul: [mediawiki/extensions/Flow] Drop from extension gate [20:27:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:27:56] 10GitLab, 10Release-Engineering-Team (Priority Backlog 📥): WMCS GitLab runners running frequently running out of disk space - https://phabricator.wikimedia.org/T340887 (10dduvall) Just adding that `buildctl prune` does not remove the cache entries either, which confirms there are indeed not functionally reclai... [20:28:12] (03CR) 10Kosta Harlan: "I'm a bit worried that this means breakages in Flow won't get seen until they're in production." [integration/config] - 10https://gerrit.wikimedia.org/r/931687 (owner: 10Jforrester) [20:30:16] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab, 10Patch-For-Review: contint2002 service implementation tracking - https://phabricator.wikimedia.org/T324659 (10Dzahn) https://wikitech.wikimedia.org/wiki/Contint2001 and https://wikitech.wikimedia.org/wiki/Contint2002 with fingerprint p... [20:38:06] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab, 10Patch-For-Review: contint2002 service implementation tracking - https://phabricator.wikimedia.org/T324659 (10Dzahn) https://www.mediawiki.org/wiki/Continuous_integration/Data_center_switch exists but at least one thing is outdated. it... [20:38:08] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble: Quibble: Update nodeJS to at least v14.18.0 - https://phabricator.wikimedia.org/T332387 (10Jdforrester-WMF) 05Open→03Resolved Provisionally done. Please shout if anything's broken! [20:45:32] maintenance-disconnect-full-disks build 504682 integration-agent-docker-1038 (/: 29%, /srv: 12%, /var/lib/docker: 98%): OFFLINE due to disk space [20:50:28] maintenance-disconnect-full-disks build 504683 integration-agent-docker-1038 (/: 29%, /srv: 12%, /var/lib/docker: 82%): RECOVERY disk space OK [20:53:52] 10GitLab (CI & Job Runners), 10commit-message-validator, 10Patch-For-Review, 10User-bd808: Update commit-message-validator to work nicely with GitLab repos - https://phabricator.wikimedia.org/T339307 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/repos/ci-tools/commit-message-validator/-/merge... [21:11:18] maintenance-disconnect-full-disks build 504687 integration-agent-docker-1028 (/: 30%, /srv: 28%, /var/lib/docker: 99%): OFFLINE due to disk space [21:11:19] maintenance-disconnect-full-disks build 504687 integration-agent-docker-1038 (/: 29%, /srv: 27%, /var/lib/docker: 100%): OFFLINE due to disk space [21:15:41] maintenance-disconnect-full-disks build 504688 integration-agent-docker-1028 (/: 30%, /srv: 18%, /var/lib/docker: 87%): RECOVERY disk space OK [21:25:58] maintenance-disconnect-full-disks build 504690 integration-agent-docker-1038 (/: 29%, /srv: 14%, /var/lib/docker: 100%): RECOVERY disk space OK [21:30:37] maintenance-disconnect-full-disks build 504691 integration-agent-docker-1038 (/: 29%, /srv: 14%, /var/lib/docker: 98%): OFFLINE due to disk space [21:40:57] maintenance-disconnect-full-disks build 504693 integration-agent-docker-1026 (/: 29%, /srv: 29%, /var/lib/docker: 96%): OFFLINE due to disk space [21:50:57] maintenance-disconnect-full-disks build 504695 integration-agent-docker-1026 (/: 29%, /srv: 11%, /var/lib/docker: 94%): RECOVERY disk space OK [21:50:57] maintenance-disconnect-full-disks build 504695 integration-agent-docker-1031 (/: 29%, /srv: 19%, /var/lib/docker: 95%): OFFLINE due to disk space [21:50:57] maintenance-disconnect-full-disks build 504695 integration-agent-docker-1038 (/: 29%, /srv: 14%, /var/lib/docker: 95%): still OFFLINE due to disk space [21:56:32] maintenance-disconnect-full-disks build 504696 integration-agent-docker-1026 (/: 29%, /srv: 12%, /var/lib/docker: 96%): OFFLINE due to disk space [22:06:44] 10GitLab, 10Release-Engineering-Team (Priority Backlog 📥): WMCS GitLab runners running frequently running out of disk space - https://phabricator.wikimedia.org/T340887 (10dancy) I'm very curious to learn what you discover. [22:15:46] maintenance-disconnect-full-disks build 504700 integration-agent-docker-1026 (/: 29%, /srv: 12%, /var/lib/docker: 99%): still OFFLINE due to disk space [22:15:46] maintenance-disconnect-full-disks build 504700 integration-agent-docker-1030 (/: 30%, /srv: 33%, /var/lib/docker: 96%): OFFLINE due to disk space [22:15:46] maintenance-disconnect-full-disks build 504700 integration-agent-docker-1031 (/: 29%, /srv: 19%, /var/lib/docker: 95%): still OFFLINE due to disk space [22:15:47] maintenance-disconnect-full-disks build 504700 integration-agent-docker-1038 (/: 29%, /srv: 14%, /var/lib/docker: 95%): still OFFLINE due to disk space [22:25:43] maintenance-disconnect-full-disks build 504702 integration-agent-docker-1035 (/: 31%, /srv: 15%, /var/lib/docker: 100%): OFFLINE due to disk space [22:30:51] maintenance-disconnect-full-disks build 504703 integration-agent-docker-1034 (/: 29%, /srv: 19%, /var/lib/docker: 100%): OFFLINE due to disk space [22:35:45] maintenance-disconnect-full-disks build 504704 integration-agent-docker-1034 (/: 29%, /srv: 15%, /var/lib/docker: 95%): RECOVERY disk space OK [22:37:20] 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab, 10Patch-For-Review: contint2002 service implementation tracking - https://phabricator.wikimedia.org/T324659 (10Dzahn) a:05Dzahn→03None [22:40:49] maintenance-disconnect-full-disks build 504705 integration-agent-docker-1026 (/: 29%, /srv: 12%, /var/lib/docker: 99%): still OFFLINE due to disk space [22:40:49] maintenance-disconnect-full-disks build 504705 integration-agent-docker-1028 (/: 30%, /srv: 19%, /var/lib/docker: 95%): OFFLINE due to disk space [22:40:50] maintenance-disconnect-full-disks build 504705 integration-agent-docker-1030 (/: 30%, /srv: 33%, /var/lib/docker: 96%): still OFFLINE due to disk space [22:40:50] maintenance-disconnect-full-disks build 504705 integration-agent-docker-1031 (/: 29%, /srv: 19%, /var/lib/docker: 95%): still OFFLINE due to disk space [22:40:50] maintenance-disconnect-full-disks build 504705 integration-agent-docker-1034 (/: 29%, /srv: 15%, /var/lib/docker: 96%): OFFLINE due to disk space [22:40:51] maintenance-disconnect-full-disks build 504705 integration-agent-docker-1035 (/: 31%, /srv: 15%, /var/lib/docker: 100%): still OFFLINE due to disk space [22:40:51] maintenance-disconnect-full-disks build 504705 integration-agent-docker-1037 (/: 29%, /srv: 15%, /var/lib/docker: 100%): OFFLINE due to disk space [22:40:52] maintenance-disconnect-full-disks build 504705 integration-agent-docker-1038 (/: 29%, /srv: 14%, /var/lib/docker: 95%): still OFFLINE due to disk space [22:45:46] maintenance-disconnect-full-disks build 504706 integration-agent-docker-1037 (/: 29%, /srv: 15%, /var/lib/docker: 85%): RECOVERY disk space OK [23:06:04] maintenance-disconnect-full-disks build 504710 integration-agent-docker-1026 (/: 29%, /srv: 12%, /var/lib/docker: 99%): still OFFLINE due to disk space [23:06:04] maintenance-disconnect-full-disks build 504710 integration-agent-docker-1028 (/: 30%, /srv: 19%, /var/lib/docker: 95%): still OFFLINE due to disk space [23:06:04] maintenance-disconnect-full-disks build 504710 integration-agent-docker-1029 (/: 29%, /srv: 16%, /var/lib/docker: 95%): OFFLINE due to disk space [23:06:04] maintenance-disconnect-full-disks build 504710 integration-agent-docker-1030 (/: 30%, /srv: 33%, /var/lib/docker: 96%): still OFFLINE due to disk space [23:06:05] maintenance-disconnect-full-disks build 504710 integration-agent-docker-1031 (/: 29%, /srv: 19%, /var/lib/docker: 95%): still OFFLINE due to disk space [23:06:05] maintenance-disconnect-full-disks build 504710 integration-agent-docker-1033 (/: 29%, /srv: 13%, /var/lib/docker: 97%): OFFLINE due to disk space [23:06:06] maintenance-disconnect-full-disks build 504710 integration-agent-docker-1034 (/: 29%, /srv: 15%, /var/lib/docker: 96%): still OFFLINE due to disk space [23:06:06] maintenance-disconnect-full-disks build 504710 integration-agent-docker-1035 (/: 31%, /srv: 15%, /var/lib/docker: 100%): still OFFLINE due to disk space [23:06:07] maintenance-disconnect-full-disks build 504710 integration-agent-docker-1036 (/: 30%, /srv: 18%, /var/lib/docker: 98%): OFFLINE due to disk space [23:06:07] maintenance-disconnect-full-disks build 504710 integration-agent-docker-1037 (/: 29%, /srv: 15%, /var/lib/docker: 97%): OFFLINE due to disk space [23:06:08] maintenance-disconnect-full-disks build 504710 integration-agent-docker-1038 (/: 29%, /srv: 14%, /var/lib/docker: 95%): still OFFLINE due to disk space [23:10:59] maintenance-disconnect-full-disks build 504711 integration-agent-docker-1023 (/: 30%, /srv: 37%, /var/lib/docker: 100%): OFFLINE due to disk space [23:10:59] maintenance-disconnect-full-disks build 504711 integration-agent-docker-1027 (/: 29%, /srv: 29%, /var/lib/docker: 96%): OFFLINE due to disk space [23:16:02] maintenance-disconnect-full-disks build 504712 integration-agent-docker-1023 (/: 30%, /srv: 20%, /var/lib/docker: 100%): RECOVERY disk space OK [23:16:02] maintenance-disconnect-full-disks build 504712 integration-agent-docker-1027 (/: 29%, /srv: 14%, /var/lib/docker: 96%): RECOVERY disk space OK [23:30:58] maintenance-disconnect-full-disks build 504715 integration-agent-docker-1023 (/: 30%, /srv: 22%, /var/lib/docker: 99%): OFFLINE due to disk space [23:30:58] maintenance-disconnect-full-disks build 504715 integration-agent-docker-1026 (/: 29%, /srv: 12%, /var/lib/docker: 99%): still OFFLINE due to disk space [23:30:58] maintenance-disconnect-full-disks build 504715 integration-agent-docker-1028 (/: 30%, /srv: 19%, /var/lib/docker: 95%): still OFFLINE due to disk space [23:30:58] maintenance-disconnect-full-disks build 504715 integration-agent-docker-1029 (/: 29%, /srv: 16%, /var/lib/docker: 99%): still OFFLINE due to disk space [23:30:59] maintenance-disconnect-full-disks build 504715 integration-agent-docker-1030 (/: 30%, /srv: 33%, /var/lib/docker: 96%): still OFFLINE due to disk space [23:30:59] maintenance-disconnect-full-disks build 504715 integration-agent-docker-1031 (/: 29%, /srv: 19%, /var/lib/docker: 95%): still OFFLINE due to disk space [23:31:00] maintenance-disconnect-full-disks build 504715 integration-agent-docker-1033 (/: 29%, /srv: 13%, /var/lib/docker: 96%): still OFFLINE due to disk space [23:31:01] maintenance-disconnect-full-disks build 504715 integration-agent-docker-1034 (/: 29%, /srv: 15%, /var/lib/docker: 96%): still OFFLINE due to disk space [23:31:01] maintenance-disconnect-full-disks build 504715 integration-agent-docker-1035 (/: 31%, /srv: 15%, /var/lib/docker: 100%): still OFFLINE due to disk space [23:31:02] maintenance-disconnect-full-disks build 504715 integration-agent-docker-1036 (/: 30%, /srv: 18%, /var/lib/docker: 99%): still OFFLINE due to disk space [23:31:02] maintenance-disconnect-full-disks build 504715 integration-agent-docker-1037 (/: 29%, /srv: 15%, /var/lib/docker: 97%): still OFFLINE due to disk space [23:31:03] maintenance-disconnect-full-disks build 504715 integration-agent-docker-1038 (/: 29%, /srv: 14%, /var/lib/docker: 95%): still OFFLINE due to disk space [23:36:02] maintenance-disconnect-full-disks build 504716 integration-agent-docker-1023 (/: 30%, /srv: 20%, /var/lib/docker: 99%): RECOVERY disk space OK [23:55:53] maintenance-disconnect-full-disks build 504720 integration-agent-docker-1026 (/: 29%, /srv: 12%, /var/lib/docker: 99%): still OFFLINE due to disk space [23:55:53] maintenance-disconnect-full-disks build 504720 integration-agent-docker-1028 (/: 30%, /srv: 19%, /var/lib/docker: 95%): still OFFLINE due to disk space [23:55:53] maintenance-disconnect-full-disks build 504720 integration-agent-docker-1029 (/: 29%, /srv: 16%, /var/lib/docker: 99%): still OFFLINE due to disk space [23:55:54] maintenance-disconnect-full-disks build 504720 integration-agent-docker-1030 (/: 30%, /srv: 33%, /var/lib/docker: 96%): still OFFLINE due to disk space [23:55:54] maintenance-disconnect-full-disks build 504720 integration-agent-docker-1031 (/: 29%, /srv: 19%, /var/lib/docker: 95%): still OFFLINE due to disk space [23:55:54] maintenance-disconnect-full-disks build 504720 integration-agent-docker-1033 (/: 29%, /srv: 13%, /var/lib/docker: 96%): still OFFLINE due to disk space [23:55:55] maintenance-disconnect-full-disks build 504720 integration-agent-docker-1034 (/: 29%, /srv: 15%, /var/lib/docker: 96%): still OFFLINE due to disk space [23:55:56] maintenance-disconnect-full-disks build 504720 integration-agent-docker-1035 (/: 31%, /srv: 15%, /var/lib/docker: 100%): still OFFLINE due to disk space [23:55:56] maintenance-disconnect-full-disks build 504720 integration-agent-docker-1036 (/: 30%, /srv: 18%, /var/lib/docker: 99%): still OFFLINE due to disk space [23:55:56] maintenance-disconnect-full-disks build 504720 integration-agent-docker-1037 (/: 29%, /srv: 15%, /var/lib/docker: 97%): still OFFLINE due to disk space [23:55:57] maintenance-disconnect-full-disks build 504720 integration-agent-docker-1038 (/: 29%, /srv: 14%, /var/lib/docker: 95%): still OFFLINE due to disk space