[03:16:20] PROBLEM - Check systemd state on doc1003 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc2002.codfw.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [04:13:20] RECOVERY - Check systemd state on doc1003 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [06:24:08] 10Release-Engineering-Team, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 B): eventutillities-python should publish python doc to doc.wikimedia.org - https://phabricator.wikimedia.org/T337475 (10gmodena) a:03gmodena [07:37:03] 10Continuous-Integration-Infrastructure, 10serviceops-collab, 10Patch-For-Review: Raise version of PHP on integration.wikimedia.org from 7.3 to 7.4+ - https://phabricator.wikimedia.org/T334954 (10hashar) As a result of the upgrade, we have removed php 7.3 support from `integration/docroot`: https://gerrit.wi... [07:38:51] 10Continuous-Integration-Infrastructure, 10serviceops-collab, 10Patch-For-Review: Raise version of PHP on integration.wikimedia.org from 7.3 to 7.4+ - https://phabricator.wikimedia.org/T334954 (10hashar) [07:45:45] 10Deployments, 10Release-Engineering-Team: Scap train-presync failed to prepare 1.41.0-wmf.12 - https://phabricator.wikimedia.org/T338205 (10hashar) [07:46:02] 10Deployments, 10Release-Engineering-Team: Scap train-presync failed to prepare 1.41.0-wmf.12 - https://phabricator.wikimedia.org/T338205 (10hashar) [07:46:07] 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T337526 (10hashar) [07:46:55] 10Deployments, 10Release-Engineering-Team: Scap train-presync failed to prepare 1.41.0-wmf.12 - https://phabricator.wikimedia.org/T338205 (10hashar) p:05Triageβ†’03Unbreak! That is obviously blocking the train and as such is an {nav Unbreak Now!} [08:06:16] 10Deployments, 10Release-Engineering-Team: Scap train-presync failed to prepare 1.41.0-wmf.12 - https://phabricator.wikimedia.org/T338205 (10hashar) The few things I found from a quick look on `deployment.eqiad.wmnet` (`deploy1002.eqiad.wmnet`): Ghost uid ======== `/srv/patches/` is owned by non existing use... [08:20:18] 10Scap: Consider updating /srv/deployment/scap on deployment servers - https://phabricator.wikimedia.org/T338209 (10hashar) [08:21:53] well [08:21:57] I have no clue what is going on :D [08:22:01] 10Release-Engineering-Team, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 B), 10Patch-For-Review: eventutillities-python should publish python doc to doc.wikimedia.org - https://phabricator.wikimedia.org/T337475 (10CodeReviewBot) gmodena opened https://gitlab.wikimedia.org/repos/data-enginee... [08:22:13] 10Release-Engineering-Team, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 B), 10Patch-For-Review: eventutillities-python should publish python doc to doc.wikimedia.org - https://phabricator.wikimedia.org/T337475 (10CodeReviewBot) [08:22:54] ah found something [08:29:45] 10Deployments, 10Release-Engineering-Team: Scap train-presync failed to prepare 1.41.0-wmf.12 - https://phabricator.wikimedia.org/T338205 (10hashar) Possible culprit is https://gerrit.wikimedia.org/r/c/operations/puppet/+/927269 ` CommitDate: Mon Jun 5 19:41:13 2023 +0000 fix-stagging-perms: Fix group own... [08:36:36] 10Deployments, 10Release-Engineering-Team, 10Patch-For-Review: Scap train-presync failed to prepare 1.41.0-wmf.12 - https://phabricator.wikimedia.org/T338205 (10Urbanecm) I thought there has to be a reason for the deployment group ownership :). Uploaded a fixing patch; deploying it and running the fixing scr... [08:38:15] 10Deployments, 10Release-Engineering-Team, 10Patch-For-Review: Scap train-presync failed to prepare 1.41.0-wmf.12 - https://phabricator.wikimedia.org/T338205 (10hashar) T338180 had: > In addition to this, I noticed that **the group owner of `/srv/patches` changed to `deployment`** (this may or may not be the... [08:43:24] 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Release, 10Train Deployments: 1.41.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T337527 (10daniel) ##### Risky Patch! πŸš‚πŸ”₯ * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/899731 * **Summary**: ** Rewrite of hook handler regist... [08:43:48] 10Release-Engineering-Team, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 B), 10Patch-For-Review: eventutillities-python should publish python doc to doc.wikimedia.org - https://phabricator.wikimedia.org/T337475 (10gmodena) Hey @jnuche, Couple of questions re integrating this workflow in o... [08:45:05] 10Deployments, 10Release-Engineering-Team, 10Patch-For-Review: Scap train-presync failed to prepare 1.41.0-wmf.12 - https://phabricator.wikimedia.org/T338205 (10hashar) Short story: `/srv/patches/.git` is now owned by `wikidev` group due to the script being fixed but it should be in the `deployment` group f... [09:03:11] 10Deployments, 10Release-Engineering-Team, 10Patch-For-Review: Scap train-presync failed to prepare 1.41.0-wmf.12 - https://phabricator.wikimedia.org/T338205 (10Urbanecm) The ownership should be fixed now. Leaving re-running the presync command to @hashar / releng. [09:14:08] 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Release, 10Train Deployments: 1.41.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T337527 (10Lucas_Werkmeister_WMDE) > This code is used for all hooks, it is exercised hundreds of times with every request, and most test cases. If somethign is wr... [09:17:48] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10serviceops-collab: Provide mechanism to publish to doc.wikimedia.org from GitLab CI - https://phabricator.wikimedia.org/T336168 (10jnuche) @Legoktm Right now developers in your project will need to be added to docpub by a member of R... [09:19:58] 10Deployments, 10Release-Engineering-Team: Scap train-presync failed to prepare 1.41.0-wmf.12 - https://phabricator.wikimedia.org/T338205 (10hashar) [09:20:05] 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T337526 (10hashar) [09:25:11] 10Deployments, 10Release-Engineering-Team, 10Sustainability (Incident Followup): Scap train-presync failed to prepare 1.41.0-wmf.12 - https://phabricator.wikimedia.org/T338205 (10hashar) p:05Unbreak!β†’03Medium @Urbanecm fixed it up and @jcrespo reran the train-presync systemd service. The train is progres... [09:25:40] 10Deployments, 10Release-Engineering-Team, 10Sustainability (Incident Followup): Scap train-presync failed to prepare 1.41.0-wmf.12 - https://phabricator.wikimedia.org/T338205 (10hashar) [09:55:32] 10GitLab (Auth & Access), 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10CAS-SSO, 10Infrastructure-Foundations, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10jbond) > Here are logs from my login to the admin page: https://logstash.wikimedia.org/goto/6... [10:50:16] 10Release-Engineering-Team, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 B), 10Patch-For-Review: eventutillities-python should publish python doc to doc.wikimedia.org - https://phabricator.wikimedia.org/T337475 (10jnuche) > As a test, I wanted to trigger .docpub:publish-docs (derived) manua... [11:15:59] 10GitLab, 10Release-Engineering-Team, 10serviceops: Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10kostajh) [11:24:24] 10GitLab, 10Release-Engineering-Team, 10serviceops: Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10Joe) Specifically, what we want is that image tags are obviously sortable in chronological o... [11:32:57] 10GitLab, 10Release-Engineering-Team, 10serviceops, 10Patch-For-Review: Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10CodeReviewBot) kharlan opened https://gitlab.wikimedia.org/repos/media... [11:33:07] 10GitLab, 10Release-Engineering-Team, 10serviceops, 10Patch-For-Review: Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10CodeReviewBot) [11:46:55] 10GitLab, 10Release-Engineering-Team, 10serviceops, 10Patch-For-Review: Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10kostajh) I think we could do something like `export CI_IMAGE_PUBLISH_D... [12:20:48] maintenance-disconnect-full-disks build 497669 integration-agent-docker-1037 (/: 29%, /srv: 99%, /var/lib/docker: 46%): OFFLINE due to disk space [12:21:48] 10Release-Engineering-Team, 10serviceops-collab: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517 (10LSobanski) LGTM. One question to @hashar is whether we still want the primary to be in codfw or would it be better to move to eqiad as part of this? [12:26:09] maintenance-disconnect-full-disks build 497670 integration-agent-docker-1037 (/: 29%, /srv: 25%, /var/lib/docker: 44%): RECOVERY disk space OK [12:54:59] 10Release-Engineering-Team, 10serviceops-collab: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517 (10hashar) In short I don't know, there is a long tail of checks that needs to happen for the Bullseye upgrade and I haven't checked any of them yet. From the top of my mind: * Java... [13:02:06] 10Release-Engineering-Team, 10serviceops-collab: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517 (10hashar) There is also T324659 to make it possible to switch over the services from host to host since last time that caused major havoc and is the reason the services are still on... [13:31:35] 10Deployments, 10Release-Engineering-Team, 10Patch-For-Review, 10Sustainability (Incident Followup): Scap train-presync failed to prepare 1.41.0-wmf.12 - https://phabricator.wikimedia.org/T338205 (10hashar) [13:37:08] so many changes in gate-and-submit right now ._. [13:37:28] 10Deployments, 10Release-Engineering-Team, 10Patch-For-Review, 10Sustainability (Incident Followup): Scap train-presync failed to prepare 1.41.0-wmf.12 - https://phabricator.wikimedia.org/T338205 (10hashar) I have send Puppet patches from 3 of the 4 actionable. Not sure who can review them though. The las... [13:40:52] maintenance-disconnect-full-disks build 497685 integration-agent-docker-1030 (/: 30%, /srv: 99%, /var/lib/docker: 45%): OFFLINE due to disk space [13:45:44] maintenance-disconnect-full-disks build 497686 integration-agent-docker-1030 (/: 30%, /srv: 9%, /var/lib/docker: 43%): RECOVERY disk space OK [13:52:08] Lucas_WMDE: How dare people review and try to merge code [13:52:58] can’t they do that in other time zones where I’m not working [13:54:28] 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Anti-Harassment, 10Projects-Cleanup, 10Security-Team, 10iPoid-Service: Migrate mediawiki/services/ipoid to GitLab - https://phabricator.wikimedia.org/T337714 (10Mstyles) As this project migrates over to Gitlab, the new security pipeline templates can be used.... [13:55:51] (03PS1) 10Daimona Eaytoy: Zuul: [mediawiki/extensions/CampaignEvents] Enable commit-message-validator [integration/config] - 10https://gerrit.wikimedia.org/r/927683 (https://phabricator.wikimedia.org/T333167) [13:58:57] Obligatory "we should just add more workers" <3 [13:59:16] * Reedy assigns TheresNoTime to merging code on behalf of jenkins [13:59:31] 😭 [14:09:31] speaking of merging code, it seems not only more workers, but also more disk space: https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81-docker/7821/consoleFull [14:37:37] oh for god sake gitlab is killing me [14:37:53] * hashar fixes stuff [14:45:57] maintenance-disconnect-full-disks build 497698 integration-agent-docker-1030 (/: 30%, /srv: 95%, /var/lib/docker: 45%): OFFLINE due to disk space [14:49:22] 10GitLab, 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Projects-Cleanup, 10dev-images, 10User-brennen: Migrate releng/dev-images to GitLab - https://phabricator.wikimedia.org/T290259 (10hashar) 05Resolvedβ†’03Open I got a bunch of confusion today since I still had a copy of the Gerrit dev images and... [14:50:49] maintenance-disconnect-full-disks build 497699 integration-agent-docker-1030 (/: 30%, /srv: 19%, /var/lib/docker: 43%): RECOVERY disk space OK [15:08:14] (03PS1) 10Hashar: Repository has moved to GitLab [releng/dev-images] - 10https://gerrit.wikimedia.org/r/927727 (https://phabricator.wikimedia.org/T290259) [15:10:51] 10GitLab, 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Projects-Cleanup, 10dev-images, and 2 others: Migrate releng/dev-images to GitLab - https://phabricator.wikimedia.org/T290259 (10hashar) 05Openβ†’03Resolved ` git clone --mirror https://gerrit.wikimedia.org:443/r/releng/dev-images.git cd dev-images... [15:12:01] 10GitLab, 10Release-Engineering-Team (Yak Shaving πŸƒπŸͺ’), 10Projects-Cleanup, 10dev-images, 10User-brennen: Migrate releng/dev-images to GitLab - https://phabricator.wikimedia.org/T290259 (10hashar) Note: I have no complain, I just got confused cause I still had copy about the old repository :] [15:37:23] 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Anti-Harassment, 10Projects-Cleanup, 10Security-Team, 10iPoid-Service: Use Gitlab Security Pipeline for ipoid - https://phabricator.wikimedia.org/T338238 (10Mstyles) [15:37:58] 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Anti-Harassment, 10Projects-Cleanup, 10Security-Team, 10iPoid-Service: Use Gitlab Security Pipeline for ipoid - https://phabricator.wikimedia.org/T338238 (10Mstyles) a:05kostajhβ†’03Mstyles [15:48:57] 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Anti-Harassment, 10Projects-Cleanup, 10Security-Team, 10iPoid-Service: Use Gitlab Security Pipeline for ipoid - https://phabricator.wikimedia.org/T338238 (10sbassett) Just FYI, we need to address some issues with the #gitlab-application-security-pipeline, see... [15:54:00] 10GitLab, 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10serviceops, 10Patch-For-Review: Provide ability to tag GitLab CI built images with a datetime format, set as default in pipeline-to-gitlab conversion - https://phabricator.wikimedia.org/T338224 (10thcipriani) [15:57:10] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/CampaignEvents] Enable commit-message-validator [integration/config] - 10https://gerrit.wikimedia.org/r/927683 (https://phabricator.wikimedia.org/T333167) (owner: 10Daimona Eaytoy) [15:58:40] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CampaignEvents] Enable commit-message-validator [integration/config] - 10https://gerrit.wikimedia.org/r/927683 (https://phabricator.wikimedia.org/T333167) (owner: 10Daimona Eaytoy) [16:01:51] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/927683 # T333167 [16:01:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:01:54] T333167: Do we want to enable commit-message-validator for the CampaignEvents repo? - https://phabricator.wikimedia.org/T333167 [16:23:02] (03CR) 10Daimona Eaytoy: "Thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/927683 (https://phabricator.wikimedia.org/T333167) (owner: 10Daimona Eaytoy) [18:04:04] 10Release-Engineering-Team, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 B), 10Patch-For-Review: eventutillities-python should publish python doc to doc.wikimedia.org - https://phabricator.wikimedia.org/T337475 (10CodeReviewBot) gmodena merged https://gitlab.wikimedia.org/repos/data-enginee... [18:45:12] 10Release-Engineering-Team, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 B), 10Patch-For-Review: eventutillities-python should publish python doc to doc.wikimedia.org - https://phabricator.wikimedia.org/T337475 (10CodeReviewBot) gmodena updated https://gitlab.wikimedia.org/repos/data-engine... [19:05:58] 10Scap: Consider updating /srv/deployment/scap on deployment servers - https://phabricator.wikimedia.org/T338209 (10dancy) Some notes: Using `scap install-world`, scap deploys itself using stuff under /var/lib/scap (which is exported on the deploy server via rsync under the name `scap-install-staging`) as sourc... [19:12:27] 10Release-Engineering-Team (They Live πŸ•ΆοΈπŸ§Ÿ), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T337526 (10Reedy) [19:30:56] 10Release-Engineering-Team, 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 B), 10Patch-For-Review: eventutillities-python should publish python doc to doc.wikimedia.org - https://phabricator.wikimedia.org/T337475 (10gmodena) Project documentation is available at https://doc.wikimedia.org/dat... [20:49:02] 10Release-Engineering-Team, 10Puppet: Puppet git::clone probably does not need `umask` parameter - https://phabricator.wikimedia.org/T338277 (10hashar) [21:34:42] 10Phabricator, 10DBA, 10Data-Persistence-Backup, 10serviceops-collab, 10Patch-For-Review: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10Dzahn) oops, patch does not belong here. accidental