[00:30:20] 10Phabricator, 10DBA, 10serviceops-collab: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10Dzahn) [00:31:15] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab, 10User-brennen: Migrate phabricator.wikimedia.org to Phorge as upstream - https://phabricator.wikimedia.org/T333885 (10Dzahn) [00:31:19] 10Phabricator, 10DBA, 10serviceops-collab: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10Dzahn) [00:33:06] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab: upgrade phab (phorge) hosts to bullseye - https://phabricator.wikimedia.org/T334519 (10Dzahn) [00:39:32] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab, 10User-brennen: Migrate phabricator.wikimedia.org to Phorge as upstream - https://phabricator.wikimedia.org/T333885 (10Dzahn) [00:39:38] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab, 10User-brennen: Migrate phabricator.wikimedia.org to Phorge as upstream - https://phabricator.wikimedia.org/T333885 (10Dzahn) T335080 - is a subtask how to handle the database part of the migration T334519 - is an (relatively unrelated) task... [00:40:53] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab, 10User-brennen: Migrate phabricator.wikimedia.org to Phorge as upstream - https://phabricator.wikimedia.org/T333885 (10Dzahn) [00:42:13] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab, 10User-brennen: Migrate phabricator.wikimedia.org to Phorge as upstream - https://phabricator.wikimedia.org/T333885 (10Dzahn) [00:44:05] 10Phabricator (Upstream), 10Release-Engineering-Team (Seen), 10serviceops-collab, 10Upstream: Consider migrating from Phabricator to Phorge once that new upstream is ready for such a migration - https://phabricator.wikimedia.org/T302528 (10Dzahn) After T328595 this is now actually going to happen in produc... [00:44:43] (03PS1) 10Jforrester: Zuul: [mediawiki/tools/wikilambda-cli] Archived, moved to GitLab [integration/config] - 10https://gerrit.wikimedia.org/r/910107 [00:45:16] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/tools/wikilambda-cli] Archived, moved to GitLab [integration/config] - 10https://gerrit.wikimedia.org/r/910107 (owner: 10Jforrester) [00:46:25] (03Merged) 10jenkins-bot: Zuul: [mediawiki/tools/wikilambda-cli] Archived, moved to GitLab [integration/config] - 10https://gerrit.wikimedia.org/r/910107 (owner: 10Jforrester) [00:56:05] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10Abstract Wikipedia team, 10function-orchestrator, 10Patch-For-Review: Move the mediawiki/services/function-orchestrator repo from gerrit+pipeline to GitLab - https://phabricator.wikimedia.org/T334397 (10Jdforrester-WMF) 05I... [00:56:38] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10API Platform, 10Anti-Harassment, and 19 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10Jdforrester-WMF) [01:13:47] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10Abstract Wikipedia team, 10function-evaluator, 10Patch-For-Review: Move the mediawiki/services/function-evaluator repo from gerrit+pipeline to GitLab - https://phabricator.wikimedia.org/T334398 (10Jdforrester-WMF) OK, other... [06:55:48] 10Phabricator, 10DBA, 10Data-Persistence-Backup, 10serviceops-collab: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10Marostegui) p:05Triage→03Medium a:03Marostegui Thanks for the task. I would feel more comfortable if we can test this before hand rath... [07:28:33] 10Phabricator, 10DBA, 10Data-Persistence-Backup, 10serviceops-collab: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10Marostegui) [07:44:36] 10Phabricator, 10DBA, 10Data-Persistence-Backup, 10serviceops-collab: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10jcrespo) >>! In T335080#8795200, @Marostegui wrote: > If all works fine, we can arrange a day/time to do the migration itself in production.... [09:00:53] 10Continuous-Integration-Config, 10Codex, 10Design-Systems-Team: docs: Consider making the latest release branch the default for the live docs site - https://phabricator.wikimedia.org/T311097 (10bmartinezcalvo) >>! In T311097#8794353, @CCiufo-WMF wrote: > @AnneT @ldelench_wmf in February, we pulled this out... [09:07:10] (03CR) 10Giuseppe Lavagetto: [C: 03+2] helm-linter: move to bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/909996 (owner: 10Giuseppe Lavagetto) [09:08:26] (03Merged) 10jenkins-bot: helm-linter: move to bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/909996 (owner: 10Giuseppe Lavagetto) [09:13:27] !log Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/909996 [09:13:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:43:56] (03PS1) 10Giuseppe Lavagetto: ci: use helm-linter 0.4.6 [integration/config] - 10https://gerrit.wikimedia.org/r/910433 [09:45:53] (03CR) 10Giuseppe Lavagetto: [C: 03+2] ci: use helm-linter 0.4.6 [integration/config] - 10https://gerrit.wikimedia.org/r/910433 (owner: 10Giuseppe Lavagetto) [09:47:00] (03Merged) 10jenkins-bot: ci: use helm-linter 0.4.6 [integration/config] - 10https://gerrit.wikimedia.org/r/910433 (owner: 10Giuseppe Lavagetto) [09:54:41] jnuche / hashar: I'd need one of you to deploy the jjb update for 'helm-lint'; apparently my credentials aren't enough to do so [09:56:21] sorry to ping the two of you but you're the TZ-appropriate people :) [09:56:40] joe: h.ashar is on vacation, let me take a look after lunch [09:56:53] or I can do it for you now [09:58:37] yes please [09:58:44] I'm kinda on a strict clock today [09:58:49] thanks a lot taavi [09:58:54] INFO:jenkins_jobs.cli.subcommand.update:Number of jobs updated: 1 [09:59:03] 🙏 [09:59:06] thanks :) [09:59:12] !log update helm-lint jjb job for https://gerrit.wikimedia.org/r/c/integration/config/+/910433/ [09:59:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:59:16] yw [09:59:21] I don't know when it happened that I lost the ability to do so [09:59:56] thank you taavi :) [09:59:57] it seems I get regularly demoted from jenkins admin, then it gets re-added when I make people notice I have root on the servers so this is just an annoyance [10:00:03] you would need to be in the ciadmin ldap group [10:00:13] taavi: yes I know [10:00:34] tbh I think it makes little sense that people who have global root are not included, but well :) [10:39:25] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Infrastructure-Foundations, 10puppet-compiler: PCC runs failing with complaints about disk space - https://phabricator.wikimedia.org/T335111 (10jbond) [10:42:11] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10Infrastructure-Foundations, 10puppet-compiler: PCC runs failing with complaints about disk space - https://phabricator.wikimedia.org/T335111 (10jbond) p:05Triage→03Medium [12:57:05] 10GitLab (Infrastructure), 10serviceops-collab: Troubleshoot partman config for two additional disks on GitLab hosts - https://phabricator.wikimedia.org/T333674 (10Jelto) Ok thanks @MoritzMuehlenhoff for the context. I'll add a short section in wikitech about how to bootstrap a new instance and the new script.... [14:45:34] maintenance-disconnect-full-disks build 484169 integration-agent-docker-1024 (/: 26%, /srv: 100%, /var/lib/docker: 47%): OFFLINE due to disk space [14:50:29] maintenance-disconnect-full-disks build 484170 integration-agent-docker-1024 (/: 26%, /srv: 7%, /var/lib/docker: 44%): RECOVERY disk space OK [15:33:34] 10Phabricator, 10DBA, 10Data-Persistence-Backup, 10serviceops-collab: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10Dzahn) a:05Marostegui→03Dzahn [16:28:05] 10Continuous-Integration-Config, 10Codex, 10Design-Systems-Team: docs: Consider making the latest release branch the default for the live docs site - https://phabricator.wikimedia.org/T311097 (10AnneT) @bmartinezcalvo +1, if we want to do this we should do it now before the 1.0 release. I do think it would b... [16:44:19] 10Release-Engineering-Team, 10serviceops-collab: upgrade gerrit servers to bullseye - https://phabricator.wikimedia.org/T334521 (10LSobanski) p:05Triage→03High [16:44:28] 10Phabricator, 10Release-Engineering-Team, 10serviceops-collab: upgrade phab (phorge) hosts to bullseye - https://phabricator.wikimedia.org/T334519 (10LSobanski) p:05Triage→03Medium [16:44:39] 10Release-Engineering-Team, 10serviceops-collab: upgrade contint servers to bullseye - https://phabricator.wikimedia.org/T334517 (10LSobanski) p:05Triage→03High [16:48:41] 10Continuous-Integration-Infrastructure, 10serviceops-collab: Raise version of PHP on integration.wikimedia.org from 7.3 to 7.4+ - https://phabricator.wikimedia.org/T334954 (10LSobanski) @hashar do you see value in doing this or should we go directly for the upgrade? CC @thcipriani in case someone else can ma... [16:56:30] 10Release-Engineering-Team, 10serviceops-collab: upgrade gerrit servers to bullseye - https://phabricator.wikimedia.org/T334521 (10Dzahn) a:03Dzahn [16:56:39] 10Release-Engineering-Team, 10serviceops-collab: upgrade gerrit servers to bullseye - https://phabricator.wikimedia.org/T334521 (10Dzahn) 05Open→03In progress [17:04:22] 10Release-Engineering-Team, 10Release Pipeline (Blubber): Blubber seems confused when copying to a folder that is a symlink - https://phabricator.wikimedia.org/T335156 (10dcausse) [18:02:46] 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Jelto) [18:10:56] 10GitLab (Infrastructure), 10serviceops-collab: Troubleshoot partman config for two additional disks on GitLab hosts - https://phabricator.wikimedia.org/T333674 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jelto@cumin2002 for host gitlab1003.wikimedia.org with OS bullseye [18:17:06] 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Dzahn) a:03Dzahn [18:50:57] 10GitLab (Infrastructure), 10serviceops-collab: Troubleshoot partman config for two additional disks on GitLab hosts - https://phabricator.wikimedia.org/T333674 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jelto@cumin2002 for host gitlab1003.wikimedia.org with OS bullseye completed: -... [19:48:53] 10GitLab (Infrastructure), 10serviceops-collab: Troubleshoot partman config for two additional disks on GitLab hosts - https://phabricator.wikimedia.org/T333674 (10Jelto) Reimage of `gitlab1003` went fine. The disk names changed again during the reimage (`sda` and `sdc` old disk, `sdb` and `sdd` new disks). Ho... [21:03:00] 10Release-Engineering-Team, 10serviceops-collab: upgrade gerrit servers to bullseye - https://phabricator.wikimedia.org/T334521 (10Dzahn) basically duplicate of https://phabricator.wikimedia.org/T326368 with small variation that the cloud instance should also be upgraded and gerrit1001 be decom'ed [21:08:22] 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Dzahn) instance name: gitlab-prod-1002.devtools.eqiad1.wikimedia.cloud [21:09:56] 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Dzahn) We should already have this fully automated: ` 66 if $use_acmechief { 67 # Certificates will be available under... [21:11:43] 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Dzahn) ` Apr 20 05:11:03 gitlab-prod-1002 certbot[5595]: Challenge failed for domain gitlab.devtools.wmcloud.org Apr 20 05:11:04 gitl... [21:24:54] 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Dzahn) So.. there seems to be a more general issue here.. since I can't access https://gitlab-prod.wmflabs.org/ (the domain proxy poi... [21:28:37] 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Dzahn) Locally I can talk to nginx on the same IP, on port 80: ` curl 172.16.3.213 Welcome to... [21:30:02] <wikibugs> 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Dzahn) nginx is instead listening on 172.16.7.146:443 tcp 0 0 172.16.7.146:443 0.0.0.0:* LISTEN... [21:39:13] <wikibugs> 10GitLab (Infrastructure), 10serviceops-collab: Let's Encrypt certificate expiration notice for domain gitlab.devtools.wmcloud.org - https://phabricator.wikimedia.org/T335161 (10Dzahn) I was going to experiment with creating a new proxy, use port 80, us the other IP etc, but I am also getting errors from the p... [23:54:46] <wikibugs> 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10Abstract Wikipedia team, 10function-evaluator, 10Patch-For-Review: Move the mediawiki/services/function-evaluator repo from gerrit+pipeline to GitLab - https://phabricator.wikimedia.org/T334398 (10Jdforrester-WMF) [23:55:03] <wikibugs> 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10Abstract Wikipedia team, 10function-evaluator, 10Patch-For-Review: Move the mediawiki/services/function-evaluator repo from gerrit+pipeline to GitLab - https://phabricator.wikimedia.org/T334398 (10Jdforrester-WMF) [23:55:06] <wikibugs> 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10Abstract Wikipedia team, 10function-orchestrator, 10Patch-For-Review: Move the mediawiki/services/function-orchestrator repo from gerrit+pipeline to GitLab - https://phabricator.wikimedia.org/T334397 (10Jdforrester-WMF)