[04:24:42] 10Gerrit, 10VPS-project-Codesearch, 10VPS-project-Extdist, 10serviceops-collab: Move clients off of gerrit-replica.wikimedia.org back to gerrit.wikimedia.org - https://phabricator.wikimedia.org/T336710 (10Bawolff) [04:25:47] 10Gerrit, 10VPS-project-Codesearch, 10VPS-project-Extdist, 10serviceops-collab: Move clients off of gerrit-replica.wikimedia.org back to gerrit.wikimedia.org - https://phabricator.wikimedia.org/T336710 (10Bawolff) Extdist is also using replica currently. [06:58:38] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog 📥), 10serviceops-collab: Provide mechanism to publish to doc.wikimedia.org from GitLab CI - https://phabricator.wikimedia.org/T336168 (10Jelto) >>! In T336168#8852463, @Dzahn wrote: > P.S. Currently /srv/doc already has files in it on... [07:03:38] 10GitLab (Auth & Access), 10Release-Engineering-Team: Requesting GitLab non-external access/account unlock for Mvolz - https://phabricator.wikimedia.org/T336578 (10Mvolz) 05Resolved→03Open p:05Triage→03High [07:05:12] 10GitLab (Auth & Access), 10Release-Engineering-Team: Requesting GitLab non-external access/account unlock for Mvolz - https://phabricator.wikimedia.org/T336578 (10Mvolz) Sorry to re-open - it turns out I need this rather urgently (sorry) or, alternatively, someone to un-archive the old respository on gerrit:... [07:38:32] 10GitLab (Auth & Access), 10Release-Engineering-Team: Requesting GitLab non-external access/account unlock for Mvolz - https://phabricator.wikimedia.org/T336578 (10brennen) > And the "pick a group or namespace" option doesn't seem to be anywhere on that page... permissions issue with my account maybe? (Do I ne... [08:08:30] 10GitLab (Auth & Access), 10Release-Engineering-Team: Requesting GitLab non-external access/account unlock for Mvolz - https://phabricator.wikimedia.org/T336578 (10Mvolz) [08:10:01] 10GitLab (Auth & Access), 10Release-Engineering-Team: Requesting GitLab non-external access/account unlock for Mvolz - https://phabricator.wikimedia.org/T336578 (10Mvolz) >>! In T336578#8853306, @brennen wrote: >> And the "pick a group or namespace" option doesn't seem to be anywhere on that page... permission... [08:10:19] 10GitLab (Auth & Access), 10Release-Engineering-Team: Requesting GitLab non-external access/account unlock for Mvolz - https://phabricator.wikimedia.org/T336578 (10Mvolz) 05Open→03Resolved a:03brennen [08:16:55] 10GitLab (Auth & Access), 10Release-Engineering-Team: Requesting GitLab non-external access/account unlock for Mvolz - https://phabricator.wikimedia.org/T336578 (10Mvolz) Sorry, one more thing: I was trying to preserve the structure of the original url, which is nested further under services: https://gerrit.... [08:49:51] 10Release-Engineering-Team, 10API Platform, 10AQS2.0, 10Platform Engineering, and 5 others: Define a procedure/pattern to populate test environments - https://phabricator.wikimedia.org/T334851 (10Sfaci) @Htriedman I have some questions about the project you created: - Which python version are you using? Cu... [09:11:53] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account approval for Florian Cuny - https://phabricator.wikimedia.org/T336734 (10Poslovitch) [09:30:05] 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops, 10serviceops-collab: Gitlab downtime blocking scap backport - https://phabricator.wikimedia.org/T336162 (10jnuche) 05Open→03Resolved Change is now in prod. Scap should now complete deployments when gitlab is not available. [09:44:57] 10GitLab (Infrastructure), 10ops-eqiad, 10serviceops-collab: gitlab-runner1003 is not coming back online - https://phabricator.wikimedia.org/T336737 (10Jelto) [10:00:03] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity, 10MW-1.40-notes (1.40.0-wmf.26; 2023-03-06), 10MW-1.41-notes (1.41.0-wmf.4; 2023-04-10): Provide early feedback when a patch has job failures - https://phabricator.wikimedia.org/T323750 (10kostajh) >>! In T323750#8852326, @Tgr... [10:10:38] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity, 10MW-1.40-notes (1.40.0-wmf.26; 2023-03-06), 10MW-1.41-notes (1.41.0-wmf.4; 2023-04-10): Provide early feedback when a patch has job failures - https://phabricator.wikimedia.org/T323750 (10hashar) I don't think we should attemp... [10:16:02] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Seen), 10Zuul: Display Zuul status of jobs for a change on Gerrit UI - https://phabricator.wikimedia.org/T214068 (10hashar) Ben Rohlfs answer: > The error manager listens on document for show-alert events. So yes, you can fire s... [10:27:51] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity, 10MW-1.40-notes (1.40.0-wmf.26; 2023-03-06), 10MW-1.41-notes (1.41.0-wmf.4; 2023-04-10): Provide early feedback when a patch has job failures - https://phabricator.wikimedia.org/T323750 (10hashar) I have found an example on Chr... [10:33:09] hashar: do you have a minute for https://gerrit.wikimedia.org/r/c/integration/config/+/914785 ? The image is already build [10:55:34] !log update operations-puppet-tests-buster-docker jjb job for https://gerrit.wikimedia.org/r/c/integration/config/+/914785 [10:55:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:55:39] (03PS3) 10Majavah: jjb/operations-puppet: Bump image version [integration/config] - 10https://gerrit.wikimedia.org/r/914785 (https://phabricator.wikimedia.org/T304660) (owner: 10JMeybohm) [10:55:43] (03CR) 10Majavah: [C: 03+2] "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/914785 (https://phabricator.wikimedia.org/T304660) (owner: 10JMeybohm) [10:56:50] jayme: done! [10:56:55] (03Merged) 10jenkins-bot: jjb/operations-puppet: Bump image version [integration/config] - 10https://gerrit.wikimedia.org/r/914785 (https://phabricator.wikimedia.org/T304660) (owner: 10JMeybohm) [10:59:22] (03CR) 10STran: [C: 03+1] Rename security-api to ipoid [integration/config] - 10https://gerrit.wikimedia.org/r/919873 (https://phabricator.wikimedia.org/T336218) (owner: 10Tchanders) [11:05:12] 10Phabricator, 10Content-Transform-Team-WIP: Remove Herald rule tagging #Product-Infrastructure-Team-Backlog-Deprecated (H228) - https://phabricator.wikimedia.org/T336151 (10MSantos) [11:05:34] 10Phabricator, 10Content-Transform-Team-WIP: Remove Herald rule tagging #Product-Infrastructure-Team-Backlog-Deprecated (H228) - https://phabricator.wikimedia.org/T336151 (10MSantos) @Aklapper thanks for the link, I wasn't aware of. I'll just close this as duplicated. [11:22:59] 10Gerrit, 10VPS-project-Codesearch, 10VPS-project-Extdist, 10serviceops-collab: Move clients off of gerrit-replica.wikimedia.org back to gerrit.wikimedia.org - https://phabricator.wikimedia.org/T336710 (10Ladsgroup) I deployed the codesearch change. So it shouldn't be affected by the maint window. Maybe ev... [11:27:59] (03CR) 10Hashar: [C: 03+2] "Jobs deployed:" [integration/config] - 10https://gerrit.wikimedia.org/r/919873 (https://phabricator.wikimedia.org/T336218) (owner: 10Tchanders) [11:29:07] (03Merged) 10jenkins-bot: Rename security-api to ipoid [integration/config] - 10https://gerrit.wikimedia.org/r/919873 (https://phabricator.wikimedia.org/T336218) (owner: 10Tchanders) [11:30:12] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/919873 | T336218 [11:30:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:30:15] T336218: Rename security-api to iPoid - https://phabricator.wikimedia.org/T336218 [12:27:33] 10GitLab (Auth & Access), 10Release-Engineering-Team: Requesting GitLab non-external access/account unlock for Mvolz - https://phabricator.wikimedia.org/T336578 (10Mvolz) That was almost good enough, but only being a developer means I don't have access to the settings tab, which means I can't set the default... [12:46:49] 10GitLab (Auth & Access), 10Release-Engineering-Team: Requesting GitLab non-external access/account unlock for Mvolz - https://phabricator.wikimedia.org/T336578 (10Mvolz) Hmm, I also can't write to the target branch (master, currently) because it's protected. [12:47:00] 10GitLab (Auth & Access), 10Release-Engineering-Team: Requesting GitLab non-external access/account unlock for Mvolz - https://phabricator.wikimedia.org/T336578 (10Mvolz) 05Resolved→03Open [12:47:13] taavi: thanks! [14:39:18] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Radar), 10Upstream: ircbot-plugin emits SEVERE java.lang.IndexOutOfBoundsException when parsing AWAY IRCv3 notifications - https://phabricator.wikimedia.org/T283009 (10hashar) 05Declined→03Open We currently run 2.36 Pircbo... [15:04:19] !log CI Jenkins: updating IRC plugin from 2.42 to 3.829.v12d4b_d1f7650 and instant-messaging plugin from 1.48 to 2.666.va_6c1e97cc252 # T283009 [15:04:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:04:22] T283009: ircbot-plugin emits SEVERE java.lang.IndexOutOfBoundsException when parsing AWAY IRCv3 notifications - https://phabricator.wikimedia.org/T283009 [15:11:53] !log Killing stall https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ job [15:11:54] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:12:00] that does not smell good [15:18:19] so yeah somehow https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/3682143/console is stall [15:18:23] and it is triggered by all jobs [15:18:27] so everything is stall somehow [15:46:49] I think I got them covered [15:46:56] plugins upgrades have side effects unfortunately [16:04:03] enough breaking for today [16:38:16] 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review: Kokkuri should allow dockerfile.v0 frontend - https://phabricator.wikimedia.org/T326569 (10dancy) @xcollazo I've updated kokkuri to support Dockerfiles. Sample use: ` include: - project: 'repos/releng/kokkuri' file: 'includes/images.yaml'... [16:39:52] 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review: Kokkuri should allow dockerfile.v0 frontend - https://phabricator.wikimedia.org/T326569 (10dancy) Btw, the dockerfile frontend does not work right with `.kokkuri:build-and-run-image` because the run-image logic is only implemented by the blubber/b... [16:44:29] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10serviceops-collab: Provide mechanism to publish to doc.wikimedia.org from GitLab CI - https://phabricator.wikimedia.org/T336168 (10thcipriani) [16:56:14] 10GitLab (Auth & Access), 10Release-Engineering-Team (They Live 🕶️🧟), 10User-brennen: Requesting GitLab non-external access/account unlock for Mvolz - https://phabricator.wikimedia.org/T336578 (10brennen) [17:42:47] 10Release-Engineering-Team (They Live 🕶️🧟), 10Patch-For-Review: Kokkuri should allow dockerfile.v0 frontend - https://phabricator.wikimedia.org/T326569 (10dancy) 05Open→03Resolved [18:12:52] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account approval for Florian Cuny - https://phabricator.wikimedia.org/T336734 (10Poslovitch) Thank you for handling this request, but it seems I am unable to create a group. Is that due to the fact the GitLab is still being rolled out? [18:17:18] 10GitLab (Account Approval), 10Release-Engineering-Team: Requesting GitLab account approval for Florian Cuny - https://phabricator.wikimedia.org/T336734 (10brennen) See here for group creation: https://phabricator.wikimedia.org/maniphest/task/edit/form/105/ [19:10:49] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10User-brennen: Define a permissions model for the /repos/mediawiki/ namespace on GitLab - https://phabricator.wikimedia.org/T336807 (10brennen) [19:12:08] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10User-brennen: Migrate mediawiki/ namespace from Gerrit to GitLab - https://phabricator.wikimedia.org/T335921 (10brennen) [19:33:42] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10User-brennen: Define a permissions model for the /repos/mediawiki/ namespace on GitLab - https://phabricator.wikimedia.org/T336807 (10brennen) [19:36:54] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10User-brennen: Migrate mediawiki/ namespace from Gerrit to GitLab - https://phabricator.wikimedia.org/T335921 (10brennen) [19:59:22] hey, this logstash_checker.py that is what only humans run manually to check things, am I right? [19:59:25] as in https://gerrit.wikimedia.org/r/c/operations/puppet/+/919365 [19:59:50] risk low? [20:00:26] would close a ticket from 2019, actually use urllib3.exceptions.ConnectionError [20:06:16] mutante: logstash_checker.py is used by scap during its "canary checks" phase. [20:06:56] oooh, that, i have mixed it up. thanks dancy [20:07:21] carefully deploying puppet change on gerrit hosts [20:07:43] that will be noop in prod but let us re-enable puppet on gerrit1001.. WITHOUT starting gerrit service [20:07:55] but only then we can disable monitoring and keep it around for the grace period [20:08:09] verified on 2002 first it was noop [20:08:58] 10Gerrit, 10VPS-project-Codesearch, 10VPS-project-Extdist, 10serviceops-collab, 10Patch-For-Review: Move clients off of gerrit-replica.wikimedia.org back to gerrit.wikimedia.org - https://phabricator.wikimedia.org/T336710 (10hashar) I am replying here to the [[ https://lists.wikimedia.org/hyperkitty/list... [20:12:59] all is good [20:13:18] now gerrit service is masked on gerrit1001 [20:13:27] but puppet runs there for the unrelated things [20:13:50] that is what the patch was supposed to do .. and it was complete noop on gerrit1003/prod [20:14:31] now I can actually disable monitoring for the old host, but still in grace period. laters [20:23:37] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10User-brennen: Define a permissions model for the /repos/mediawiki/ namespace on GitLab - https://phabricator.wikimedia.org/T336807 (10taavi) > This should probably be explicit about group membership imported from LDAP groups. Yeah. I... [20:26:34] (03PS2) 10Hashar: Do not carry Verified score on no code change [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/919831 (https://phabricator.wikimedia.org/T336660) [20:27:04] (03CR) 10Hashar: Do not carry Verified score on no code change (031 comment) [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/919831 (https://phabricator.wikimedia.org/T336660) (owner: 10Hashar) [20:28:51] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10User-brennen: Define a permissions model for the /repos/mediawiki/ namespace on GitLab - https://phabricator.wikimedia.org/T336807 (10brennen) > Who would that be? I guess a similar set of people who are currently Gerrit admins or ma... [20:31:32] (03CR) 10Hashar: [C: 03+2] Rename security-api to ipoid (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/919873 (https://phabricator.wikimedia.org/T336218) (owner: 10Tchanders) [20:32:11] icinga checks for gerrit1001 being removed [20:54:06] 10GitLab (Project Migration), 10Release-Engineering-Team (They Live 🕶️🧟), 10User-brennen: Define a permissions model for the /repos/mediawiki/ namespace on GitLab - https://phabricator.wikimedia.org/T336807 (10hashar) Implied by #mediawiki-gerrit-group-requests is the policy https://www.mediawiki.org/wiki/Ge... [21:16:24] PROBLEM - Check systemd state on doc1002 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc2002.codfw.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [21:34:11] 10GitLab (Infrastructure), 10Release-Engineering-Team (Radar), 10serviceops-collab: Add GitLab upgrades and maintenance to deployment calendar - https://phabricator.wikimedia.org/T336470 (10brennen) [22:13:44] RECOVERY - Check systemd state on doc1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state