[00:21:32] 10Phabricator, 10Release-Engineering-Team (Next), 10serviceops: Deprecate git-ssh service on phabricator.wikimedia.org - https://phabricator.wikimedia.org/T296022 (10Dzahn) [00:21:40] 10Phabricator, 10serviceops: test pushing to phabricator repos over https - https://phabricator.wikimedia.org/T301889 (10Dzahn) [00:22:41] 10Phabricator, 10serviceops: test pushing to phabricator repos over https - https://phabricator.wikimedia.org/T301889 (10Dzahn) [06:47:07] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.4.1 - https://phabricator.wikimedia.org/T302464 (10Joe) 05Open→03In progress a:03Joe [06:47:13] 10Release-Engineering-Team (Done by Feb 23🔥), 10MW-on-K8s, 10serviceops, 10Patch-For-Review: Build MediaWiki images for kubernetes on the deployment servers - https://phabricator.wikimedia.org/T297673 (10Joe) [07:03:11] 10Release-Engineering-Team, 10Scap, 10serviceops: Deploy Scap version 4.4.1 - https://phabricator.wikimedia.org/T302464 (10Joe) I installed the new scap on restbase-dev, the mw canaries and the deployment servers. Our smoke tests seem all to work, we will update the rest of production later today. [08:18:43] (03CR) 10Hashar: [C: 03+2] "I have updated the integration-quibble-fullrun-sqlite job" [integration/config] - 10https://gerrit.wikimedia.org/r/766798 (https://phabricator.wikimedia.org/T302706) (owner: 10Kosta Harlan) [08:20:43] (03Merged) 10jenkins-bot: Revert "jjb: Skip api-testing for Quibble fullrun with SQLite" [integration/config] - 10https://gerrit.wikimedia.org/r/766798 (https://phabricator.wikimedia.org/T302706) (owner: 10Kosta Harlan) [08:27:40] (03PS1) 10Hashar: jjb: add wrappers to integration-quibble-fullrun-extensions [integration/config] - 10https://gerrit.wikimedia.org/r/767048 [08:27:58] (03CR) 10Hashar: [C: 03+2] jjb: add wrappers to integration-quibble-fullrun-extensions [integration/config] - 10https://gerrit.wikimedia.org/r/767048 (owner: 10Hashar) [08:29:46] (03Merged) 10jenkins-bot: jjb: add wrappers to integration-quibble-fullrun-extensions [integration/config] - 10https://gerrit.wikimedia.org/r/767048 (owner: 10Hashar) [09:13:53] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/VueTest] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767052 [09:13:55] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [extensions/VueTest] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767052 (owner: 10QChris) [09:14:03] (03PS1) 10QChris: Import done. Revoke import grants [extensions/VueTest] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767053 [09:14:05] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [extensions/VueTest] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/767053 (owner: 10QChris) [09:17:27] (03PS2) 10Hashar: deployment-calendar: run tests via tox for CI [tools/release] - 10https://gerrit.wikimedia.org/r/730428 [09:18:45] (03CR) 10jerkins-bot: [V: 04-1] deployment-calendar: run tests via tox for CI [tools/release] - 10https://gerrit.wikimedia.org/r/730428 (owner: 10Hashar) [09:20:29] (03CR) 10Hashar: "Rebased" [tools/release] - 10https://gerrit.wikimedia.org/r/730428 (owner: 10Hashar) [09:33:52] (03PS1) 10Func: zuul: Add Bhsd to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/767057 [09:34:30] (03PS2) 10Func: zuul: Add Bhsd to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/767057 [09:41:11] hm, why does https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20220308T0200 say wmf/1.38.0-wmf.25? the branch that was cut (and will be deployed this week) is wmf.24, not wmf.25 [09:41:54] oh wait, that’s next week’s window. nevermind [09:42:39] (https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20220301T0200 is correct) [09:49:55] (03PS1) 10Hashar: Move make-deploy-notes requirements to its own file [tools/release] - 10https://gerrit.wikimedia.org/r/767058 [09:51:59] (03PS1) 10Hashar: Remove python2.7 support in tox [tools/release] - 10https://gerrit.wikimedia.org/r/767059 [09:55:31] (03PS1) 10Hashar: Move make-release to a standalone tox environment [tools/release] - 10https://gerrit.wikimedia.org/r/767060 [10:01:14] (03PS2) 10Hashar: Remove python2.7 support in tox [tools/release] - 10https://gerrit.wikimedia.org/r/767059 [10:01:16] (03PS2) 10Hashar: Move make-release to a standalone tox environment [tools/release] - 10https://gerrit.wikimedia.org/r/767060 [10:01:18] (03PS3) 10Hashar: deployment-calendar: run tests via tox for CI [tools/release] - 10https://gerrit.wikimedia.org/r/730428 [10:47:43] 10Phabricator (Upstream): Send reminder email on open/stalled tasks when Due Date is reached - https://phabricator.wikimedia.org/T302785 (10Aklapper) [11:17:14] 10Phabricator, 10Release-Engineering-Team: Puppet failure on phabricator-stage-1001.devtools - https://phabricator.wikimedia.org/T299997 (10hashar) I have deleted `/srv/deployment/phabricator/deployment-cache/` and manually ran: sudo -u deploy-devtools /usr/bin/scap deploy-local --repo phabricator/deploym... [11:24:49] 10Project-Admins, 10Tool-DrTrigonBot---General: Archive DrTrigonBot* Phab projects and decline its open tasks - https://phabricator.wikimedia.org/T300969 (10Aklapper) @DrTrigon: >>! In T300969#7724397, @DrTrigon wrote: > 1. What tool exactly? See the task description: https://drtrigonbot.toolforge.org/ > 2... [11:40:23] (03PS1) 10Kosta Harlan: jjb: Separate job for Quibble fullrun PHPUnit tests [integration/config] - 10https://gerrit.wikimedia.org/r/767081 [11:54:18] (03CR) 10Hashar: [C: 03+2] jjb: Separate job for Quibble fullrun PHPUnit tests [integration/config] - 10https://gerrit.wikimedia.org/r/767081 (owner: 10Kosta Harlan) [11:56:08] (03Merged) 10jenkins-bot: jjb: Separate job for Quibble fullrun PHPUnit tests [integration/config] - 10https://gerrit.wikimedia.org/r/767081 (owner: 10Kosta Harlan) [11:56:36] (03PS1) 10Jaime Nuche: cli config args: passthrough to local `scap` subprocess calls [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) [11:57:23] (03CR) 10jerkins-bot: [V: 04-1] cli config args: passthrough to local `scap` subprocess calls [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) (owner: 10Jaime Nuche) [12:23:41] (03PS2) 10Jaime Nuche: cli config args: passthrough to local `scap` subprocess calls [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) [12:24:48] (03CR) 10jerkins-bot: [V: 04-1] cli config args: passthrough to local `scap` subprocess calls [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) (owner: 10Jaime Nuche) [12:26:40] (03PS3) 10Jaime Nuche: cli config args: passthrough to local `scap` subprocess calls [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) [12:27:43] (03CR) 10jerkins-bot: [V: 04-1] cli config args: passthrough to local `scap` subprocess calls [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) (owner: 10Jaime Nuche) [12:30:41] (03PS4) 10Jaime Nuche: cli config args: passthrough to local `scap` subprocess calls [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) [12:31:16] (03CR) 10jerkins-bot: [V: 04-1] cli config args: passthrough to local `scap` subprocess calls [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) (owner: 10Jaime Nuche) [12:35:43] (03PS5) 10Jaime Nuche: cli config args: passthrough to local `scap` subprocess calls [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) [12:36:51] (03CR) 10jerkins-bot: [V: 04-1] cli config args: passthrough to local `scap` subprocess calls [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) (owner: 10Jaime Nuche) [12:41:03] (03PS6) 10Jaime Nuche: cli config args: passthrough to local `scap` subprocess calls [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) [12:41:07] 10Project-Admins, 10Tool-DrTrigonBot---General: Archive DrTrigonBot* Phab projects and decline its open tasks - https://phabricator.wikimedia.org/T300969 (10DrTrigon) > > 1. What tool exactly? > > See the task description: https://drtrigonbot.toolforge.org/ drtrigonbot used to do several things - more spe... [13:10:29] (03CR) 10Hashar: [C: 03+2] zuul: Add Bhsd to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/767057 (owner: 10Func) [13:12:18] (03Merged) 10jenkins-bot: zuul: Add Bhsd to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/767057 (owner: 10Func) [13:43:33] 10GitLab (Infrastructure), 10Cloud-VPS (Project-requests): Create additional network port in project devtools - https://phabricator.wikimedia.org/T302803 (10Jelto) [13:48:12] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Jelto) [13:48:20] 10GitLab (Infrastructure), 10Cloud-VPS (Project-requests): Create additional network port in project devtools - https://phabricator.wikimedia.org/T302803 (10Jelto) [13:53:25] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Jelto) SSH access to the test instance is not working because of different networking behavior on WMCS/VPS. The public floating IP ("service ip") is NATed to the... [13:57:45] 10Deployments, 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Patch-For-Review: deploy mediawiki/tools/releases on deployment server with scap - https://phabricator.wikimedia.org/T274255 (10hashar) 05Open→03Declined @jnuche talked about moving the scripts `deploy-promote` and `stage-train` from {nav medi... [14:02:54] 10Release-Engineering-Team (Doing), 10Scap, 10serviceops, 10Patch-For-Review: scap's canary check gives confusing logstash link - https://phabricator.wikimedia.org/T291870 (10hashar) 05Open→03Declined My idea was to use a semantic tag instead of a hardcoded list of hosts. Then revisiting this tasks a... [14:13:45] (03CR) 10Hashar: [C: 03+2] [WMF] script to build our plugins [software/gerrit] (wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/756108 (owner: 10Hashar) [14:26:59] (03Merged) 10jenkins-bot: [WMF] script to build our plugins [software/gerrit] (wmf/stable-3.4) - 10https://gerrit.wikimedia.org/r/756108 (owner: 10Hashar) [14:27:16] (03CR) 10Jaime Nuche: [C: 04-2] cli config args: passthrough to local `scap` subprocess calls [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) (owner: 10Jaime Nuche) [14:35:56] (03PS7) 10Jaime Nuche: cli config args: passthrough to local `scap` subprocess calls [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) [14:37:48] (03CR) 10jerkins-bot: [V: 04-1] cli config args: passthrough to local `scap` subprocess calls [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) (owner: 10Jaime Nuche) [14:37:53] (03PS8) 10Jaime Nuche: cli config args: passthrough to local `scap` subprocess calls [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) [14:38:25] (03CR) 10jerkins-bot: [V: 04-1] cli config args: passthrough to local `scap` subprocess calls [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) (owner: 10Jaime Nuche) [14:41:34] (03PS9) 10Jaime Nuche: cli config args: passthrough to local `scap` subprocess calls [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) [14:45:55] (03PS10) 10Jaime Nuche: cli config args: passthrough to local `scap` subprocess calls [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) [16:32:03] (03CR) 10Ahmon Dancy: [C: 04-1] cli config args: passthrough to local `scap` subprocess calls (034 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) (owner: 10Jaime Nuche) [17:06:09] 10Project-Admins, 10Tool-DrTrigonBot---General: Archive DrTrigonBot* Phab projects and decline its open tasks - https://phabricator.wikimedia.org/T300969 (10Aklapper) >>! In T300969#7744297, @DrTrigon wrote: >> > 1. What tool exactly? >> >> See the task description: https://drtrigonbot.toolforge.org/ > >... [17:15:40] PROBLEM - Check systemd state on contint2001 is CRITICAL: CRITICAL - degraded: The following units failed: ipmiseld.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [17:58:25] 10Phabricator: When a project is not public, it appears to users outside "Visible To" criteria as "Unknown Object", which is confusing - https://phabricator.wikimedia.org/T302525 (10Aklapper) Heh... I think the current behavior makes sense as it is consistent with tasks or applications (Herald) that users with d... [18:06:46] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T300201 (10matmarex) [18:06:48] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T300200 (10matmarex) [18:10:28] i just noticed dduvall is listed as train backup on T300200 while the deployment calendar has dancy. unsure which is correct, although i assume it's the ticket. [18:10:28] T300200: 1.38.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T300200 [18:13:35] brennen: I'm pretty sure it's me [18:13:47] * dduvall checks etherpad [18:14:42] yep, the etherpad and task say it's me :) [18:16:41] 10GitLab (Infrastructure), 10Cloud-VPS (Project-requests): Create additional network port in project devtools - https://phabricator.wikimedia.org/T302803 (10Majavah) 05Open→03Resolved a:03Majavah Done: * Reserved secondary IP `172.16.7.146` by creating a new Neutron port * Configured the primary port of... [18:16:45] yeah, looks like the calendar is a bit weird now that i glance at it again - showing wmf.25 for today [18:16:50] 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Majavah) [18:17:01] wait, no it's not. hrm. [18:17:26] window jumping button is just hitting march 8th. [18:18:03] ...but the entry for the 8th is a bit muddled. cc: thcipriani [18:18:27] uhhh [18:20:41] yes, dan is backup [18:20:51] I must've messed up the calendar entry [18:21:06] er, messed up the task which is the source of the calendar entry [18:21:09] * thcipriani corrects both [18:22:52] uh, calendar and task look right [18:23:05] the "jump to current event" button seems to jump to next week for some reason [18:23:50] which is probably the source of the confusion :) [18:26:12] ah, yeah, you're right - entries are correct. [19:28:56] 10GitLab (Project Migration), 10Release-Engineering-Team (Done by Feb 23🔥): Create new GitLab project group: API Platform - https://phabricator.wikimedia.org/T301164 (10WDoranWMF) @brennen @thcipriani Would it be possible to get a sense of when this might complete? It's moving in the direction of a blocker for... [19:30:08] 10GitLab (Project Migration), 10Release-Engineering-Team (Done by Feb 23🔥): Create new GitLab project group: API Platform - https://phabricator.wikimedia.org/T301164 (10brennen) 05Open→03In progress a:03brennen One moment... [19:33:02] (03CR) 10Jaime Nuche: cli config args: passthrough to local `scap` subprocess calls (034 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/767084 (https://phabricator.wikimedia.org/T302487) (owner: 10Jaime Nuche) [20:10:03] 10GitLab (Project Migration), 10Release-Engineering-Team (Done by Feb 23🔥): Create new GitLab project group: API Platform - https://phabricator.wikimedia.org/T301164 (10brennen) 05In progress→03Resolved [20:15:19] dancy: I understand why the state is out of sync regarding the alert. What's escaping me is why we didn't get the alert in the other direction in the past. It seems inevitable that deploy and mw* aren't always in sync at all times exactly. The script [1] [2] has affordances for this using a DEFAULT_WINDOW of 60 (seconds). I guess that's it then. We used to sync the deploy server first (the source of comparison) and... within a minute [20:15:19] sync to all? Maybe. But if that was true, then I would expect us to still be fine since the dpeloy server among the list would still take a minute. The check is performed against mtime of the deploy source not of the current appserver. So I guess the issue is that the alert fires on mw* servers behind deploy* in the list, because presumably deploy* isn't the first in the list and/or not ensured to finish before any other. So we have a [20:15:19] chance of some checking themselves with the new file but deploy not having it yet. [20:15:39] If none of this is new or useful, feel free to discard :) This is what I gathered while looking into it briefly. [20:16:15] 10GitLab (Project Migration), 10Release-Engineering-Team (Done by Feb 23🔥): Create new GitLab project group: API Platform - https://phabricator.wikimedia.org/T301164 (10brennen) Apologies; I've let several of these pile up again while thinking about simplifying our access scheme for projects. These are, to be... [20:25:08] (03PS1) 10Subramanya Sastry: Suppress talk page header shown on some wikis [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/767239 [20:31:27] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T300200 (10brennen) [20:38:36] (03PS2) 10Subramanya Sastry: Suppress talk page header + DiscussionTools reply links [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/767239 [20:42:41] 10Project-Admins, 10Tool-DrTrigonBot---General: Archive DrTrigonBot* Phab projects and decline its open tasks - https://phabricator.wikimedia.org/T300969 (10DrTrigon) > >>> 1. What tool exactly? > >> > >> See the task description: https://drtrigonbot.toolforge.org/ > > > > drtrigonbot used to do several t... [20:44:26] Krinkle: That's exactly right. deploy1002:/srv/mediawiki used to be updated very early in the sync-world process. Now it's updated at some random time during sync-apaches, which could take 30-40 minutes on the first train operation of the week. [20:50:55] but seems fast enough not to trigger alerts once you're just doing a quick sync-wikiversions, i think [20:59:33] Yeah. The checker has an allowance of 60 minutes out of date. [20:59:49] But in the bad case it was looking at a wikiversions.json that was updated last week. [22:02:31] (03PS3) 10Subramanya Sastry: Adapt HTML postprocessing for talk namespace and DiscussionTools [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/767239 [22:09:16] (03PS4) 10Subramanya Sastry: Adapt HTML postprocessing for talk namespace and DiscussionTools [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/767239 [23:11:39] 10Phabricator, 10Release-Engineering-Team (Done by Feb 23🔥), 10User-brennen: Phabricator admin knowledge transfer - https://phabricator.wikimedia.org/T300693 (10brennen) 05In progress→03Resolved [23:14:53] 10GitLab (Integrations), 10preview-environment, 10Patch-For-Review, 10User-brennen, 10User-jeena: Spike: Research hosts for preview environment - https://phabricator.wikimedia.org/T283894 (10brennen)