[00:32:19] 10Release-Engineering-Team (Radar), 10serviceops-collab: discuss whether to keep the gerrit-test instance in devtools - https://phabricator.wikimedia.org/T330248 (10Dzahn) [01:03:29] (03CR) 10Arlolra: [C: 03+2] Update wiki-specific CSS based on bug fixes to Cite CSS script [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/890916 (owner: 10Subramanya Sastry) [01:04:07] (03Merged) 10jenkins-bot: Update wiki-specific CSS based on bug fixes to Cite CSS script [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/890916 (owner: 10Subramanya Sastry) [02:00:50] (03CR) 10Jforrester: [C: 03+2] Zuul: Drop legacy node12 template, unused [integration/config] - 10https://gerrit.wikimedia.org/r/890925 (owner: 10Jforrester) [02:01:37] (03CR) 10Jforrester: [C: 03+2] jjb: Drop generic-node12-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/890926 (owner: 10Jforrester) [02:01:46] (03CR) 10Jforrester: [C: 03+2] Docker: Drop node12 images, all now unused [integration/config] - 10https://gerrit.wikimedia.org/r/890927 (owner: 10Jforrester) [02:02:20] (03Merged) 10jenkins-bot: Zuul: Drop legacy node12 template, unused [integration/config] - 10https://gerrit.wikimedia.org/r/890925 (owner: 10Jforrester) [02:02:50] (03Merged) 10jenkins-bot: jjb: Drop generic-node12-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/890926 (owner: 10Jforrester) [02:02:52] (03Merged) 10jenkins-bot: Docker: Drop node12 images, all now unused [integration/config] - 10https://gerrit.wikimedia.org/r/890927 (owner: 10Jforrester) [02:03:31] (Queue (Jenkins jobs + Zuul functions) alert) firing: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [02:08:31] (Queue (Jenkins jobs + Zuul functions) alert) resolved: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [02:26:00] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/890928 (owner: 10Jforrester) [02:26:08] (03CR) 10Jforrester: [C: 03+2] Zuul: [operations/software/gerrit] Switch node job to node16 [integration/config] - 10https://gerrit.wikimedia.org/r/890930 (owner: 10Jforrester) [02:26:39] (03CR) 10Jforrester: [C: 03+2] jjb: Drop gerrit-node14-docker [integration/config] - 10https://gerrit.wikimedia.org/r/890931 (owner: 10Jforrester) [02:27:07] (03Merged) 10jenkins-bot: jjb: Provide gerrit-node16-docker [integration/config] - 10https://gerrit.wikimedia.org/r/890928 (owner: 10Jforrester) [02:27:16] (03Merged) 10jenkins-bot: Zuul: [operations/software/gerrit] Switch node job to node16 [integration/config] - 10https://gerrit.wikimedia.org/r/890930 (owner: 10Jforrester) [02:27:46] (03Merged) 10jenkins-bot: jjb: Drop gerrit-node14-docker [integration/config] - 10https://gerrit.wikimedia.org/r/890931 (owner: 10Jforrester) [02:37:00] (03PS1) 10Jforrester: jjb: Provide branchdeploy-wikidata-query-builder-node16-npm-docker [integration/config] - 10https://gerrit.wikimedia.org/r/890943 (https://phabricator.wikimedia.org/T314470) [02:38:48] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/890932 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [02:38:54] (03CR) 10Jforrester: [C: 03+2] Zuul: Make mwgate-node16-docker jobs run in series [integration/config] - 10https://gerrit.wikimedia.org/r/890935 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [02:39:57] (03Merged) 10jenkins-bot: jjb: Provide node16 versions of non-generic jobs [integration/config] - 10https://gerrit.wikimedia.org/r/890932 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [02:40:01] (03Merged) 10jenkins-bot: Zuul: Make mwgate-node16-docker jobs run in series [integration/config] - 10https://gerrit.wikimedia.org/r/890935 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [02:40:40] (03PS2) 10Jforrester: jjb: Provide branchdeploy-wikidata-query-builder-node16-npm-docker [integration/config] - 10https://gerrit.wikimedia.org/r/890943 (https://phabricator.wikimedia.org/T314470) [02:42:03] (03CR) 10Jforrester: [C: 03+2] jjb: [oojs-ui-doxygen-publish] Switch to node16 [integration/config] - 10https://gerrit.wikimedia.org/r/890938 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [02:42:06] (03CR) 10Jforrester: [C: 03+2] jjb: Provide branchdeploy-wikidata-query-builder-node16-npm-docker [integration/config] - 10https://gerrit.wikimedia.org/r/890943 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [02:42:19] (03CR) 10Jforrester: [C: 03+2] Zuul: Migrate all non-MediaWiki-land node jobs to node16 [integration/config] - 10https://gerrit.wikimedia.org/r/890936 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [02:43:12] (03Merged) 10jenkins-bot: jjb: [oojs-ui-doxygen-publish] Switch to node16 [integration/config] - 10https://gerrit.wikimedia.org/r/890938 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [02:43:22] (03Merged) 10jenkins-bot: jjb: Provide branchdeploy-wikidata-query-builder-node16-npm-docker [integration/config] - 10https://gerrit.wikimedia.org/r/890943 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [02:43:25] (03Merged) 10jenkins-bot: Zuul: Migrate all non-MediaWiki-land node jobs to node16 [integration/config] - 10https://gerrit.wikimedia.org/r/890936 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [02:43:48] !log Zuul: Migrate all non-MediaWiki-land node jobs to node16 for T314470 [02:43:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:43:51] T314470: Upgrade all CI jobs for WMF-deployed projects from Node 14 to Node 16 - https://phabricator.wikimedia.org/T314470 [02:44:08] (03CR) 10Jforrester: [C: 03+2] Zuul: Migrate all MediaWiki-land node jobs to node16 [integration/config] - 10https://gerrit.wikimedia.org/r/890937 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [02:45:24] (03Merged) 10jenkins-bot: Zuul: Migrate all MediaWiki-land node jobs to node16 [integration/config] - 10https://gerrit.wikimedia.org/r/890937 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [02:45:43] !log Zuul: Migrate all MediaWiki-land node jobs to node16 for T314470 [02:45:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:46:25] (03CR) 10Jforrester: [C: 03+2] Zuul: Migrate further MediaWiki-adjunct node code testing from 14 to 16 [integration/config] - 10https://gerrit.wikimedia.org/r/890939 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [02:47:35] (03Merged) 10jenkins-bot: Zuul: Migrate further MediaWiki-adjunct node code testing from 14 to 16 [integration/config] - 10https://gerrit.wikimedia.org/r/890939 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [02:48:04] !log Zuul: Migrate further MediaWiki-adjunct node code testing from 14 to 16 for T314470 [02:48:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:56:52] (03CR) 10Jforrester: [C: 03+2] Zuul: Drop node14 jobs, all now unused [integration/config] - 10https://gerrit.wikimedia.org/r/890941 (owner: 10Jforrester) [02:58:00] (03Merged) 10jenkins-bot: Zuul: Drop node14 jobs, all now unused [integration/config] - 10https://gerrit.wikimedia.org/r/890941 (owner: 10Jforrester) [03:03:55] 10Continuous-Integration-Config, 10Patch-For-Review: Upgrade all CI jobs for WMF-deployed projects from Node 14 to Node 16 - https://phabricator.wikimedia.org/T314470 (10Jdforrester-WMF) [03:04:10] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/890942 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [03:05:16] (03Merged) 10jenkins-bot: jjb: Drop node14 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/890942 (https://phabricator.wikimedia.org/T314470) (owner: 10Jforrester) [03:07:55] (03PS1) 10Jforrester: jjb: Switch incidental uses of nodejs images from 14 to 16 [integration/config] - 10https://gerrit.wikimedia.org/r/890944 [03:07:57] (03PS1) 10Jforrester: Dockerfiles: [ajv] Switch parent image from node14 to node16 [integration/config] - 10https://gerrit.wikimedia.org/r/890945 [03:07:59] (03PS1) 10Jforrester: Docker: Drop node14 images, all now unused [integration/config] - 10https://gerrit.wikimedia.org/r/890946 [03:08:05] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/890944 (owner: 10Jforrester) [03:09:14] (03CR) 10Jforrester: [C: 03+2] Dockerfiles: [ajv] Switch parent image from node14 to node16 [integration/config] - 10https://gerrit.wikimedia.org/r/890945 (owner: 10Jforrester) [03:09:16] (03Merged) 10jenkins-bot: jjb: Switch incidental uses of nodejs images from 14 to 16 [integration/config] - 10https://gerrit.wikimedia.org/r/890944 (owner: 10Jforrester) [03:09:18] (03CR) 10Jforrester: [C: 03+2] Docker: Drop node14 images, all now unused [integration/config] - 10https://gerrit.wikimedia.org/r/890946 (owner: 10Jforrester) [03:10:31] (03Merged) 10jenkins-bot: Dockerfiles: [ajv] Switch parent image from node14 to node16 [integration/config] - 10https://gerrit.wikimedia.org/r/890945 (owner: 10Jforrester) [03:10:33] (03Merged) 10jenkins-bot: Docker: Drop node14 images, all now unused [integration/config] - 10https://gerrit.wikimedia.org/r/890946 (owner: 10Jforrester) [07:14:10] 10Gerrit, 10Release-Engineering-Team (Radar), 10serviceops-collab: discuss whether to keep the gerrit-test instance in devtools - https://phabricator.wikimedia.org/T330248 (10hashar) That is `gerrit-prod-1001.devtools.eqiad1.wikimedia.cloud` isn't it? I use it to test upgrades, scap deployment or Puppet refa... [08:20:09] 10Continuous-Integration-Config, 10Patch-For-Review: Upgrade all CI jobs for WMF-deployed projects from Node 14 to Node 16 - https://phabricator.wikimedia.org/T314470 (10kostajh) @Jdforrester-WMF Should we update npm to version 8 in the node 16 jobs? I see this warning from rollup in https://integration.wikime... [09:08:11] 10Phabricator: Phabricator catch-all task should be at bottom - https://phabricator.wikimedia.org/T330262 (10Jidanni) [09:09:16] 10Phabricator: Phabricator catch-all task should be at bottom - https://phabricator.wikimedia.org/T330262 (10Jidanni) [09:47:03] 10Release-Engineering-Team, 10serviceops: scap build-and-push-container-images failed due to backend fetch error - https://phabricator.wikimedia.org/T330264 (10hashar) 05Open→03Declined I have filed it to keep track of the issue in case it occurs again in the future. [09:53:49] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Jelto) [09:55:07] 10Phabricator: Phabricator catch-all task should be at bottom - https://phabricator.wikimedia.org/T330262 (10hashar) This is what I get, given I have some extra privileges / customized the entries: {F36864713 size=full} The list can be amended by clicking {nav Edit Favorites} which brings you to https://phabri... [10:10:46] FYI, thcipriani, I added the datacenter switchover windows to the deployment calendar, and strokethrough the conflicting windows to signify cancellation. If there is a better/more canonical way of doing it, I'm all ears :) [10:17:05] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Jelto) [10:17:39] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Jelto) [10:20:04] 10Continuous-Integration-Infrastructure: Revisit how CI works for released branches - https://phabricator.wikimedia.org/T298043 (10zeljkofilipin) Is this still relevant, since MobileFrontend doesn't have WebdriverIO tests since [[ https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MobileFrontend/+/859535 | 85... [10:23:04] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10Browser-Tests, 10User-zeljkofilipin: Jenkins job should fail when all (selenium) tests are skipped - https://phabricator.wikimedia.org/T324480 (10zeljkofilipin) [10:30:34] maintenance-disconnect-full-disks build 467702 integration-agent-docker-1028 (/: 29%, /srv: 95%, /var/lib/docker: 45%): OFFLINE due to disk space [10:35:37] maintenance-disconnect-full-disks build 467703 integration-agent-docker-1028 (/: 29%, /srv: 64%, /var/lib/docker: 42%): RECOVERY disk space OK [10:42:13] hmmm, the job names for still-queued jobs didn’t use to be links, did they? https://usercontent.irccloud-cdn.com/file/XQxMP9OC/image.png [10:42:19] (if you’ll pardon my broken english ^^) [10:42:30] the last link is https://integration.wikimedia.org/ci/job/mwext-php74-phan-docker/None/console, so, not quite right anyway [10:43:59] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Jelto) [10:45:36] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Jelto) [10:48:28] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Jelto) [11:05:41] (03PS1) 10Kosta Harlan: doc: Add GrowthExperiments to MediaWiki components [integration/docroot] - 10https://gerrit.wikimedia.org/r/891255 (https://phabricator.wikimedia.org/T329034) [11:16:30] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Jelto) [11:20:40] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10eoghan) [11:24:54] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Jelto) [11:36:38] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Jelto) [11:37:25] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Jelto) [11:37:37] 10GitLab (Infrastructure), 10serviceops-collab: Investigate gitlab maintenance/read-only mode - https://phabricator.wikimedia.org/T330277 (10eoghan) [12:03:31] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover, 10Patch-For-Review: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10eoghan) [12:10:44] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10eoghan) [12:12:29] 10GitLab (Infrastructure), 10serviceops-collab: gitlab_default_can_create_group setting deprecation - https://phabricator.wikimedia.org/T330282 (10eoghan) [12:27:58] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10eoghan) [12:32:19] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Jelto) [12:45:13] 10Phabricator: Phabricator catch-all task should be at bottom - https://phabricator.wikimedia.org/T330262 (10Aklapper) 05Open→03Invalid @Jidanni: It would be great if you could consider bringing up things in support forums first. Numerous invalid / mis-filed tickets got created lately and that's not the best... [12:52:46] back! [12:53:21] jnuche: I am adding a new credential to the CI Jenkins for kostajh , then I guess we don't have the yaml files for the CI Jenkins yet so should not affect anything [12:53:59] hashar: sounds good [13:06:28] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity: Provide early feedback when a patch has job failures - https://phabricator.wikimedia.org/T323750 (10hashar) I have added the API key to the Jenkins credential store with the id `quibble-earlywarningbot-api-key` (private link https:... [13:20:02] (03CR) 10Zfilipin: [C: 03+2] zuul: [mediawiki/extensions/DismissableSiteNotice] Enable Sonar Codehealth (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/889197 (https://phabricator.wikimedia.org/T321837) (owner: 10Pwangai) [13:29:24] 10Continuous-Integration-Infrastructure, 10Wikimedia-Portals: wikimedia-portals-build permission error: "dubious ownership in repository" - https://phabricator.wikimedia.org/T330194 (10Ladsgroup) Maybe this didn't get deployed? https://gerrit.wikimedia.org/r/c/integration/config/+/886927/1/jjb/wikimedia.yaml... [13:40:00] (03PS1) 10Hashar: Add GitHub API key to mwcore-phpunit-coverage-master [integration/config] - 10https://gerrit.wikimedia.org/r/891274 [13:41:53] 10Scap, 10Prod-Kubernetes: scap deploys are taking > 30 minutes due to docker images timing out - https://phabricator.wikimedia.org/T330291 (10Ladsgroup) [13:42:10] 10Scap, 10Prod-Kubernetes: scap deploys are taking > 30 minutes due to docker images timing out - https://phabricator.wikimedia.org/T330291 (10Ladsgroup) p:05Triage→03Unbreak! [13:42:28] 10Scap, 10Prod-Kubernetes, 10serviceops: scap deploys are taking > 30 minutes due to docker images timing out - https://phabricator.wikimedia.org/T330291 (10Ladsgroup) [14:05:30] (03PS2) 10Hashar: Add GitHub API key for composer to some jobs [integration/config] - 10https://gerrit.wikimedia.org/r/891274 [14:05:32] (03PS1) 10Hashar: jjb: add QUIBBLE_API_KEY to Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/891278 (https://phabricator.wikimedia.org/T323750) [14:13:19] (03PS3) 10Hashar: Add GitHub API key to mwcore-phpunit-coverage-master [integration/config] - 10https://gerrit.wikimedia.org/r/891274 [14:14:53] (03PS2) 10Hashar: jjb: add QUIBBLE_API_KEY to Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/891278 (https://phabricator.wikimedia.org/T323750) [14:15:12] (03CR) 10Hashar: [C: 03+2] Add GitHub API key to mwcore-phpunit-coverage-master [integration/config] - 10https://gerrit.wikimedia.org/r/891274 (owner: 10Hashar) [14:16:27] (03Merged) 10jenkins-bot: Add GitHub API key to mwcore-phpunit-coverage-master [integration/config] - 10https://gerrit.wikimedia.org/r/891274 (owner: 10Hashar) [14:17:25] 10Scap, 10Prod-Kubernetes, 10serviceops: scap deploys are taking > 30 minutes due to docker images timing out - https://phabricator.wikimedia.org/T330291 (10akosiaris) p:05Unbreak!→03High We added back the lost capacity (4 nodes) in T330048, we should be ok again. Lowering and once we got confirmation, w... [14:19:27] !log Updating all jobs to add `QUIBBLE_API_KEY` to the Quibble jobs | https://gerrit.wikimedia.org/r/c/integration/config/+/891278/ | T323750 [14:19:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:19:29] T323750: Provide early feedback when a patch has job failures - https://phabricator.wikimedia.org/T323750 [14:22:07] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team, 10SecTeam-Processed, and 2 others: Jenkins plugins security advisory - 2023-02-15 - https://phabricator.wikimedia.org/T329755 (10hashar) The newish configuration management has been updated to reflect those changes: https://git... [14:22:42] (03CR) 10Hashar: [C: 03+2] jjb: add QUIBBLE_API_KEY to Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/891278 (https://phabricator.wikimedia.org/T323750) (owner: 10Hashar) [14:23:32] 10Scap, 10Prod-Kubernetes, 10serviceops: scap deploys are taking > 30 minutes due to docker images timing out - https://phabricator.wikimedia.org/T330291 (10Ladsgroup) The broken one: ` 14:15:34 Synchronized wmf-config/core-Permissions.php: Move all of userrights config out of IS.php to a dedicated file, par... [14:24:06] (03Merged) 10jenkins-bot: jjb: add QUIBBLE_API_KEY to Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/891278 (https://phabricator.wikimedia.org/T323750) (owner: 10Hashar) [14:28:20] 10Scap, 10Prod-Kubernetes, 10serviceops: scap deploys are taking > 30 minutes due to docker images timing out - https://phabricator.wikimedia.org/T330291 (10akosiaris) 05Open→03Resolved a:03akosiaris Cool, I think we fixed this one. Thanks for raising it! [14:36:17] (03PS1) 10Reedy: layout.yaml: Disable FR cldr gate-and-submit job [integration/config] - 10https://gerrit.wikimedia.org/r/891281 (https://phabricator.wikimedia.org/T328240) [14:37:25] (03CR) 10CI reject: [V: 04-1] layout.yaml: Disable FR cldr gate-and-submit job [integration/config] - 10https://gerrit.wikimedia.org/r/891281 (https://phabricator.wikimedia.org/T328240) (owner: 10Reedy) [14:39:01] 10Continuous-Integration-Infrastructure, 10Wikimedia-Portals: wikimedia-portals-build permission error: "dubious ownership in repository" - https://phabricator.wikimedia.org/T330194 (10Ladsgroup) Maybe @hashar knows more :] [14:40:16] (03PS2) 10Reedy: layout.yaml: Disable FR cldr gate-and-submit job [integration/config] - 10https://gerrit.wikimedia.org/r/891281 (https://phabricator.wikimedia.org/T328240) [14:40:57] 10Continuous-Integration-Config, 10Patch-For-Review: Upgrade all CI jobs for WMF-deployed projects from Node 14 to Node 16 - https://phabricator.wikimedia.org/T314470 (10Jdforrester-WMF) >>! In T314470#8636102, @kostajh wrote: > @Jdforrester-WMF Should we update npm to version 8 in the node 16 jobs? I see this... [14:41:25] (03CR) 10CI reject: [V: 04-1] layout.yaml: Disable FR cldr gate-and-submit job [integration/config] - 10https://gerrit.wikimedia.org/r/891281 (https://phabricator.wikimedia.org/T328240) (owner: 10Reedy) [14:42:37] (03PS3) 10Reedy: layout.yaml: Disable FR cldr gate-and-submit job [integration/config] - 10https://gerrit.wikimedia.org/r/891281 (https://phabricator.wikimedia.org/T328240) [14:45:06] (03CR) 10Reedy: [C: 03+2] "Hopefully temporary to unblock some CLDR patch merges.. can be reverted at will (and I may do so later today)" [integration/config] - 10https://gerrit.wikimedia.org/r/891281 (https://phabricator.wikimedia.org/T328240) (owner: 10Reedy) [14:46:12] (03Merged) 10jenkins-bot: layout.yaml: Disable FR cldr gate-and-submit job [integration/config] - 10https://gerrit.wikimedia.org/r/891281 (https://phabricator.wikimedia.org/T328240) (owner: 10Reedy) [14:48:01] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/891281 T328240 [14:48:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:48:04] T328240: quibble-fundraising-cldr-REL1_39-php74-docker failure on CLDR master - https://phabricator.wikimedia.org/T328240 [14:57:00] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Jelto) After switching `gitlab2002` and `gitlab1003` I get a SSH key warning, although the hosts use the same host key: ` $ git push Wa... [15:07:40] 10Phabricator, 10Trust-and-Safety, 10cloud-services-team, 10wikitech.wikimedia.org: Reset 2FA for Developer account 'Rosalie Perside (WMDE)' and Phabricator account @Rosalie_WMDE - https://phabricator.wikimedia.org/T329179 (10Rosalie_WMDE) Thank you. Sorry for the delay I was out of office for a while. Her... [15:46:36] (03PS1) 10Urbanecm: [Zuul] Make tox-docker a voting job for operations/mediawiki-config [integration/config] - 10https://gerrit.wikimedia.org/r/891317 (https://phabricator.wikimedia.org/T329231) [15:49:19] 10Gerrit, 10serviceops-collab, 10Patch-For-Review: Issues with Gerrit test instance - https://phabricator.wikimedia.org/T329444 (10Dzahn) 05Declined→03Open [15:52:49] 10Gerrit, 10serviceops-collab, 10Patch-For-Review: Issues with Gerrit test instance - https://phabricator.wikimedia.org/T329444 (10Dzahn) Specifically the instance is down and can't keep using the current auth scheme. I am going to bring this up in today's IC meeting and I created follow-up task T330248. Th... [15:55:38] 10Gerrit, 10Release-Engineering-Team (Radar), 10serviceops-collab: discuss whether to keep the gerrit-test instance in devtools - https://phabricator.wikimedia.org/T330248 (10Dzahn) Yes, that's the one. I reopened T329444 as well. Currently the instance is down and we can't just start it again. We fixed thin... [16:09:28] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10serviceops-collab: Disallow direct access to the docker.io registry from gitlab runners (setup a mirror of docker hub) - https://phabricator.wikimedia.org/T329679 (10Dzahn) [16:09:54] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10serviceops-collab, 10serviceops-radar: Disallow direct access to the docker.io registry from gitlab runners (setup a mirror of docker hub) - https://phabricator.wikimedia.org/T329679 (10Dzahn) [16:14:46] 10GitLab (Infrastructure), 10serviceops-collab, 10Datacenter-Switchover: Switchover gitlab-replica (gitlab2002 -> gitlab1003) - https://phabricator.wikimedia.org/T329930 (10Dzahn) >>! In T329930#8637475, @Jelto wrote: > I'm not sure how we handled that last time. I think we announced the warning will appear... [16:20:49] (03CR) 10Hashar: [C: 03+2] doc: Add GrowthExperiments to MediaWiki components [integration/docroot] - 10https://gerrit.wikimedia.org/r/891255 (https://phabricator.wikimedia.org/T329034) (owner: 10Kosta Harlan) [16:21:31] (03Merged) 10jenkins-bot: doc: Add GrowthExperiments to MediaWiki components [integration/docroot] - 10https://gerrit.wikimedia.org/r/891255 (https://phabricator.wikimedia.org/T329034) (owner: 10Kosta Harlan) [16:22:30] (03CR) 10Hashar: [C: 03+2] "Looks like I forgot to deploy this change :-\ I did it right now." [integration/docroot] - 10https://gerrit.wikimedia.org/r/878860 (owner: 10Lucas Werkmeister (WMDE)) [16:22:59] hashar: I thought it was live already [16:29:41] (03CR) 10Hashar: [C: 03+2] "Deployed. Note https://doc.wikimedia.org/ is behind the Varnish/ATS cache and is cached for up to an hour." [integration/docroot] - 10https://gerrit.wikimedia.org/r/891255 (https://phabricator.wikimedia.org/T329034) (owner: 10Kosta Harlan) [16:32:05] 10Phabricator, 10Trust-and-Safety, 10cloud-services-team, 10wikitech.wikimedia.org: Reset 2FA for Developer account 'Rosalie Perside (WMDE)' and Phabricator account @Rosalie_WMDE - https://phabricator.wikimedia.org/T329179 (10bd808) >>! In T329179#8637506, @Rosalie_WMDE wrote: > Thank you. Sorry for the de... [16:43:24] 10Gerrit, 10Release-Engineering-Team (Radar), 10serviceops-collab: discuss whether to keep the gerrit-test instance in devtools - https://phabricator.wikimedia.org/T330248 (10hashar) 05Open→03Resolved a:03hashar @Dzahn and I talked about it during the weekly SRE/Releng meeting a few minutes ago and I c... [16:49:33] 10Continuous-Integration-Config, 10Wikimedia-Site-requests, 10Patch-For-Review, 10User-Urbanecm: CI should ensure that wmf-config/logos.php matches logos/config.yaml - https://phabricator.wikimedia.org/T329231 (10Urbanecm) Last step here should be deploying [r891317](https://gerrit.wikimedia.org/r/c/integr... [17:35:08] (03PS1) 10Jforrester: Zuul: [mediawiki/services/parsoid/deploy] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/891342 (https://phabricator.wikimedia.org/T271222) [17:54:38] so the fun thing when I run docker-compose v2 pointing to Podman, is it pulls an image `docker.io/moby/buildkit:buildx-stable-1` [17:54:56] apparently that somehow build with buildkit (rather than Buildah) [17:55:04] but get stuck when trying to upload the tarballed docker image [17:55:15] maybe one day I will figure it out ;) [17:55:42] workaround is to disable buildkit: `DOCKER_BUILDKIT=0 docker-compose` [17:57:38] 10Beta-Cluster-Infrastructure, 10Browser-Tests, 10User-zeljkofilipin: selenium-daily-betacommons-MediaWiki fails with `Error: internal_api_error_JobQueueError: [...] Exception caught: Could not enqueue jobs` - https://phabricator.wikimedia.org/T322761 (10zeljkofilipin) 05Open→03Invalid Looks like this ha... [18:02:29] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10serviceops-collab, 10serviceops-radar: Disallow direct access to the docker.io registry from gitlab runners (setup a mirror of docker hub) - https://phabricator.wikimedia.org/T329679 (10dancy) [18:17:55] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team: Upgrade Jenkins to latest LTS 2.375.3 - https://phabricator.wikimedia.org/T330045 (10hashar) >>! In T330045#8628642, @hashar wrote: > ... > Which might let us finally get rid of that plugin (`windows-slaves`). With scap3-dev, I... [18:29:30] claime: thanks for updating the calendar, I went ahead and removed the cancelled events (only to avoid jouncebot pinging people) [18:37:34] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/services/parsoid/deploy] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/891342 (https://phabricator.wikimedia.org/T271222) (owner: 10Jforrester) [18:38:39] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/parsoid/deploy] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/891342 (https://phabricator.wikimedia.org/T271222) (owner: 10Jforrester) [19:00:00] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄): Reggie raises frequent "sqlite3.OperationalError: database is locked" under high load - https://phabricator.wikimedia.org/T330239 (10dancy) https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/46 [19:43:37] 10GitLab (Auth & Access), 10Desktop Improvements (Vector 2022), 10Readers-Web-Backlog: Update label for TOC navigation region and heading - https://phabricator.wikimedia.org/T330325 (10bwang) [19:48:17] 10GitLab (Auth & Access), 10Desktop Improvements (Vector 2022), 10Readers-Web-Backlog: Update label for TOC navigation region and heading - https://phabricator.wikimedia.org/T330325 (10bwang) p:05Triage→03Low [20:12:02] 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10Scap: Scap fails on debian bullseye targets - https://phabricator.wikimedia.org/T326668 (10jbond) this seems to be affecting all of the stat, logstash and net{mon,box} listed on https://puppetboard.wikimedia.org/nodes?status=failed. We see the followi... [20:16:17] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: scap backport does not handle Depends-On header correctly - https://phabricator.wikimedia.org/T324275 (10jeena) 05Open→03Resolved a:03jeena [21:44:34] 10GitLab (Integrations), 10GitLab-Test, 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), and 2 others: Experiment with GitLab-Phabricator integration - https://phabricator.wikimedia.org/T265617 (10hashar) [21:44:49] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Comment on Phabricator tasks for new, merged, and abandoned changes on GitLab - https://phabricator.wikimedia.org/T324150 (10hashar) 05Resolved→03Open >>! In T324150#8474354, @hashar wrote: > The Gi... [22:09:28] 10Gerrit, 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10User-brennen: Script closing/archiving of migrated repositories on Gerrit - https://phabricator.wikimedia.org/T330345 (10brennen) [22:10:30] 10Gerrit, 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10User-brennen: Script closing/archiving of migrated repositories on Gerrit - https://phabricator.wikimedia.org/T330345 (10brennen) 05Open→03In progress [22:12:21] 10Continuous-Integration-Infrastructure: Revisit how CI works for released branches - https://phabricator.wikimedia.org/T298043 (10Jdlrobson) Yes this is still relevant as it talks about a generic problem, not specific to MobileFrontend. The same problem appears in Popups for example. [22:17:44] 10Gerrit, 10GitLab (Misc), 10Release-Engineering-Team, 10Wikimedia-GitHub, 10User-brennen: A tool for displaying the canonical location of all Wikimedia repositories and any known mirrors - https://phabricator.wikimedia.org/T330347 (10brennen) [22:55:26] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Comment on Phabricator tasks for new, merged, and abandoned changes on GitLab - https://phabricator.wikimedia.org/T324150 (10brennen) > I am reopening the task since it is still reporting Gitlab actions... [23:47:23] 10Gerrit, 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10User-brennen: Script closing/archiving of migrated repositories on Gerrit - https://phabricator.wikimedia.org/T330345 (10Aklapper) Kind of piggyback riding: Somehow updating the custom `Source Repo` field in P... [23:50:37] 10Gerrit, 10GitLab (Project Migration), 10Release-Engineering-Team (GitLab V: Event Horizon 🌄), 10User-brennen: Script closing/archiving of migrated repositories on Gerrit - https://phabricator.wikimedia.org/T330345 (10brennen) > Kind of piggyback riding: Somehow updating the custom Source Repo field in Ph... [23:59:06] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic: Rename deployment-cache-(text|upload)0x to deployment-cp0x - https://phabricator.wikimedia.org/T280393 (10BCornwall) cp hosts have now been updated to bullseye, FYI