[00:16:17] 10Release-Engineering-Team (Priority Backlog ๐Ÿ“ฅ), 10Release, 10Train Deployments: 1.40.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T325585 (10demon) 05Openโ†’03Resolved [00:16:45] 10Release-Engineering-Team (Priority Backlog ๐Ÿ“ฅ), 10Release, 10Train Deployments: 1.40.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T325585 (10demon) 05Resolvedโ†’03Open [02:20:02] Project beta-update-databases-eqiad build #64144: 04FAILURE in 1 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/64144/ [02:44:09] ^ fixing patch is https://gerrit.wikimedia.org/r/c/mediawiki/core/+/870739 [03:20:02] Project beta-update-databases-eqiad build #64145: 04STILL FAILING in 0.95 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/64145/ [04:29:55] Yippee, build fixed! [04:29:56] Project beta-update-databases-eqiad build #64146: 09FIXED in 9 min 54 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/64146/ [05:42:29] 10Project-Admins, 10User-Slst2020: Create Toolhunt project tag for Outreachy internship project - https://phabricator.wikimedia.org/T324317 (10Slst2020) [05:45:38] 10Project-Admins, 10User-Slst2020: Create Toolhunt project tag for Outreachy internship project - https://phabricator.wikimedia.org/T324317 (10Slst2020) >>! In T324317#8492325, @Aklapper wrote: > https://github.com/wikimedia/toolhunt has "Issues" active, so there are two places where people could file (duplica... [07:49:12] (03CR) 10Hashar: [C: 03+2] zuul/parameter_functions.py: Make Phonos depend on TimedMediaHandler [integration/config] - 10https://gerrit.wikimedia.org/r/874439 (https://phabricator.wikimedia.org/T322368) (owner: 10MusikAnimal) [07:51:26] (03Merged) 10jenkins-bot: zuul/parameter_functions.py: Make Phonos depend on TimedMediaHandler [integration/config] - 10https://gerrit.wikimedia.org/r/874439 (https://phabricator.wikimedia.org/T322368) (owner: 10MusikAnimal) [07:52:05] (03CR) 10Hashar: [C: 03+2] zuul: Add Thanks as dependency / phan dependency for GrowthExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/869193 (owner: 10Kosta Harlan) [07:52:48] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/874439/ Make Phonos depend on TimedMediaHandler # T322368 [07:52:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:52:51] T322368: Some audio formats are not playable in Phonos on iOS/MacOS/Safari - https://phabricator.wikimedia.org/T322368 [07:53:54] (03Merged) 10jenkins-bot: zuul: Add Thanks as dependency / phan dependency for GrowthExperiments [integration/config] - 10https://gerrit.wikimedia.org/r/869193 (owner: 10Kosta Harlan) [07:57:31] (03CR) 10Hashar: [C: 03+2] Run quibble composer tests for BlueSpiceEchoConnector as tests exist [integration/config] - 10https://gerrit.wikimedia.org/r/870998 (owner: 10Dreamy Jazz) [07:57:38] (03CR) 10Hashar: [C: 03+2] Run quibble-composer tests for BlueSpiceRSSFeeder [integration/config] - 10https://gerrit.wikimedia.org/r/871169 (owner: 10Dreamy Jazz) [07:59:49] (03Merged) 10jenkins-bot: Run quibble composer tests for BlueSpiceEchoConnector as tests exist [integration/config] - 10https://gerrit.wikimedia.org/r/870998 (owner: 10Dreamy Jazz) [07:59:53] (03Merged) 10jenkins-bot: Run quibble-composer tests for BlueSpiceRSSFeeder [integration/config] - 10https://gerrit.wikimedia.org/r/871169 (owner: 10Dreamy Jazz) [08:12:40] FYI: https://pastebin.com/7SzDcYr3 [09:30:35] 10Continuous-Integration-Config, 10Release-Engineering-Team: Speed up integration-config-shellcheck-docker job - https://phabricator.wikimedia.org/T321536 (10hashar) a:03hashar [09:30:39] (03PS1) 10Hashar: Speed up integration-config-shellcheck-docker [integration/config] - 10https://gerrit.wikimedia.org/r/874780 (https://phabricator.wikimedia.org/T321536) [09:31:24] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Patch-For-Review: Speed up integration-config-shellcheck-docker job - https://phabricator.wikimedia.org/T321536 (10hashar) By parallelizing the processing I get it down from 1 minutes 17 seconds to 21 seconds. [09:33:47] (03CR) 10Hashar: "The utils/shellcheck.sh goes from ~ 80 seconds down to 20 seconds." [integration/config] - 10https://gerrit.wikimedia.org/r/874780 (https://phabricator.wikimedia.org/T321536) (owner: 10Hashar) [09:35:55] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Patch-For-Review: Speed up integration-config-shellcheck-docker job - https://phabricator.wikimedia.org/T321536 (10hashar) p:05Triageโ†’03Medium [10:34:07] Project beta-code-update-eqiad build #424928: 04FAILURE in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/424928/ [10:44:53] Yippee, build fixed! [10:44:54] Project beta-code-update-eqiad build #424929: 09FIXED in 1 min 52 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/424929/ [10:55:48] ^^ failed cause Gerrit server was being restarted [11:23:26] 10Beta-Cluster-Infrastructure, 10Patch-For-Review: Replace deployment-prometheus02 - https://phabricator.wikimedia.org/T324782 (10TheresNoTime) [11:53:55] (03Abandoned) 10Clรฉment Goubert: build-mv-image: Copy GeoIP data on build from base [tools/release] - 10https://gerrit.wikimedia.org/r/868391 (https://phabricator.wikimedia.org/T288375) (owner: 10Clรฉment Goubert) [12:26:27] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10SRE, 10Patch-For-Review: git: detected dubious ownership in repository at '/srv/mediawiki-staging' - https://phabricator.wikimedia.org/T325128 (10hashar) a:05hasharโ†’03None I have cherry-picked https://gerrit.wikimedia.org/r/868002 to fix the... [13:40:21] 10Phabricator: Phabricator phd.service refuses to start on machine boot - https://phabricator.wikimedia.org/T326146 (10hashar) [13:41:54] 10Phabricator, 10serviceops-collab: Phabricator phd.service refuses to start on machine boot - https://phabricator.wikimedia.org/T326146 (10hashar) [14:20:37] !log integration: restarting all instances for Linux kernel upgrade [14:20:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:21:40] 10Release-Engineering-Team, 10SRE, 10serviceops-collab: wmf_auto_restart_{jenkins,rsync} failing on releases2002 - https://phabricator.wikimedia.org/T267795 (10LSobanski) [14:35:39] Hi! Are untrusted runners on gitlab a thing yet? Can anyone use them? (if so, how?) I ask for WMCS and for toolforge/cloud users too :) [14:43:13] 10GitLab, 10Release-Engineering-Team, 10serviceops-collab, 10Patch-For-Review: Align the GitLab runner tags - https://phabricator.wikimedia.org/T325069 (10Jelto) [14:53:33] dcaro: hi, jelto would probably know about gitlab untrusted runners. We also have the #wikimedia-gitlab channel ;) [14:53:39] (I don't have the answer) [14:56:40] true, I'm in that channel too xd [15:14:45] 10Release-Engineering-Team, 10SRE, 10serviceops-collab, 10Patch-For-Review: wmf_auto_restart_{jenkins,rsync} failing on releases2002 - https://phabricator.wikimedia.org/T267795 (10MoritzMuehlenhoff) For jenkins this was already fixed in 2e88687c45b980bfb31b3fdc25e11377d1a49f48, for rsyncd I created https:/... [16:56:43] 10Gerrit: Unable to Log into Gerrit UI or Pull/Push from Terminal - https://phabricator.wikimedia.org/T326159 (10JMando) [17:35:54] 10Gerrit: Unable to Log into Gerrit UI or Pull/Push from Terminal - https://phabricator.wikimedia.org/T326159 (10hashar) a:03hashar The Gerrit account `jmando` has been deactived on Tue Dec 6 19:24:23 2022 +0000 It is similar or might be the same as T306297. We have blocked a user `JM` (https://wikitech.wikim... [17:38:21] !log gerrit: reactivated account for `jmando` using `gerrit set-account 9431 --active` | T326159 [17:38:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:38:23] T326159: Unable to Log into Gerrit UI or Pull/Push from Terminal - https://phabricator.wikimedia.org/T326159 [17:39:14] 10Gerrit: Unable to Log into Gerrit UI or Pull/Push from Terminal - https://phabricator.wikimedia.org/T326159 (10hashar) 05Openโ†’03Resolved Should be good now. If it still does not work, please reopen the task. The root cause is T306297 [18:27:40] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [18:36:25] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [19:12:10] !log Disable publishing of rPMSM, ref T143162, T326112 [19:12:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:12:13] T326112: Firefox do not starts when running WebPageReplay - https://phabricator.wikimedia.org/T326112 [19:12:14] T143162: Reduce task notification noise/frequency of changes to associated open patchsets - https://phabricator.wikimedia.org/T143162 [19:14:55] !log restarting zuul [19:14:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:18:31] (03PS1) 10Majavah: zuul: Move Puppet tests from test-prop to test [integration/config] - 10https://gerrit.wikimedia.org/r/874926 [19:19:42] I don't think you meant T143162, Krinkle [19:19:42] T143162: Reduce task notification noise/frequency of changes to associated open patchsets - https://phabricator.wikimedia.org/T143162 [19:20:01] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [19:20:21] Platonides: I think I did :) [19:20:32] It's the task for making that the default so I don't have to keep doing it manually [19:21:20] then it may be T326112 the wrong one? [19:21:20] T326112: Firefox do not starts when running WebPageReplay - https://phabricator.wikimedia.org/T326112 [19:21:27] 10Release-Engineering-Team (Priority Backlog ๐Ÿ“ฅ), 10Patch-For-Review, 10Release, 10Train Deployments: 1.40.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T325580 (10Jdlrobson) ##### Risky Patch! ๐Ÿš‚๐Ÿ”ฅ * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/867732 * **Summary**: **... [19:21:29] T143162 and T326112 look unrelated to me [19:22:40] (Queue (Jenkins jobs + Zuul functions) alert) firing: (2) Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [19:34:58] Is jenkins/zuul failing right now or should I just re-submit my patches? [19:42:40] (Queue (Jenkins jobs + Zuul functions) alert) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [19:44:21] nevermind, I resubmitted and things seem ok [19:55:49] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [20:19:40] (Queue (Jenkins jobs + Zuul functions) alert) firing: Queue (Jenkins jobs + Zuul functions) alert - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [20:46:35] 10GitLab (Infrastructure), 10Release-Engineering-Team (Yak Shaving ๐Ÿƒ๐Ÿช’), 10serviceops-collab, 10Upstream: Self-reported GitLab SSH host key fingerprints donโ€™t appear to match actual host key fingerprints - https://phabricator.wikimedia.org/T296944 (10Dzahn) >>! In T296944#8486844, @Jelto wrote: > So we shou... [20:49:33] 10GitLab (Infrastructure), 10Release-Engineering-Team (Yak Shaving ๐Ÿƒ๐Ÿช’), 10serviceops-collab, 10Upstream: Self-reported GitLab SSH host key fingerprints donโ€™t appear to match actual host key fingerprints - https://phabricator.wikimedia.org/T296944 (10Dzahn) 05Openโ†’03Resolved a:03Dzahn [21:07:41] 10Release-Engineering-Team, 10Scap: `scap backport` should rebase patches to the same repository on top of each other - https://phabricator.wikimedia.org/T326175 (10taavi) [21:08:04] 10Release-Engineering-Team, 10Scap: `scap backport` should not +2 already merged patches - https://phabricator.wikimedia.org/T326176 (10taavi) [21:21:49] 10Phabricator, 10serviceops-collab: Phabricator phd.service refuses to start on machine boot - https://phabricator.wikimedia.org/T326146 (10Dzahn) The `systemd::service phd` already requires the class `phabricator::phd` where the directory is created which fixed this issue in the past. Sure there were not any... [21:27:03] 10Phabricator, 10serviceops-collab: Phabricator phd.service refuses to start on machine boot - https://phabricator.wikimedia.org/T326146 (10Dzahn) So if we use the reboot cookbook and that includes running puppet and running puppet fixes it and the whole process is only done after the cookbook finished.. then... [21:29:49] 10Phabricator, 10serviceops-collab: Phabricator phd.service refuses to start on machine boot - https://phabricator.wikimedia.org/T326146 (10taavi) > Sure there were not any manual starts of the service involved and this was all puppet? systemd starts `phd.service` when the server boots up before puppet has tim... [21:31:56] 10Phabricator, 10serviceops-collab: Phabricator phd.service refuses to start on machine boot - https://phabricator.wikimedia.org/T326146 (10Dzahn) >>! In T326146#8496518, @taavi wrote: >> Sure there were not any manual starts of the service involved and this was all puppet? > systemd starts `phd.service` when... [22:01:12] 10Release-Engineering-Team, 10SRE, 10serviceops-collab, 10Patch-For-Review: wmf_auto_restart_{jenkins,rsync} failing on releases2002 - https://phabricator.wikimedia.org/T267795 (10Dzahn) Thanks all for surfacing this older ticket and providing the fix. Deployed. on releases2002: ` Notice: /Stage[main]/... [22:04:43] 10Release-Engineering-Team, 10SRE, 10serviceops-collab, 10Patch-For-Review: wmf_auto_restart_{jenkins,rsync} failing on releases2002 - https://phabricator.wikimedia.org/T267795 (10Dzahn) 05Openโ†’03Resolved a:03Dzahn ` [releases2002:~] $ systemctl list-unit-files | grep wmf_auto | grep -E 'jenkins|rsyn... [22:08:29] 10Diffusion, 10Phabricator, 10serviceops-collab, 10Patch-For-Review: Redirect https://phabricator.wikimedia.org/r/ to https://gerrit.wikimedia.org/g/ - https://phabricator.wikimedia.org/T324311 (10Dzahn) also see T228507 [22:08:39] 10Release-Engineering-Team, 10Wikimedia-Phabricator-Extensions, 10serviceops-collab: Disable "Browse Gerrit Projects" on https://phabricator.wikimedia.org/r/ - https://phabricator.wikimedia.org/T228507 (10Dzahn) also see T324311 [22:09:11] 10Release-Engineering-Team, 10Wikimedia-Phabricator-Extensions, 10serviceops-collab: Disable "Browse Gerrit Projects" on https://phabricator.wikimedia.org/r/ - https://phabricator.wikimedia.org/T228507 (10Dzahn) [22:09:13] 10Diffusion, 10Phabricator, 10serviceops-collab, 10Patch-For-Review: Redirect https://phabricator.wikimedia.org/r/ to https://gerrit.wikimedia.org/g/ - https://phabricator.wikimedia.org/T324311 (10Dzahn) [22:16:15] 10Continuous-Integration-Infrastructure, 10PHP 8.2 support: Create PHP 8.2 CI images and jobs for early testing - https://phabricator.wikimedia.org/T314093 (10Reedy) [22:16:54] 10Continuous-Integration-Config, 10PHP 8.2 support: CI composer-package-php82-docker needs newer composer/symfony components - https://phabricator.wikimedia.org/T325302 (10Reedy) p:05Triageโ†’03High