[01:15:42] 10Phabricator (Upstream), 10Upstream, 10User-IN: When visiting events thousands of years in the future throw exception "Unhandled Exception ("AphrontQueryException"): #1264: Out of range value for column 'utcInitialEpoch' at row 1" - https://phabricator.wikimedia.org/T269996 (10I) The recently added missions... [01:25:59] 10Project-Admins: Create a #Maintenance tag - https://phabricator.wikimedia.org/T328669 (10Aklapper) I did that now but I'm afraid that didn't help? [02:25:29] 10Release-Engineering-Team, 10Security-Team: Add --pause-after-testserver-sync option to deploy_security.py - https://phabricator.wikimedia.org/T328667 (10sbassett) >>! In T328667#8582143, @jnuche wrote: > @taavi also mentioned the behavior to apply a manual security patch should probably be integrated with sc... [05:24:07] 10Continuous-Integration-Config, 10NormalizedException, 10Documentation, 10User-Tgr: NormalizedException not shown on doc.wikimedia.org - https://phabricator.wikimedia.org/T327604 (10Tgr) [05:31:16] 10Gerrit, 10Upstream: Cherry pick to multiple branches - https://phabricator.wikimedia.org/T311703 (10hashar) 05Openβ†’03Declined > Even though it takes one, you can foreach over an input that cherry-picks but doesn't go to it but rather reloads the page at the end. Possibly yes. Also a one click button ch... [06:07:46] 10Gerrit, 10Upstream: Gerrit: All WIP changes are marked as "Merge Conflict" in the dashboard, even when they are not - https://phabricator.wikimedia.org/T328843 (10hashar) Nice finding @Winston_Sung when querying Gerrit for changes with [[ https://gerrit.wikimedia.org/r/q/is:wip+is:open | is:wip is:open ]], a... [07:02:10] (03PS2) 10Hashar: scap: remove plugins/.eslintrc.json before promote [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/884317 (https://phabricator.wikimedia.org/T328134) [07:02:58] 10Gerrit, 10Release-Engineering-Team (Seen), 10Patch-For-Review: Remove plugins/.eslintrc.json - https://phabricator.wikimedia.org/T328134 (10hashar) [07:04:00] (03CR) 10Hashar: [C: 03+2] "Look" [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/884317 (https://phabricator.wikimedia.org/T328134) (owner: 10Hashar) [07:04:31] (03Merged) 10jenkins-bot: scap: remove plugins/.eslintrc.json before promote [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/884317 (https://phabricator.wikimedia.org/T328134) (owner: 10Hashar) [07:13:23] Project beta-code-update-eqiad build #429807: 04FAILURE in 0.86 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/429807/ [07:14:41] Project mediawiki-core-doxygen-docker build #41029: 04FAILURE in 0.75 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/41029/ [07:17:33] 10Gerrit, 10Release-Engineering-Team (Seen), 10Patch-For-Review: Remove plugins/.eslintrc.json - https://phabricator.wikimedia.org/T328134 (10hashar) The scap script (https://gerrit.wikimedia.org/r/c/operations/software/gerrit/+/884317) does not seem to delete the file from the proper directory cause I had t... [07:22:13] 10Gerrit, 10Release-Engineering-Team (Seen), 10Patch-For-Review: Remove plugins/.eslintrc.json - https://phabricator.wikimedia.org/T328134 (10hashar) From [[ https://doc.wikimedia.org/scap/scap3/checks.html | scap checks documentation ]]: > `$SCAP_FINAL_PATH` > > A symbolic link created during the promote s... [07:27:07] Yippee, build fixed! [07:27:08] Project beta-code-update-eqiad build #429808: 09FIXED in 4 min 7 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/429808/ [08:11:40] Yippee, build fixed! [08:11:41] Project mediawiki-core-doxygen-docker build #41030: 09FIXED in 7 min 14 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/41030/ [08:19:47] (03PS1) 10Hashar: scap: use proper path when deleting .eslintrc.json [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/886801 (https://phabricator.wikimedia.org/T328134) [08:20:14] (03CR) 10Hashar: [C: 03+2] scap: use proper path when deleting .eslintrc.json [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/886801 (https://phabricator.wikimedia.org/T328134) (owner: 10Hashar) [08:20:45] (03Merged) 10jenkins-bot: scap: use proper path when deleting .eslintrc.json [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/886801 (https://phabricator.wikimedia.org/T328134) (owner: 10Hashar) [08:21:34] (03CR) 10Hashar: [C: 03+2] remove reference to tools/release [integration/config] - 10https://gerrit.wikimedia.org/r/886433 (https://phabricator.wikimedia.org/T290260) (owner: 10Jeena Huneidi) [08:22:05] 10Gerrit, 10Release-Engineering-Team (Seen), 10Patch-For-Review: Remove plugins/.eslintrc.json - https://phabricator.wikimedia.org/T328134 (10hashar) 05Openβ†’03Resolved a:03hashar [08:22:44] (03Merged) 10jenkins-bot: remove reference to tools/release [integration/config] - 10https://gerrit.wikimedia.org/r/886433 (https://phabricator.wikimedia.org/T290260) (owner: 10Jeena Huneidi) [08:24:13] 10Gerrit: Support OAuth for login onto gerrit.wikimedia.org - https://phabricator.wikimedia.org/T147864 (10SLyngshede-WMF) [09:12:00] (03CR) 10Hashar: [C: 03+2] tox-buster: add bats to the list of packages to install (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/871178 (owner: 10David Caro) [09:12:42] dcaro: I am finally processing your change to add `bats` to the tox images [09:13:27] (03Merged) 10jenkins-bot: tox-buster: add bats to the list of packages to install [integration/config] - 10https://gerrit.wikimedia.org/r/871178 (owner: 10David Caro) [09:18:12] hashar: thanks! [09:24:34] 10Continuous-Integration-Infrastructure: Stop using integration/composer and then archive the repo - https://phabricator.wikimedia.org/T249949 (10hashar) [09:24:43] 10Continuous-Integration-Infrastructure: Stop using integration/composer and then archive the repo - https://phabricator.wikimedia.org/T249949 (10hashar) [09:25:40] 10Continuous-Integration-Infrastructure: Stop using integration/composer and then archive the repo - https://phabricator.wikimedia.org/T249949 (10hashar) 05Openβ†’03Stalled Marking this tracking task stalled. It is pending on Toolforge Docker images (T287900) which are based on Jessie, Stretch, Buster and Bull... [09:29:28] (03PS1) 10Hashar: jjb: switch tox jobs to images having `bats` [integration/config] - 10https://gerrit.wikimedia.org/r/886832 [09:29:42] (03CR) 10Hashar: [C: 03+2] "Successfully published image docker-registry.discovery.wmnet/releng/tox-acme-chief:0.7.0-s1" [integration/config] - 10https://gerrit.wikimedia.org/r/871178 (owner: 10David Caro) [09:30:30] !log Built tox Docker images to include `bats` | https://gerrit.wikimedia.org/r/c/integration/config/+/871178/ [09:30:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:33:11] dcaro: `bats` is now available to the `tox` Jenkins jobs [09:33:17] (03CR) 10Hashar: [C: 03+2] "Images generated by https://gerrit.wikimedia.org/r/c/integration/config/+/871178/" [integration/config] - 10https://gerrit.wikimedia.org/r/886832 (owner: 10Hashar) [09:33:36] hashar: πŸŽ‰ [09:33:40] thanks! [09:35:14] (03Merged) 10jenkins-bot: jjb: switch tox jobs to images having `bats` [integration/config] - 10https://gerrit.wikimedia.org/r/886832 (owner: 10Hashar) [10:07:04] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: Make gitlab-replica-old accessible - https://phabricator.wikimedia.org/T328635 (10Jelto) 05Openβ†’03Resolved Login to https://gitlab-replica-old.wikimedia.org/ works now (for ops/wmf/nda) and without any redirects to the other replica. I'... [11:03:43] 10Gerrit, 10Upstream: Gerrit: All WIP changes are marked as "Merge Conflict" in the dashboard, even when they are not - https://phabricator.wikimedia.org/T328843 (10Winston_Sung) 05Openβ†’03Stalled [14:16:15] 10GitLab (Infrastructure), 10serviceops-collab: Automate GitLab version upgrade process - https://phabricator.wikimedia.org/T323569 (10Jelto) [14:16:18] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱): Automate integration Jenkins deployment and config changes - https://phabricator.wikimedia.org/T319406 (10jnuche) [14:16:21] 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱), 10serviceops-collab, 10Patch-For-Review: CasC configuration for Releasing Jenkins instance - https://phabricator.wikimedia.org/T323909 (10jnuche) 05In progressβ†’03Resolved [14:18:09] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱): Casc - https://phabricator.wikimedia.org/T328920 (10jnuche) [14:19:32] 10Continuous-Integration-Infrastructure, 10Jenkins: CasC configuration for CI Jenkins - https://phabricator.wikimedia.org/T328920 (10jnuche) [14:38:04] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Wikimedia-Portals: wikimedia-portals-build permission error: "dubious ownership in repository" - https://phabricator.wikimedia.org/T328924 (10Jdrewniak) [14:47:42] (03PS1) 10Jdrewniak: Mark working dir as safe for wikimedia-portals-build job [integration/config] - 10https://gerrit.wikimedia.org/r/886901 (https://phabricator.wikimedia.org/T328924) [14:49:02] 10Release-Engineering-Team (Seen), 10Scap: Running "scap pull" on deploy1001 halts for 2 minutes, then reports a php-fpm error - https://phabricator.wikimedia.org/T246959 (10Krinkle) Alternatively, we could aim to swap things around and actually phase out the deploy-host concept of the staging directory (among... [14:49:32] (03PS2) 10Jdrewniak: jjb: Mark working dir as safe for wikimedia-portals-build job [integration/config] - 10https://gerrit.wikimedia.org/r/886901 (https://phabricator.wikimedia.org/T328924) [14:51:16] 10Phabricator, 10DBA, 10Patch-For-Review: Switchover m3 master db1159 -> db1164 - https://phabricator.wikimedia.org/T328404 (10Marostegui) [14:53:27] 10Phabricator, 10DBA, 10Patch-For-Review: Switchover m3 master db1159 -> db1164 - https://phabricator.wikimedia.org/T328404 (10Marostegui) [14:54:09] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Wikimedia-Portals, 10Patch-For-Review: wikimedia-portals-build permission error: "dubious ownership in repository" - https://phabricator.wikimedia.org/T328924 (10Jdrewniak) Looks like this is a known issue and related to T325128 due to a new Git feature. [15:04:18] (03CR) 10Hashar: [C: 03+2] "Job updated!" [integration/config] - 10https://gerrit.wikimedia.org/r/886901 (https://phabricator.wikimedia.org/T328924) (owner: 10Jdrewniak) [15:04:35] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Wikimedia-Portals, 10Patch-For-Review: wikimedia-portals-build permission error: "dubious ownership in repository" - https://phabricator.wikimedia.org/T328924 (10hashar) Indeed, we had the same issue with wikidata query builder service T328441. We will... [15:05:06] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Wikimedia-Portals, 10Patch-For-Review: wikimedia-portals-build permission error: "dubious ownership in repository" - https://phabricator.wikimedia.org/T328924 (10hashar) 05Openβ†’03Resolved a:03Jdrewniak I have updated the Jenkins job with your patc... [15:05:50] (03Merged) 10jenkins-bot: jjb: Mark working dir as safe for wikimedia-portals-build job [integration/config] - 10https://gerrit.wikimedia.org/r/886901 (https://phabricator.wikimedia.org/T328924) (owner: 10Jdrewniak) [15:33:48] 10Phabricator, 10observability: "General / Phabricator" grafana board tracks decommissioned phab1001 rather than active phab1004 - https://phabricator.wikimedia.org/T328826 (10herron) 05Openβ†’03Resolved a:03herron Updated this dashboard to use phab.* in queries and instance labels in legends so this shoul... [15:43:43] 10Continuous-Integration-Infrastructure, 10Jenkins: Spontaneous β€œNo valid crumb was included in the request” Jetty error when loading Jenkins logs - https://phabricator.wikimedia.org/T326970 (10Lucas_Werkmeister_WMDE) Just got another one, when loading https://integration.wikimedia.org/ci/job/quibble-vendor-my... [15:43:59] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱), 10serviceops-collab, 10Patch-For-Review: Automate integration Jenkins deployment and config changes - https://phabricator.wikimedia.org/T319406 (10Jelto) > Use Scap3 to install Jenkins. We are cur... [15:50:28] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team: Cleanup some plugins from the releases Jenkins - https://phabricator.wikimedia.org/T328687 (10hashar) [16:09:15] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱), 10serviceops-collab, 10Patch-For-Review: Automate integration Jenkins deployment and config changes - https://phabricator.wikimedia.org/T319406 (10jnuche) > What is the reason to switch from Debia... [16:10:12] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (GitLab IV: Mise En Place 🍱), 10serviceops-collab, 10Patch-For-Review: Automate integration Jenkins deployment and config changes - https://phabricator.wikimedia.org/T319406 (10jnuche) [16:22:06] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10User-brennen: Comment on Phabricator tasks for new, merged, and abandoned changes on GitLab - https://phabricator.wikimedia.org/T324150 (10dancy) 05Openβ†’03Resolved [16:22:11] 10GitLab (Integrations), 10GitLab-Test, 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), and 2 others: Experiment with GitLab-Phabricator integration - https://phabricator.wikimedia.org/T265617 (10dancy) [16:51:56] (03PS1) 10Pwangai: Zuul: [mediawiki/extensions/VisualEditor] Enable Sonar Codehealth Checks [integration/config] - 10https://gerrit.wikimedia.org/r/886921 (https://phabricator.wikimedia.org/T321837) [17:20:30] 10Phabricator, 10observability: "General / Phabricator" grafana board tracks decommissioned phab1001 rather than active phab1004 - https://phabricator.wikimedia.org/T328826 (10bd808) Thanks @herron. I BOLDly reverted @valerio.bozzolan's edits on wikitech now that the dashboard works again. [17:48:39] (03PS1) 10Jdrewniak: jjb: Followup to 28dc208e [integration/config] - 10https://gerrit.wikimedia.org/r/886927 (https://phabricator.wikimedia.org/T328924) [19:08:21] (03PS1) 10Subramanya Sastry: Fix dewiki styles to reflect changes on dewiki [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/886933 [19:19:08] (03CR) 10Arlolra: [C: 03+2] Fix dewiki styles to reflect changes on dewiki [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/886933 (owner: 10Subramanya Sastry) [19:19:47] (03Merged) 10jenkins-bot: Fix dewiki styles to reflect changes on dewiki [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/886933 (owner: 10Subramanya Sastry) [19:24:59] 10GitLab (CI & Job Runners), 10serviceops-collab: add disk space usage to grafana dashboard for gitlab-runners - https://phabricator.wikimedia.org/T327435 (10Jelto) @Dzahn and I created multiple dashboards for runner disk space usage: Overview over all Trusted Runners: https://grafana.wikimedia.org/d/Chb-gC07... [19:32:43] You can now rebase chains within the UI on gerrit: https://groups.google.com/g/repo-discuss/c/-Klk6YY9hLo [19:41:48] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Release, 10Train Deployments: 1.40.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T325586 (10thcipriani) a:05hasharβ†’03dduvall [19:42:38] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Release, 10Train Deployments: 1.40.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T325587 (10thcipriani) p:05Triageβ†’03Medium [20:09:16] 10GitLab (CI & Job Runners), 10serviceops-collab: add disk space usage to grafana dashboard for gitlab-runners - https://phabricator.wikimedia.org/T327435 (10dancy) Awesome! [20:22:17] 10Phabricator, 10observability: "General / Phabricator" grafana board tracks decommissioned phab1001 rather than active phab1004 - https://phabricator.wikimedia.org/T328826 (10valerio.bozzolan) (Thank you folks for this fix in Grafana and for that edit in Wikitech) [20:45:24] 10Project-Admins: Create a #Maintenance tag - https://phabricator.wikimedia.org/T328669 (10kostajh) >>! In T328669#8588329, @Aklapper wrote: > I did that now but I'm afraid that didn't help? Indeed, it's still not possible to find #worktype-maintenance when typing "maintenance". I suspect it is much more likel... [20:49:27] 10Beta-Cluster-Infrastructure, 10Growth-Team, 10PageTriage, 10Wikimedia-production-error: PHP Notice: Undefined index: afc_state - https://phabricator.wikimedia.org/T323647 (10Aklapper) [22:06:09] 10GitLab (CI & Job Runners), 10serviceops-collab: add disk space usage to grafana dashboard for gitlab-runners - https://phabricator.wikimedia.org/T327435 (10Dzahn) follow-up task created at T328972 [22:15:46] 10GitLab (CI & Job Runners), 10serviceops-collab: add disk space usage to grafana dashboard for gitlab-runners - https://phabricator.wikimedia.org/T327435 (10Dzahn) follow-up task was declined almost immediately and we are now being blocked by that. T328972#8591290 [23:10:09] 10Project-Admins: Create a #Maintenance tag - https://phabricator.wikimedia.org/T328669 (10Aklapper) I renamed to `#Maintenance-Worktype` and `#NewFunctionality-Worktype`. Seems like that helped