[01:01:42] 10GitLab (Project Migration), 10Release-Engineering-Team, 10Community-Tech: Create new GitLab project group: commtech - https://phabricator.wikimedia.org/T304580 (10Samwilson) [06:21:14] 10Release-Engineering-Team (Radar), 10CommRel-Specialists-Support, 10DBA, 10User-notice: Stop announcing and scheduling primary database switchovers - https://phabricator.wikimedia.org/T303605 (10Marostegui) >>! In T303605#7801530, @Trizek-WMF wrote: > > In order to really inform the communities, we shoul... [08:15:18] good morning :] [08:15:22] jnuche: I am back in action [08:16:08] morning! [09:05:49] 10Release-Engineering-Team (🚂🧪 Trainsperiment Week): Delete wmf branches from Gerrit repositories - https://phabricator.wikimedia.org/T303828 (10jnuche) a:05jnuche→03hashar [09:08:56] 10Gerrit: Mentiras - https://phabricator.wikimedia.org/T304588 (10ESTEBANFABIANGUTICANI) [09:36:33] hi, it looks like beta cluster code syncing is stuck, could anyone help get it unblocked? https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/ [09:37:57] !log killed a beta-scap-sync-world job manually, let’s see if that helps getting beta updates unstuck [09:37:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:38:31] next one should hopefully appear within ten minutes [09:40:12] \o/ [09:43:45] 10Release-Engineering-Team (🚂🧪 Trainsperiment Week), 10Patch-For-Review, 10Release, 10Train Deployments: 🧪🚂 Trainsperiment Week: 1.39.0-wmf.1, 1.39.0-wmf.2, 1.39.0-wmf.3, 1.39.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T300203 (10hashar) T304564 fix has been backported and deployed T3... [09:48:59] thank you Lucas_WMDE ! [09:49:33] hm, I’m not sure if it’s actually helped [09:49:45] https://integration.wikimedia.org/ci/job/beta-scap-sync-world/ shows “waiting for next available executor” again [09:59:56] Project beta-build-scap-deb build #217: 04FAILURE in 14 hr: https://integration.wikimedia.org/ci/job/beta-build-scap-deb/217/ [10:13:32] Project beta-build-scap-deb build #218: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/beta-build-scap-deb/218/ [10:30:58] Project beta-code-update-eqiad build #384998: 15ABORTED in 31 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/384998/ [10:31:08] Project beta-code-update-eqiad build #384999: 15ABORTED in 10 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/384999/ [10:44:59] Project beta-code-update-eqiad build #385004: 15ABORTED in 32 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/385004/ [11:19:30] Yippee, build fixed! [11:19:30] Project beta-build-scap-deb build #219: 09FIXED in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-build-scap-deb/219/ [11:43:25] Lucas_WMDE: i guess deb was why it's stuck [11:43:45] ok [11:44:25] It passed once that finished [11:44:46] 10Release-Engineering-Team (Radar), 10CommRel-Specialists-Support, 10DBA, 10Patch-For-Review, 10User-notice: Stop announcing and scheduling primary database switchovers - https://phabricator.wikimedia.org/T303605 (10Trizek-WMF) >>! In T303605#7802487, @Marostegui wrote: > Not a problem, we can start incl... [11:54:55] 10Release-Engineering-Team (Radar), 10Data-Persistence (Consultation), 10CommRel-Specialists-Support (Jan-Mar-2022): Update Tech News preloads and documentation for the new primary database switchovers workflow - https://phabricator.wikimedia.org/T303937 (10Trizek-WMF) [11:55:09] 10Release-Engineering-Team (Radar), 10Data-Persistence (Consultation), 10CommRel-Specialists-Support (Jan-Mar-2022): Update Tech News preloads and documentation for the new primary database switchovers workflow - https://phabricator.wikimedia.org/T303937 (10Trizek-WMF) 05Open→03Resolved [11:55:18] 10Release-Engineering-Team (Radar), 10CommRel-Specialists-Support, 10DBA, 10Patch-For-Review, 10User-notice: Stop announcing and scheduling primary database switchovers - https://phabricator.wikimedia.org/T303605 (10Trizek-WMF) [13:03:33] 10Release-Engineering-Team (🚂🧪 Trainsperiment Week), 10Patch-For-Review, 10Release, 10Train Deployments: 🧪🚂 Trainsperiment Week: 1.39.0-wmf.1, 1.39.0-wmf.2, 1.39.0-wmf.3, 1.39.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T300203 (10Jdforrester-WMF) [13:43:58] 10Release-Engineering-Team (Radar), 10CommRel-Specialists-Support, 10DBA, 10Patch-For-Review, 10User-notice: Stop announcing and scheduling primary database switchovers - https://phabricator.wikimedia.org/T303605 (10Ladsgroup) I'm a bit hesitant on the current state. It's too close to the status quo. We... [13:56:03] (03PS1) 10Hashar: make-release: hardened branch existence detection [tools/release] - 10https://gerrit.wikimedia.org/r/773504 (https://phabricator.wikimedia.org/T303828) [13:56:32] pff that patch is finally ready :] [14:03:27] 10Release-Engineering-Team (🚂🧪 Trainsperiment Week), 10Patch-For-Review, 10Release, 10Train Deployments: 🧪🚂 Trainsperiment Week: 1.39.0-wmf.1, 1.39.0-wmf.2, 1.39.0-wmf.3, 1.39.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T300203 (10hashar) [14:04:29] 10Release-Engineering-Team (🚂🧪 Trainsperiment Week), 10Patch-For-Review, 10Release, 10Train Deployments: 🧪🚂 Trainsperiment Week: 1.39.0-wmf.1, 1.39.0-wmf.2, 1.39.0-wmf.3, 1.39.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T300203 (10hashar) >>! In T300203#7802754, @hashar wrote: > T30455... [14:04:52] jnuche: I guess I will wait for our US colleagues to show up before resuming the train [14:05:14] one blocker got resolved (we deployed it this morning) the other we dropped it from the blockers after it got investigated [14:05:25] so I think wmf.4 is ready for deployment [14:05:33] hashar: roger [14:05:35] but I am not sure whether we should move it forward [14:05:37] so I will wait :] [14:06:22] 10Release-Engineering-Team (Radar), 10CommRel-Specialists-Support, 10DBA, 10Patch-For-Review, 10User-notice: Stop announcing and scheduling primary database switchovers - https://phabricator.wikimedia.org/T303605 (10Marostegui) That was my assumption too @Ladsgroup that all this was for these "tests". I... [14:11:47] 10Release-Engineering-Team (Doing), 10Phatality: Phatality can't report bugs from very long messages (HTTP 413) - https://phabricator.wikimedia.org/T255671 (10Krinkle) 05Open→03Resolved [14:20:59] (03PS1) 10Krinkle: Remove phatalityId concept, search for task title instead [releng/phatality] - 10https://gerrit.wikimedia.org/r/773515 [14:55:31] (03PS2) 10Jforrester: Zuul: [design/codex] Publish code coverage reports [integration/config] - 10https://gerrit.wikimedia.org/r/772493 (https://phabricator.wikimedia.org/T303899) (owner: 10Catrope) [14:55:37] (03CR) 10Jforrester: [C: 03+2] Zuul: [design/codex] Publish code coverage reports [integration/config] - 10https://gerrit.wikimedia.org/r/772493 (https://phabricator.wikimedia.org/T303899) (owner: 10Catrope) [14:58:40] (03Merged) 10jenkins-bot: Zuul: [design/codex] Publish code coverage reports [integration/config] - 10https://gerrit.wikimedia.org/r/772493 (https://phabricator.wikimedia.org/T303899) (owner: 10Catrope) [14:59:16] (03CR) 10Jforrester: "Nice. Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/771945 (https://phabricator.wikimedia.org/T302905) (owner: 10Jaime Nuche) [15:00:38] !log Zuul: [design/codex] Publish code coverage reports for T303899 [15:00:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:00:40] T303899: Public codex code coverage report - https://phabricator.wikimedia.org/T303899 [15:12:02] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10thcipriani) >>! In T215217#7796938, @Majavah wrote: > Just noting here that the deployment-prep Cloud VPS project currently h... [15:18:31] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.39.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T300204 (10thcipriani) p:05Triage→03Medium a:03hashar [15:24:11] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Radar), 10Research: GitLab Issues experiment with Research - https://phabricator.wikimedia.org/T304614 (10thcipriani) [15:24:24] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Radar), 10Research: GitLab Issues experiment with Research - https://phabricator.wikimedia.org/T304614 (10thcipriani) p:05Triage→03Medium [15:27:01] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-StopForumSpam, 10Observability-Logging, 10Patch-For-Review: ext:StopForumSpam config seemingly broke channel:StopForumSpam within beta-logs.wmcloud.org - https://phabricator.wikimedia.org/T304519 (10colewhite) 05In progress→03Resolved The logs are f... [15:34:42] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Radar), 10Security-Team, 10serviceops, and 2 others: Setup GitLab Runner in trusted environment - https://phabricator.wikimedia.org/T295481 (10Jelto) [15:53:46] (03CR) 10Krinkle: [C: 04-1] "Need quotes to avoid Phab errors for things with colons." [releng/phatality] - 10https://gerrit.wikimedia.org/r/773515 (owner: 10Krinkle) [16:09:37] 10GitLab (Infrastructure), 10serviceops: GitLab minor version upgrade: 14.9.x - https://phabricator.wikimedia.org/T304622 (10Jelto) [16:09:46] 10GitLab (Infrastructure), 10serviceops: GitLab minor version upgrade: 14.9.x - https://phabricator.wikimedia.org/T304622 (10Jelto) p:05Triage→03Medium [16:41:37] Seems to be a CI issue going on: https://phabricator.wikimedia.org/T304625 [16:52:54] Did someone enable SQL strict mode somewhere? [16:53:21] https://gerrit.wikimedia.org/r/q/8a54b5ec38314ca0a6c0669742d168fb46c7da1b [16:53:57] That's a yes :) [16:54:16] thanks zabe [17:09:06] 10Continuous-Integration-Config, 10Release-Engineering-Team (Next), 10MediaWiki-Core-Tests, 10Code-Health, and 8 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Lucas_Werkmeister_WMDE) >>! In T225730#7803420, @gerritbot wrote: > Change 771429 **... [17:11:04] yw [17:13:03] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform: Upgrade event platform related VMs in deployment-prep to Debian bullsye (or buster) - https://phabricator.wikimedia.org/T304433 (10EChetty) p:05Triage→03High a:03Ottomata [17:18:33] 10Phabricator, 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics, 10wmfdata-python: Herald rule to add Product Analytics and Data Engineering tags to Wmfdata-Python tasks - https://phabricator.wikimedia.org/T304572 (10EChetty) [17:52:48] thcipriani: why has WMF deploy calendar been made private [17:53:32] RhinosF1: I was not aware that it was [17:54:05] https://wikitech.wikimedia.org/wiki/Deployments does not look private to me [17:54:05] RhinosF1: I would guess that was fallout from something that the Foundation folks who manage our gsuite did a couple of days ago... [17:54:21] I can still enable it in my google calendar but all the entries just say “busy” (well, „beschäftigt“), and clicking reveals no details [17:54:25] thcipriani: the google one now is [17:54:30] Lucas_WMDE: ye same [17:54:45] bd808: who do I moan too? [17:54:49] * bd808 looks for the thread about this in (barf) slack [17:55:05] hrm. "Make available to public" is still checked [17:55:08] I think people were complaining about this in other context [17:55:26] but the drop down that has "See only free/busy (hide details)" doesn't have...any other option I can select :\ [17:55:45] bd808: thcipriani check #general. Message from Sadok [17:55:49] "See all event details" is grayed out. blerg. [17:56:04] "Hey folks, this is a Google Workspace setting that we have reverted back to how it was. We switched it yesterday to troubleshoot some issues we were encountering with calendars. The revert should be live in about ~20 minutes, worst case you may need to go into your calendar settings and bump the permissions to the amount you wish to share with your personal google account" [17:56:05] In theory... is fixed? [17:56:23] but I do know that in practice the calendar on my phone is still borked [17:58:05] * bd808 pings on the thread with the calendar id and actively not working info [17:58:35] seems like I should now be able to fix it at least [17:59:07] (but I can't so we'll see what IT folks say) [18:01:28] Thanks :) [18:10:07] they've flipped a switch somewhere and told me in 20 minutes I should be able to make this public again [18:17:03] 10Beta-Cluster-Infrastructure, 10Wikidata, 10Wikidata-Termbox, 10wdwb-tech, 10Cloud-VPS (Debian Stretch Deprecation): Move Termbox SSR for Beta Wikidata into deployment-prep project - https://phabricator.wikimedia.org/T304328 (10Lucas_Werkmeister_WMDE) I guess this also relates to {T215217}. [18:33:16] 10Phabricator: Access to "Duplicate" status - https://phabricator.wikimedia.org/T304639 (10rook) [18:39:37] (03PS1) 10Zabe: Archive the DebateTree extension [integration/config] - 10https://gerrit.wikimedia.org/r/773619 (https://phabricator.wikimedia.org/T299320) [18:41:31] thcipriani: any luck [18:42:28] RhinosF1: looks like it should be updated, try refreshing and see if you can see details(?) [18:43:50] thcipriani: hopefully it'll sync soon [19:08:19] 10Release-Engineering-Team (Radar), 10CommRel-Specialists-Support, 10DBA, 10Patch-For-Review, 10User-notice: Stop announcing and scheduling primary database switchovers - https://phabricator.wikimedia.org/T303605 (10Quiddity) Hiyo. After reading all the above, I'm still confused! I think the plan might h... [19:24:03] 10Phabricator: Access to "Duplicate" status - https://phabricator.wikimedia.org/T304639 (10Aklapper) 05Open→03Invalid No such setting exists; as far as i know anyone can {nav icon=anchor,name=Edit Related Tasks... > Close As Duplicate} in the upper right corner. [19:29:50] 10Release-Engineering-Team (Radar), 10CommRel-Specialists-Support, 10DBA, 10Patch-For-Review, 10User-notice: Stop announcing and scheduling primary database switchovers - https://phabricator.wikimedia.org/T303605 (10Ladsgroup) >>! In T303605#7804900, @Quiddity wrote: > Hiyo. After reading all the above,... [19:49:54] Can anyone remember how to clear something from the CI npm cache? For T304652 I think I want to clear any cache of @typescript-eslint/utils but I've forgotten how… [19:49:55] T304652: WikiLambda CI failing on HEAD during npm install (Invalid response body while trying to fetch https://registry.npmjs.org/@typescript-eslint%2futils: ENOENT) - https://phabricator.wikimedia.org/T304652 [19:52:01] James_F: that is on the integration-castor instance [19:52:39] the jobs consoleoutput something like CASTOR_NAMESPACE=.... [19:52:40] Ah, yes. [19:52:46] which is the path were stuff ends up saved on that instance [19:52:51] somewhere under /srv [19:53:09] Yeah, I remembered /srv but thought it was directly on contint1001. :-) [19:53:21] so something like: `/srv/jenkins-workspace/caches/$CASTOR_NAMESPACE` [19:57:28] (03CR) 10Hashar: [C: 03+2] Archive the DebateTree extension [integration/config] - 10https://gerrit.wikimedia.org/r/773619 (https://phabricator.wikimedia.org/T299320) (owner: 10Zabe) [19:59:22] (03Merged) 10jenkins-bot: Archive the DebateTree extension [integration/config] - 10https://gerrit.wikimedia.org/r/773619 (https://phabricator.wikimedia.org/T299320) (owner: 10Zabe) [20:00:10] !log Clearing integration-castor03:/srv/jenkins-workspace/caches/castor-mw-ext-and-skins/master/mwgate-node14-docker/_cacache/content-v2/sha512/22/ for T304652 [20:00:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:00:12] T304652: WikiLambda CI failing on HEAD during npm install (Invalid response body while trying to fetch https://registry.npmjs.org/@typescript-eslint%2futils: ENOENT) - https://phabricator.wikimedia.org/T304652 [20:00:19] !log reloading Zuul for Id844e1723a38eed627af03397cf0ad90c7b09a32 # T299320 [20:00:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:00:21] T299320: Archive the DebateTree extension - https://phabricator.wikimedia.org/T299320 [20:35:14] 10Scap, 10MediaWiki-extensions-WikimediaMaintenance: scap update-interwiki-cache throws MWException: Setting $wgInterwikiCache to a CDB path is no longer supported - https://phabricator.wikimedia.org/T304529 (10Ladsgroup) Suggestion: In order to make things work for the new wikis, run `/usr/local/bin/mwscript... [20:46:16] 10Continuous-Integration-Config, 10Quibble, 10Wikidata, 10wdwb-tech, 10ci-test-error (WMF-deployed Build Failure): jenkis CI wikibase-repo-docker failing with new quibble version - https://phabricator.wikimedia.org/T304147 (10kostajh) [21:02:34] (03PS1) 10Kosta Harlan: dockerfiles: Update Quibble to 1.4.4-s1 [integration/config] - 10https://gerrit.wikimedia.org/r/773635 (https://phabricator.wikimedia.org/T304147) [21:02:36] (03PS1) 10Kosta Harlan: jjb: Use quibble-with-supervisord entrypoint [integration/config] - 10https://gerrit.wikimedia.org/r/773636 (https://phabricator.wikimedia.org/T304147) [21:02:38] (03PS1) 10Kosta Harlan: dockerfiles: Update quibble-buster-php72-bundle to 1.4.4-s2 [integration/config] - 10https://gerrit.wikimedia.org/r/773637 (https://phabricator.wikimedia.org/T304147) [21:02:40] (03PS1) 10Kosta Harlan: jjb: Update mwselenium jobs to use quibble 1.4.4-s2 [integration/config] - 10https://gerrit.wikimedia.org/r/773638 (https://phabricator.wikimedia.org/T304147) [21:07:40] 10Scap, 10MediaWiki-extensions-WikimediaMaintenance: scap update-interwiki-cache throws MWException: Setting $wgInterwikiCache to a CDB path is no longer supported - https://phabricator.wikimedia.org/T304529 (10Urbanecm) >>! In T304529#7805220, @Ladsgroup wrote: > Suggestion: In order to make things work for t... [21:11:23] (03PS3) 10Hashar: deploymentcalendar: raise on Phabricator error [tools/release] - 10https://gerrit.wikimedia.org/r/769776 [21:13:26] (03CR) 10Hashar: "I have slightly rephrased the commit message and added a test." [tools/release] - 10https://gerrit.wikimedia.org/r/769776 (owner: 10Hashar) [21:15:31] 10phabricator maintenance bot: Phabricator-maintenance-bot: create the wiki creation subtasks for wikistats again - https://phabricator.wikimedia.org/T304659 (10Dzahn) [21:16:24] 10phabricator maintenance bot: Phabricator-maintenance-bot: create the wiki creation subtasks for wikistats again - https://phabricator.wikimedia.org/T304659 (10Dzahn) [21:16:37] 10phabricator maintenance bot, 10VPS-project-Wikistats: Phabricator-maintenance-bot: create the wiki creation subtasks for wikistats again - https://phabricator.wikimedia.org/T304659 (10Dzahn) [21:27:13] (03PS2) 10Krinkle: Remove phatalityId concept, search for task title instead [releng/phatality] - 10https://gerrit.wikimedia.org/r/773515 [21:27:19] (03CR) 10Krinkle: "Fixed, and covered by test." [releng/phatality] - 10https://gerrit.wikimedia.org/r/773515 (owner: 10Krinkle) [21:40:43] (03PS1) 10Thcipriani: Calendar: fallback to thcipriani as default deployer [tools/release] - 10https://gerrit.wikimedia.org/r/773645 [21:41:28] (03CR) 10Thcipriani: [C: 03+2] Calendar: fallback to thcipriani as default deployer [tools/release] - 10https://gerrit.wikimedia.org/r/773645 (owner: 10Thcipriani) [21:42:11] (03Merged) 10jenkins-bot: Calendar: fallback to thcipriani as default deployer [tools/release] - 10https://gerrit.wikimedia.org/r/773645 (owner: 10Thcipriani) [22:23:04] (03CR) 10Brennen Bearnes: Remove phatalityId concept, search for task title instead (031 comment) [releng/phatality] - 10https://gerrit.wikimedia.org/r/773515 (owner: 10Krinkle) [22:40:36] (03CR) 10Zabe: [C: 03+1] Start branching Wikistories for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/773211 (https://phabricator.wikimedia.org/T303004) (owner: 10Sbisson) [22:57:27] (03PS3) 10Krinkle: Remove phatalityId concept, search for task title instead [releng/phatality] - 10https://gerrit.wikimedia.org/r/773515 [22:58:16] (03CR) 10Krinkle: Remove phatalityId concept, search for task title instead (031 comment) [releng/phatality] - 10https://gerrit.wikimedia.org/r/773515 (owner: 10Krinkle) [23:12:02] did the mediawiki/extensions submodule collection stop updating? [23:14:05] I don't see anything since March 10 [23:16:12] TimStarling: https://github.com/wikimedia/mediawiki-extensions/commits/master looks like it's roughly up to date... [23:16:37] What's the last commit you have in it? [23:18:06] weird [23:18:49] sorry, I ran git pull again and now I have it -- checked my backscroll and I definitely did run it correctly the first time [23:21:50] Gerrit hiccup? [23:22:28] the packets got tired before they made it all the way from eqiad to Tim's place [23:23:55] I ran "git pull && git submodule update --init" but it seems like it only fetched, I see the fetch output 1fe41693e9..bd2bf53ad0 but not the merge/checkout [23:25:33] and that is just today's range, it hadn't checked out anything since March 10 a6611b0c3ad16c1aeba3b4f247b0fcc63f09480d