[05:03:12] (03CR) 10Hashar: [C: 03+2] Zuul: [mediawiki/extensions/EditNotify] Add Echo dependency [integration/config] - 10https://gerrit.wikimedia.org/r/905262 (owner: 10Zoranzoki21) [05:04:31] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/EditNotify] Add Echo dependency [integration/config] - 10https://gerrit.wikimedia.org/r/905262 (owner: 10Zoranzoki21) [05:04:33] (03CR) 10Hashar: [C: 03+2] "I have deployed it yesterday?" [integration/config] - 10https://gerrit.wikimedia.org/r/905190 (https://phabricator.wikimedia.org/T332930) (owner: 10Hashar) [05:04:50] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/905262 [05:04:52] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [05:05:41] (03Merged) 10jenkins-bot: jjb: update Quibble jobs to 1.5.4 [integration/config] - 10https://gerrit.wikimedia.org/r/905190 (https://phabricator.wikimedia.org/T332930) (owner: 10Hashar) [05:57:29] (03CR) 10Hashar: [C: 04-1] Remove docs from Vector CI (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/904825 (https://phabricator.wikimedia.org/T321102) (owner: 10Bernard Wang) [07:11:06] (03CR) 10Hashar: [C: 03+2] Extract and deploy upstream plugins [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/904575 (owner: 10Hashar) [07:11:19] (03CR) 10Hashar: [C: 03+2] wm-zuul-status: fix items having no build [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/902718 (https://phabricator.wikimedia.org/T214068) (owner: 10Hashar) [07:12:16] (03Merged) 10jenkins-bot: wm-zuul-status: fix items having no build [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/902718 (https://phabricator.wikimedia.org/T214068) (owner: 10Hashar) [07:12:51] (03CR) 10Hashar: [C: 03+2] Migrate from git fat to git lfs [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/904239 (https://phabricator.wikimedia.org/T333465) (owner: 10Hashar) [07:13:37] (03Merged) 10jenkins-bot: Migrate from git fat to git lfs [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/904239 (https://phabricator.wikimedia.org/T333465) (owner: 10Hashar) [07:13:40] (03Merged) 10jenkins-bot: Extract and deploy upstream plugins [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/904575 (owner: 10Hashar) [07:32:29] 10Gerrit, 10Release-Engineering-Team: Migrate Gerrit deployment from git-fat to git-lfs - https://phabricator.wikimedia.org/T333465 (10hashar) a:03hashar I have successfully deployed with `git-fat` this morning. With the upstream `gerrit.war` plugins bundled in our git repository, we no more have to run `ger... [07:42:45] 10Gerrit: Evaluate Gerrit's zuul-results-summary plugin - https://phabricator.wikimedia.org/T269803 (10hashar) I have done some work to integrate Zuul reports in the Gerrit web UI. My work is based on [[ https://gerrit.wikimedia.org/r/Documentation/pg-plugin-checks-api.html | Gerrit JavaScript Plugin Checks API... [07:43:32] 10GitLab (Infrastructure), 10SRE, 10Traffic, 10serviceops-collab, 10Patch-For-Review: Deprecate and disable port 80 for one-off sites under canonical domains - https://phabricator.wikimedia.org/T238720 (10Jelto) [07:56:07] 10Gerrit: Evaluate Gerrit's zuul-results-summary plugin - https://phabricator.wikimedia.org/T269803 (10hashar) 05Openβ†’03Declined For the upstream plugins, there are three Zuul plugins: 1. `zuul` is deployed on our Gerrit, it processes changes `Depends-On` commit message meta header and shows the cross repos... [07:59:35] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Seen), 10Zuul: Display Zuul status of jobs for a change on Gerrit UI - https://phabricator.wikimedia.org/T214068 (10hashar) With this morning update, I think we will no more encounter `TypeError: status.jobs[0] is undefined` err... [08:01:17] 10GitLab (Infrastructure), 10SRE, 10Traffic, 10serviceops-collab, 10Patch-For-Review: Deprecate and disable port 80 for one-off sites under canonical domains - https://phabricator.wikimedia.org/T238720 (10Jelto) @brennen @hashar there is a [open change](https://gerrit.wikimedia.org/r/c/operations/puppet/... [08:10:09] 10Continuous-Integration-Config, 10affects-translatewiki.net: jenkins.job=mediawiki-i18n-check-docker fails without a real error - https://phabricator.wikimedia.org/T331098 (10Nikerabbit) Recent example: https://integration.wikimedia.org/ci/job/mediawiki-i18n-check-docker/127709/console ` 09:15:02 [mediawiki-... [08:19:45] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T330209 (10hashar) `counterexample > Systemd timer ran the following command: > > /usr/bin/scap stage-train -Dfull_image_build:True... [08:22:55] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T330209 (10hashar) And indeed there is a new branch: ` * [new branch] wmf/branch_cut_pretest -> origin/wmf/branch_cut_prete... [08:26:51] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T330209 (10hashar) https://gerrit.wikimedia.org/r/admin/repos/mediawiki/core,branches shows the latest branches as: | wmf/1.41.0-wmf.3 |... [08:41:08] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T330209 (10hashar) The train script breaking on the branch is from https://gitlab.wikimedia.org/repos/releng/scap/-/blob/master/scap/util... [08:47:36] jnuche: ahmon notification was on the merge request "Rewrite make-release/automatic-branch-cut in Python" https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/27#note_23316 [08:47:37] ;) [08:48:02] so I got warned ahead of time which is great [08:51:10] hashar: ah, wasn't away of that change, thx [08:53:06] 10Gerrit: Error while fetching results for wm-zuul-status: TypeError: can't access property "pipeline", status.jobs[0] is undefined - https://phabricator.wikimedia.org/T333126 (10hashar) 05Openβ†’03Resolved a:03hashar I have fixed one of the occurrence last week and deployed the fix for another one this morn... [08:58:03] 10GitLab (Infrastructure), 10SRE, 10Traffic, 10serviceops-collab, 10Patch-For-Review: Deprecate and disable port 80 for one-off sites under canonical domains - https://phabricator.wikimedia.org/T238720 (10hashar) Gitlab being fairly recent, I don't think it ever got advertised with `http` rather than `ht... [09:02:32] 10Gerrit, 10Machine-Learning-Team, 10serviceops-radar, 10Language-Team (Language-2023-January-March): Create Gerrit repository for /services/machinetranslation and migrate code from Gitlab - https://phabricator.wikimedia.org/T331256 (10hashar) The CI Pipeline got added by 9468bc90c866bb7d2bd21cf48065ae64e7... [09:16:59] 10Project-Admins: Creation of a new project "All-and-every-Wikibooks" - https://phabricator.wikimedia.org/T330600 (10Aklapper) To clarify this, as discussion was about all-and-every-*: I am fine creating a #Wikibooks-fr group, similar to #Wiktionary-fr, if that's wanted. > Do you thnik that there is a sens to cr... [09:23:30] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T330209 (10hashar) [09:31:30] 10Project-Admins: Creation of a new project "All-and-every-Wikibooks" - https://phabricator.wikimedia.org/T330600 (10Lionel_Scheepmans) @Aklapper , I totally understand your arguments and the common sense behind them. I wonder if it would not be more relevant and even more efficient in this case to create a page... [09:38:32] 10Release-Engineering-Team, 10Scap: scap deploy-promote depends on Toolforge service - https://phabricator.wikimedia.org/T333924 (10hashar) [09:42:18] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T330209 (10hashar) [09:53:31] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T330209 (10hashar) [10:11:12] 10Project-Admins: Creation of a new project "All-and-every-Wikibooks" - https://phabricator.wikimedia.org/T330600 (10Aklapper) I want to avoid projects that soon nobody will use anymore, and I want to avoid creating `n+1` places where things get planned instead of `n` places to have even more fragmentation of d... [10:17:59] 10Project-Admins: Creation of a new project "All-and-every-Wikibooks" - https://phabricator.wikimedia.org/T330600 (10Lionel_Scheepmans) Ok. Roger out, and see you next. [11:58:45] 10GitLab (Infrastructure), 10SRE, 10Traffic, 10serviceops-collab, 10Patch-For-Review: Deprecate and disable port 80 for one-off sites under canonical domains - https://phabricator.wikimedia.org/T238720 (10Jelto) >>! In T238720#8753483, @hashar wrote: > Gitlab being fairly recent, I don't think it ever go... [12:06:50] 10Phabricator: Disallow adding Project Tags in the Subscribers field - https://phabricator.wikimedia.org/T303829 (10Aklapper) [12:32:01] 10GitLab (Infrastructure), 10SRE, 10Traffic, 10serviceops-collab, 10Patch-For-Review: Deprecate and disable port 80 for one-off sites under canonical domains - https://phabricator.wikimedia.org/T238720 (10hashar) That does not provide much information :) I say go for it, I don't think anything accesses... [12:45:39] 10GitLab (Infrastructure), 10serviceops-collab: Troubleshoot partman config for two additional disks on GitLab hosts - https://phabricator.wikimedia.org/T333674 (10Jelto) @MoritzMuehlenhoff Hi, I'm adding you here because you may have some more experience with partman/debian installer. We reimaged `gitlab2003... [13:16:59] 10Phabricator: Improve error message about Edit Policy when creating objects when Edit Policy is hidden in form - https://phabricator.wikimedia.org/T333955 (10Aklapper) [13:17:30] Project mediawiki-core-doxygen-docker build #42286: 04FAILURE in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/42286/ [13:30:07] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10WikiLambda, 10Wikifunctions: WikiLambda i18n strings no longer loaded on Beta Cluster - https://phabricator.wikimedia.org/T333958 (10Jdforrester-WMF) [13:31:48] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10WikiLambda, 10Wikifunctions: WikiLambda i18n strings no longer loaded on Beta Cluster - https://phabricator.wikimedia.org/T333958 (10Jdforrester-WMF) E.g. https://wikifunctions.beta.wmflabs.org/wiki/MediaWiki:Wikilambda-persistentobject-details-la... [13:40:36] Lucas_WMDE: you are a hero :) [13:41:09] don’t speak too soon, I haven’t tried deploying it yet ;) [14:12:10] Yippee, build fixed! [14:12:11] Project mediawiki-core-doxygen-docker build #42287: 09FIXED in 7 min 44 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/42287/ [14:26:03] okay, it looks like it might have worked after all, phew ^^ [14:26:07] train potentially unblocked [14:29:50] oh, but we have another train blocker incoming in a second I’m afraid :( [14:29:54] Lydia is filing the task atm [14:30:05] (broken i18n) [14:35:54] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T330209 (10Lucas_Werkmeister_WMDE) [14:36:29] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T330209 (10Lydia_Pintscher) T333966 seems pretty bad. Marking as blocker for now. [14:36:52] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10WikiLambda, 10Wikifunctions: WikiLambda i18n strings no longer loaded on Beta Cluster - https://phabricator.wikimedia.org/T333958 (10Lydia_Pintscher) This might be the same issue as T333966. Marked it as a train blocker. [14:39:01] Lucas_WMDE: the mediawiki-config has a bunch of other use of `$wgHooks` so I guess they will error out as well? [14:39:11] I don’t think so [14:39:22] in early initialization $wgHooks is still assigned directly [14:39:29] (before MediawikiServices is set up, iiuc) [14:39:36] the special thing about this one case was that it happened later [14:39:44] at a point where the HookContainer service should be used instead [14:39:51] and I think it was the only delayed hook [14:39:53] (not 100% sure though) [14:40:08] yeah [14:40:39] well the fix was made around like 2800 and there are usage of $wgHooks before (ex: line 386) ;) [14:40:43] and after as well [14:42:18] AH no [14:42:25] it is cause $wgHook was used in a hook callback... [14:42:30] what a hack [14:42:32] yeah [14:42:41] so yeah that should be fine :] [14:42:46] but it’s a β€œbetter hack”, the comment says so (: [14:42:56] (do we need *both* hacks? hell if I know) [14:43:21] no idea [14:43:28] as long as the deprecated error I am happy with it [14:43:32] feels safer to just not touch it [14:43:33] yeah ^^ [14:44:29] also I thought this was funny out of context :P https://bash.toolforge.org/quip/l665TIcBxE1_1c7s2CVh [14:44:47] ahahahah [14:45:06] for T333966 I am afraid I gotta summon other releng cause I don't remember anything about how the l10n cache is generated [14:45:07] T333966: message keys shown on beta and test Wikidata - https://phabricator.wikimedia.org/T333966 [14:50:18] o/ [14:50:28] Sorry about the train troubles hashar! [14:54:20] hi [14:54:30] they were easy fix :] [14:54:47] the new blocker T333966 is a bit more cumbersome, some l10n translations are missing [14:54:47] T333966: message keys shown on beta and test Wikidata - https://phabricator.wikimedia.org/T333966 [15:01:58] I am going to craft a message [15:02:03] cause I dont know anything about l10n cache [15:03:08] but apparently we are missing 34% of the messages [15:05:40] Interesting. [15:12:46] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T330209 (10hashar) The train is on hold due to T333966, the new version is only deployed on testwikis for now. I have send the blocking... [15:35:51] I am off for kids & dinner. I have updated various places :] [15:36:15] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T330209 (10hashar) I am hanging off for kid & dinner etc. I have let others from releng knows about it, I guess @dduvall can promote grou... [16:11:45] 10GitLab (Project Migration), 10Release-Engineering-Team (Priority Backlog πŸ“₯): Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10dcausse) [16:26:45] (03PS1) 10Samtar: layout.yaml: Add AKulbii [integration/config] - 10https://gerrit.wikimedia.org/r/905664 [16:28:07] (03CR) 10CI reject: [V: 04-1] layout.yaml: Add AKulbii [integration/config] - 10https://gerrit.wikimedia.org/r/905664 (owner: 10Samtar) [16:28:58] (03PS2) 10Samtar: layout.yaml: Add AKulbii [integration/config] - 10https://gerrit.wikimedia.org/r/905664 [16:29:12] mhm. [16:42:09] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Discovery-Search, 10Beta-Cluster-reproducible: Search problems on beta cluster - https://phabricator.wikimedia.org/T333864 (10Urbanecm_WMF) [16:48:14] heh [17:22:06] 10GitLab (Infrastructure), 10SRE, 10Traffic, 10serviceops-collab, 10Patch-For-Review: Deprecate and disable port 80 for one-off sites under canonical domains - https://phabricator.wikimedia.org/T238720 (10BCornwall) [17:25:36] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T330209 (10Ladsgroup) [17:25:57] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Release, 10Train Deployments: 1.41.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T330210 (10Ladsgroup) [17:27:37] did anyone work on doc2001 today. about 6 hours ago or so? just curious because we have new monitoring stuff that told us it went down.. even though the passive host [17:27:53] would just be interesting to know what that came from [17:37:16] 10GitLab (Infrastructure), 10SRE, 10ops-eqiad, 10serviceops-collab: Install additional SSDs on gitlab1003.wikimedia.org (A3) - https://phabricator.wikimedia.org/T333996 (10Jelto) [17:37:25] 10GitLab (Infrastructure), 10SRE, 10ops-eqiad, 10serviceops-collab: Install additional SSDs on gitlab1004.wikimedia.org (B1) - https://phabricator.wikimedia.org/T333997 (10Jelto) [17:37:52] 10GitLab (Infrastructure), 10SRE, 10ops-eqiad, 10serviceops-collab: Install additional SSDs on gitlab1003.wikimedia.org (A3) - https://phabricator.wikimedia.org/T333996 (10Jelto) [17:50:57] 10GitLab (Infrastructure), 10SRE, 10Traffic, 10serviceops-collab, 10Patch-For-Review: Deprecate and disable port 80 for one-off sites under canonical domains - https://phabricator.wikimedia.org/T238720 (10Jelto) I send a short message on wikitech-l, in case something breaks on GitLab so users are aware o... [18:04:14] thcipriani: I learned of https://github.com/arp242/uni today and thought that if you didn't know of it you might be interested. It's a golang cli tool for poking around in the Unicode database including an emoji search function. `uni e pig` returns 🐷,πŸ–,πŸ—, and 🐽. :) [18:06:59] cdanis: Maybe also something you would enjoy ^ [18:07:35] oh nice, ty :) [18:18:14] bd808: neat. :) [18:18:30] i think i had something like this installed under a similar name but it's a perl util [18:19:22] a ha: https://code.p1k3.com/gitea/brennen/bpb-kit/src/branch/main/home/bin/uni [18:46:46] 10Gerrit, 10GitLab (Misc), 10Release-Engineering-Team, 10Wikimedia-GitHub, 10User-brennen: A tool for displaying the canonical location of all Wikimedia repositories and any known mirrors - https://phabricator.wikimedia.org/T330347 (10brennen) To some extent this duplicates {T190891}. [19:37:51] (03PS1) 10Hashar: wm-zuul-status: change pending jobs SUCCESS > INFO [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/905708 (https://phabricator.wikimedia.org/T214068) [19:56:25] (03CR) 10Hashar: [C: 03+2] wm-zuul-status: change pending jobs SUCCESS > INFO [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/905708 (https://phabricator.wikimedia.org/T214068) (owner: 10Hashar) [19:56:57] (03Merged) 10jenkins-bot: wm-zuul-status: change pending jobs SUCCESS > INFO [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/905708 (https://phabricator.wikimedia.org/T214068) (owner: 10Hashar) [20:05:12] 10Project-Admins: Create project tag for User-Mhurd - https://phabricator.wikimedia.org/T334014 (10Mhurd) [20:12:05] 10Project-Admins: Create project tag for User-Mhurd - https://phabricator.wikimedia.org/T334014 (10Mhurd) [20:13:51] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10WikiLambda, 10Wikifunctions: WikiLambda i18n strings no longer loaded on Beta Cluster - https://phabricator.wikimedia.org/T333958 (10Jdforrester-WMF) Yup, fixed now that T333966 was. [20:13:58] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10WikiLambda, 10Wikifunctions: WikiLambda i18n strings no longer loaded on Beta Cluster - https://phabricator.wikimedia.org/T333958 (10Jdforrester-WMF) [20:14:45] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Release, 10Train Deployments: 1.41.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T330210 (10Jdforrester-WMF) [21:50:48] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Discovery-Search, 10Pywikibot, 10Pywikibot-tests: cirrussearch-backend-error on beta cluster - https://phabricator.wikimedia.org/T333952 (10LucasWerkmeister) This is also breaking the daily AC/DC browser tests running against Beta Commons. `lang=json {"er... [21:55:03] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Disallow adding Project Tags in the Subscribers field - https://phabricator.wikimedia.org/T303829 (10brennen) This might be better handled as an option upstream? Putting on my list to evaluate once we've finished {T333885}. [22:09:39] 10Release-Engineering-Team, 10Scap, 10Wikimedia-IEG-grant-review, 10serviceops-collab: scap deploy fails for iegreview - https://phabricator.wikimedia.org/T332623 (10Dzahn) [22:31:58] jnuche: deploying the "scap block execution" change [22:32:10] now that there is no deployment going on [22:32:43] my comments on ticket are not wrong, but I didn't want to leave you blocked like that [22:32:48] eh, on gerrit [22:49:56] scap execution should now be blocked on deploy1002 [23:43:37] bd808: made my day. Thanks :) [23:46:07] (of course now I'm installing a perl module :))