[00:02:57] oh! I already had a uni in my path: https://gist.github.com/thcipriani/b64c9dcda1a01a6c47ac3a6807a90af0 [00:08:26] That looks similar to the one that b.rennen had too [00:11:39] knowing me, I stole it from him [04:35:53] 10Project-Admins: Create project tag for User-Mhurd - https://phabricator.wikimedia.org/T334014 (10Aklapper) 05Openβ†’03Resolved a:03Aklapper Requested public project #User-Mhurd has been created: https://phabricator.wikimedia.org/project/view/6483/ (In case you need to edit the project or project workboard... [05:22:12] PROBLEM - Check systemd state on doc1002 is CRITICAL: CRITICAL - degraded: The following units failed: rsync-doc-doc2001.codfw.wmnet.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [06:17:50] RECOVERY - Check systemd state on doc1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [07:50:44] 10Scap, 10Patch-For-Review: Improve behavior around global Scap lock + communicate changes - https://phabricator.wikimedia.org/T330756 (10Urbanecm) Recording conversation from #wikimedia-operations at IRC. The puppet patch was reverted, because scap couldn't deploy anything with the following error message: `... [09:05:26] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T330209 (10hashar) [09:07:16] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [09:15:06] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [09:15:27] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [09:15:34] mutante: thanks a lot for still going ahead and merging the block execution change [09:15:47] (we ended up running into an issue, but that's unrelated) [09:22:39] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120 (10Clement_Goubert) [09:41:40] (03PS3) 10Hashar: Zuul: add AKulbii to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/905664 (owner: 10Samtar) [09:41:56] (03CR) 10Hashar: [C: 03+2] Zuul: add AKulbii to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/905664 (owner: 10Samtar) [09:43:04] (03Merged) 10jenkins-bot: Zuul: add AKulbii to CI allow list [integration/config] - 10https://gerrit.wikimedia.org/r/905664 (owner: 10Samtar) [09:44:03] !log Reloaded Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/905664 [09:44:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:21:18] 10GitLab (Infrastructure), 10serviceops-collab: Troubleshoot partman config for two additional disks on GitLab hosts - https://phabricator.wikimedia.org/T333674 (10MoritzMuehlenhoff) >>! In T333674#8754347, @Jelto wrote: > Do you have any idea what might be missing in the current partman config for setting up... [10:55:04] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (Seen), 10Zuul: Display Zuul status of jobs for a change on Gerrit UI - https://phabricator.wikimedia.org/T214068 (10hashar) When a new message is posted to a change, the Gerrit Web UI will shows a notification at the bottom stat... [11:46:39] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Discovery-Search, 10Pywikibot, 10Pywikibot-tests: cirrussearch-backend-error on beta cluster - https://phabricator.wikimedia.org/T333952 (10Jgiannelos) Also breaks restbase tests using beta. For example: https://github.com/wikimedia/restbase/pull/1319 [13:07:35] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Pywikibot, 10Discovery-Search (Current work), 10Pywikibot-tests: cirrussearch-backend-error on beta cluster - https://phabricator.wikimedia.org/T333952 (10dcausse) Certificates of the elastic machines seem to have expired: ` * Server certificate: * subjec... [13:20:03] Project beta-update-databases-eqiad build #66341: 04FAILURE in 1.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/66341/ [13:22:16] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Pywikibot, 10Discovery-Search (Current work), 10Pywikibot-tests: cirrussearch-backend-error on beta cluster - https://phabricator.wikimedia.org/T333952 (10bking) Last time I tried this, I was advised to generate certificates with [[ https://wikitech.wikime... [14:09:07] Yippee, build fixed! [14:09:07] Project beta-update-databases-eqiad build #66342: 09FIXED in 7 min 48 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/66342/ [14:56:42] (03PS1) 10Lucas Werkmeister (WMDE): Zuul: [mediawiki/extensions/EntitySchema] Add Wikibase dep [integration/config] - 10https://gerrit.wikimedia.org/r/906041 (https://phabricator.wikimedia.org/T333661) [15:04:38] dancy: jnuche, merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/905304, do you want me to merge the beta change for the symlink as well ? [15:04:49] yes please [15:05:59] done [15:06:18] Thanks! [15:06:23] np [15:20:55] Hi, for some reason, UnblockMe doesn't have GitHub and Diffusion mirrors. Can you check why it isn't recreated by replication? [15:23:41] Kizule: if I remember rightly, it was decided that these won't be created by default [15:24:24] claime: Things aren't working as I expected. I'll hit you up with some questions shortly [15:24:58] a'ight [15:27:57] herzog: I thinked that it is handled automatically, when repo is added as a submodule of mediawiki/extensions. [15:28:39] Kizule: I wished it was when I created repos. It's all manual, and very time consuming (specially for Diffusion) [15:29:37] github not so much though, which is the one for which replication is used [15:29:53] differential does not use replication, you need to configure the pull URIs manually [15:30:13] brings me memories :) [15:30:18] dancy: looking at deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud I think it's because you're trying to change from a directory to a symlink and it won't let you do that without force => true (in general puppet doesn't let you remove directories without force) [15:31:39] if you were to manually remove /srv/mediawiki and then rerun puppet it'd probably work, the logic is sound, it's applying it to the current state of the environment that's failing [15:31:49] I'll try that' [15:32:42] That did it. [15:36:53] Do you expect that change to be necessary more than once ? [15:37:02] (switching from directoy to symlink) [15:38:09] Hopefully not. I will want it to be automatic when deployed to deploy1002/deploy2002 though. I created https://gerrit.wikimedia.org/r/c/operations/puppet/+/906051 to include the force flag. [15:48:07] merged [15:53:11] 10Deployments, 10Release-Engineering-Team, 10Patch-For-Review, 10Performance-Team (Radar): MediaWiki deploy servers should not be mediawiki installation targets - https://phabricator.wikimedia.org/T329857 (10dancy) p:05Triageβ†’03Medium Changes have been deployed to beta cluster. [15:54:01] Thanks! [16:33:28] does anyone know how to disable this 'app' within Phab? https://phabricator.wikimedia.org/r/ [16:33:33] GUI or config ? [16:33:46] I tried to find it before [16:35:28] I suspect it's a part of the 'Wikimedia Customizations' application. the best way probably is to remove its code from whatever repository adds it [16:38:53] aha!, thanks taavi, will dig deeper [16:41:33] clones phabricator/depoyment [16:47:21] i think there's a task / patch for this somewhere. [16:47:37] mutante: i can dig into it after i get out of meetings here in a bit. [16:48:15] brennen: yea, task. I was trying to make a patch though [16:48:31] thanks! currently trying to figure out the submodule part :) [16:48:31] i thought there was something [16:49:00] anyway i'll look once i'm no longer on camera. :) [16:49:47] it's in /repos/phabricator/extensions auf gitlab [16:49:52] on gitlab, heh [16:50:05] says Andre [16:56:15] 10Release-Engineering-Team, 10Wikimedia-Phabricator-Extensions, 10serviceops-collab: Disable "Browse Gerrit Projects" on https://phabricator.wikimedia.org/r/ - https://phabricator.wikimedia.org/T228507 (10Aklapper) Seems to be defined in repo https://gitlab.wikimedia.org/repos/phabricator/extensions . `./src... [16:56:24] 10Gerrit: Reviewer-bot option to be added as CC instead of reviewer - https://phabricator.wikimedia.org/T334118 (10Tgr) [17:03:48] dancy: Ah yeah I forgot puppet would back up the dir in that case [17:05:58] (03Abandoned) 10Bernard Wang: Remove docs from Vector CI [integration/config] - 10https://gerrit.wikimedia.org/r/904825 (https://phabricator.wikimedia.org/T321102) (owner: 10Bernard Wang) [17:10:59] (03PS1) 10Tchanders: Remove SimilarEditors extension [integration/config] - 10https://gerrit.wikimedia.org/r/906072 (https://phabricator.wikimedia.org/T331718) [17:29:28] Hiyo. The 1.41.0-wmf.3 Release Notes page has not been automagically updated this week. (https://www.mediawiki.org/wiki/MediaWiki_1.41/wmf.3) [17:29:28] I wonder if a script/something needs to be nudged? (Perhaps the Jenkins script linked via https://www.mediawiki.org/wiki/Template:WMFReleasePage ?) Thanks! [17:31:53] bah, that was probably broken by the same thing that broke auto deploy. I'll check it out [17:31:56] thanks for the note [17:48:00] (03CR) 10Jforrester: [C: 04-1] "At most, you should move this to the Archived section and add the appropriate template, but in practice I'd just move it down to the regul" [integration/config] - 10https://gerrit.wikimedia.org/r/906072 (https://phabricator.wikimedia.org/T331718) (owner: 10Tchanders) [17:49:41] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/SimilarEditors] Demote from WMF production [integration/config] - 10https://gerrit.wikimedia.org/r/906072 (https://phabricator.wikimedia.org/T331718) (owner: 10Tchanders) [17:54:27] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/SimilarEditors] Demote from WMF production [integration/config] - 10https://gerrit.wikimedia.org/r/906072 (https://phabricator.wikimedia.org/T331718) (owner: 10Tchanders) [17:54:44] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/EntitySchema] Add Wikibase dep [integration/config] - 10https://gerrit.wikimedia.org/r/906041 (https://phabricator.wikimedia.org/T333661) (owner: 10Lucas Werkmeister (WMDE)) [17:55:38] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/SimilarEditors] Demote from WMF production [integration/config] - 10https://gerrit.wikimedia.org/r/906072 (https://phabricator.wikimedia.org/T331718) (owner: 10Tchanders) [17:55:53] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/EntitySchema] Add Wikibase dep [integration/config] - 10https://gerrit.wikimedia.org/r/906041 (https://phabricator.wikimedia.org/T333661) (owner: 10Lucas Werkmeister (WMDE)) [18:00:03] 10Phabricator: Add 'developer-experience' Phabricator tag - https://phabricator.wikimedia.org/T334126 (10Volker_E) [18:01:22] 10Project-Admins: Add 'developer-experience' Phabricator tag - https://phabricator.wikimedia.org/T334126 (10JJMC89) [18:05:28] !log Zuul: [mediawiki/extensions/SimilarEditors] Demote from WMF production, for T331718 [18:05:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:05:32] T331718: Undeploy SimilarEditors from Beta - https://phabricator.wikimedia.org/T331718 [18:05:47] !log Zuul: [mediawiki/extensions/EntitySchema] Add Wikibase dep, for T333661 [18:05:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:05:50] T333661: Make Wikibase available in EntitySchema CI - https://phabricator.wikimedia.org/T333661 [18:24:11] 10Project-Admins: Add 'developer-experience' Phabricator tag - https://phabricator.wikimedia.org/T334126 (10taavi) How would this differ from #developer_productivity? [19:04:33] 10MediaWiki-Releasing, 10Documentation, 10MW-1.40-release, 10Patch-For-Review: Finish up RELEASE-NOTES for 1.40.0 - https://phabricator.wikimedia.org/T329082 (10Jdforrester-WMF) a:03Jdforrester-WMF [19:06:26] 10MediaWiki-Releasing, 10Documentation, 10MW-1.40-release: Add REL1_40 to on-wiki documentation as the development snapshot - https://phabricator.wikimedia.org/T329086 (10Jdforrester-WMF) 05Openβ†’03Resolved a:03Jdforrester-WMF https://www.mediawiki.org/w/index.php?diff=5824812&oldid=5788073&title=Module... [19:06:30] 10MediaWiki-Releasing, 10MW-1.40-release, 10Release: Release MW 1.40.0 - https://phabricator.wikimedia.org/T329075 (10Jdforrester-WMF) [19:13:55] dancy: hi, we have trouble reproducing this error locally: https://phabricator.wikimedia.org/T333966 how can we see how it does what it does in beta cluster? [19:14:19] * dancy reads [19:19:25] Run `/usr/local/bin/mwscript mergeMessageFileList.php --wiki=aawiki --force-version master --list-file="/srv/mediawiki-staging/wmf-config/extension-list"` on deployment-deploy03.deployment-prep.eqiad1.wikimedia.cloud [19:21:12] And `/srv/mediawiki-staging/wmf-config/ExtensionMessages-master.php` always has the output generated from the last run of https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-sync-world/ [19:31:23] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Pywikibot, 10Discovery-Search (Current work), 10Pywikibot-tests: cirrussearch-backend-error on beta cluster - https://phabricator.wikimedia.org/T333952 (10bking) 05Openβ†’03Resolved a:03bking Upon further review, it looks like beta cluster changed how... [19:45:45] thanks [21:07:14] 10Beta-Cluster-Infrastructure, 10CirrusSearch, 10Pywikibot, 10Discovery-Search (Current work), 10Pywikibot-tests: cirrussearch-backend-error on beta cluster - https://phabricator.wikimedia.org/T333952 (10LucasWerkmeister) Yeah, now it’s working again for me. Thanks! [21:36:22] 10Project-Admins: Add 'developer-experience' Phabricator tag - https://phabricator.wikimedia.org/T334126 (10Peachey88) [21:37:23] 10Project-Admins: Add 'developer-experience' Phabricator tag - https://phabricator.wikimedia.org/T334126 (10Peachey88) @Volker_E Please fill in the standard template, I have added it into the task description, This will make creation easier for the user that actions the request. [23:53:26] PROBLEM - zuul_merger_service_running on contint2002 is CRITICAL: PROCS CRITICAL: 0 processes with regex args bin/zuul-merger https://www.mediawiki.org/wiki/Continuous_integration/Zuul [23:54:18] PROBLEM - Check systemd state on contint2002 is CRITICAL: CRITICAL - degraded: The following units failed: wmf_auto_restart_zuul-merger.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state