[02:27:50] <wikibugs>	 10MediaWiki-Releasing, 10MW-1.41-release, 10Release: Release MediaWiki 1.41.0 - https://phabricator.wikimedia.org/T346919 (10Jdforrester-WMF)
[02:28:13] <wikibugs>	 10MediaWiki-Releasing, 10MW-1.41-release, 10Patch-For-Review: Branch REL1_41 for MediaWiki and all extensions and skins - https://phabricator.wikimedia.org/T346925 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF
[02:41:39] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Vendor, 10MW-1.41-release, 10Patch-For-Review: Prune /vendor for REL1_41 - https://phabricator.wikimedia.org/T346926 (10Jdforrester-WMF) a:03Jdforrester-WMF
[02:52:01] <wikibugs>	 (03PS1) 10Krinkle: zuul: Enable CI for mediawiki/gadgets/CVNSimpleOverlay [integration/config] - 10https://gerrit.wikimedia.org/r/964619
[02:52:13] <wikibugs>	 (03CR) 10Krinkle: [C: 03+2] zuul: Enable CI for mediawiki/gadgets/CVNSimpleOverlay [integration/config] - 10https://gerrit.wikimedia.org/r/964619 (owner: 10Krinkle)
[02:53:51] <wikibugs>	 (03Merged) 10jenkins-bot: zuul: Enable CI for mediawiki/gadgets/CVNSimpleOverlay [integration/config] - 10https://gerrit.wikimedia.org/r/964619 (owner: 10Krinkle)
[07:30:54] <wikibugs>	 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10collaboration-services, 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Jelto)
[07:31:29] <wikibugs>	 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10collaboration-services, 10Patch-For-Review, 10User-brennen: Setup partial backups for GitLab - https://phabricator.wikimedia.org/T316935 (10Jelto) 05Open→03Resolved The 6h/12h schedule for backups and restores is the most reasonable approach. If...
[10:02:10] <wikibugs>	 (03CR) 10Kosta Harlan: [C: 03+2] ci-fullrun: use a virtualenv to setup Quibble [integration/quibble] - 10https://gerrit.wikimedia.org/r/963863 (owner: 10Hashar)
[10:02:38] <hashar>	 andre: I got the reviews data I was looking for in just a few minutes :)
[10:03:06] <hashar>	 using the mediawiki/core wmf/XXXX branch which has as submodules the list of extensions/skins deployed to production
[10:16:43] <andre>	 hashar, heh, yay
[10:21:11] <wikibugs>	 (03Merged) 10jenkins-bot: ci-fullrun: use a virtualenv to setup Quibble [integration/quibble] - 10https://gerrit.wikimedia.org/r/963863 (owner: 10Hashar)
[10:39:09] <wikibugs>	 10Release-Engineering-Team, 10Infrastructure-Foundations, 10Puppet CI, 10SRE: PCC failing with "No space left on device" - https://phabricator.wikimedia.org/T348176 (10hashar) The leftover files under `/home` got cleaned up: ` $ ssh pcc-worker1002.puppet-diffs.eqiad1.wikimedia.cloud df -ih / Filesystem...
[11:27:11] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Yak Shaving 🐃🪒): Have linters/tests results show up as comments in files on gerrit - https://phabricator.wikimedia.org/T209149 (10MoritzMuehlenhoff)
[11:46:17] <wikibugs>	 10GitLab (Infrastructure), 10collaboration-services, 10Patch-For-Review: Move gitlab ssh host keys to private puppet - https://phabricator.wikimedia.org/T333840 (10Jelto)
[11:46:30] <wikibugs>	 10GitLab (Infrastructure), 10collaboration-services, 10Patch-For-Review: gitlab.wikimedia.org ssh host key should appear in wmf-known-host - https://phabricator.wikimedia.org/T337107 (10Jelto) 05Open→03Resolved The public key for `gitlab.wikimedia.org` and the replicas is configured in wmf-known-hosts no...
[12:29:32] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T347081 (10hashar) There is one for which I don't know whether it is a blocker of it is a maintenance script misbehaving. In Vector skin...
[13:06:28] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Vendor, 10MW-1.41-release, 10Patch-For-Review: Prune /vendor for REL1_41 - https://phabricator.wikimedia.org/T346926 (10Jdforrester-WMF) 05Open→03Resolved
[13:06:30] <wikibugs>	 10MediaWiki-Releasing, 10MW-1.41-release, 10Release: Release MediaWiki 1.41.0 - https://phabricator.wikimedia.org/T346919 (10Jdforrester-WMF)
[13:12:06] <wikibugs>	 10MediaWiki-Releasing, 10MW-1.41-release: Add REL1_41 to on-wiki documentation as the development snapshot - https://phabricator.wikimedia.org/T346930 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF https://www.mediawiki.org/w/index.php?diff=6146864&oldid=6127900&title=Module:Version
[13:12:08] <wikibugs>	 10MediaWiki-Releasing, 10MW-1.41-release, 10Release: Release MediaWiki 1.41.0 - https://phabricator.wikimedia.org/T346919 (10Jdforrester-WMF)
[14:22:44] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10Metrics Platform Backlog, 10Data Products (Sprint 01), 10Patch-For-Review: Migrate metrics-platform repo to GitLab - https://phabricator.wikimedia.org/T344733 (10Reedy) After creating https://gerrit.wikimedia.org/r/c/mediawiki/libs/metrics-platform/+/964595... It'...
[14:44:50] <wikibugs>	 10Release-Engineering-Team, 10Patch-For-Review, 10sre-alert-triage: Alert triage: overdue critical alert - https://phabricator.wikimedia.org/T342755 (10Clement_Goubert) 05Open→03Resolved a:03Clement_Goubert
[15:45:25] <wikibugs>	 10Gitlab-Application-Security-Pipeline: Establish a more specific policy/best practices around security include failures - https://phabricator.wikimedia.org/T342469 (10mmartorana) 05In progress→03Resolved
[15:45:28] <wikibugs>	 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: [EPIC] Application Security Pipeline Components for Gitlab - Phase 2 Work - https://phabricator.wikimedia.org/T342177 (10mmartorana)
[15:46:09] <wikibugs>	 10Gitlab-Application-Security-Pipeline: Establish a more specific policy/best practices around security include failures - https://phabricator.wikimedia.org/T342469 (10mmartorana) [[ https://www.mediawiki.org/wiki/Security/Application_Security_Pipeline#Results | Documentation ]] updated.
[15:51:20] <wikibugs>	 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: Provide use case about specific workflow in documentation - https://phabricator.wikimedia.org/T348548 (10mmartorana)
[15:51:49] <wikibugs>	 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: [EPIC] Application Security Pipeline Components for Gitlab - Phase 2 Work - https://phabricator.wikimedia.org/T342177 (10mmartorana)
[15:51:55] <wikibugs>	 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: Provide use case about specific workflow in documentation - https://phabricator.wikimedia.org/T348548 (10mmartorana) 05Open→03Stalled p:05Triage→03Low
[15:52:49] <physikerwelt>	 Hi, I am looking at the deployment calendar https://wikitech.wikimedia.org/wiki/Deployments and wonder, if there is a specific window which would be suitable for a mathoid deployment? 
[16:02:27] <Reedy>	 Maybe one of the services windows.. Depending on who owns it
[16:11:49] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10Metrics Platform Backlog, 10Data Products (Sprint 01), 10Patch-For-Review: Migrate metrics-platform repo to GitLab - https://phabricator.wikimedia.org/T344733 (10phuedx) >>! In T344733#9239029, @Reedy wrote: > Plus CI still works over there...  CI should work on G...
[16:14:46] <physikerwelt>	 Reedy: I only see a Citoid window, which seems to be quite specific to citoid
[16:15:07] <Reedy>	 >Wikidata Query Service weekly deploy 
[16:15:16] <Reedy>	 >Content transform team node services (mobileapps/wikifeeds) 
[16:15:26] <Reedy>	 >Services including Gerrit, Phorge (Phabricator), GitLab 
[16:15:31] <Reedy>	 >Wikifunction Services UTC Afternoon 
[16:15:37] <Reedy>	 Not saying they're applicable, but there's others
[16:15:50] <Reedy>	 It also depends who is going to be doing said deployment too
[16:18:30] <thcipriani>	 The services window is probably ok, but needs coordination with others. But also you can add a window if there's nothing that fits---edits to the wiki page (should) not be clobbered by the bot
[16:28:38] <physikerwelt>	 I still don't understand, who would do the deployment. This is the change to be deployed. https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/964932/1/helmfile.d/services/mathoid/values.yaml if the deployer is Abstract Wikipedia team (North and South America) I don't know if that is a good fit, because there are no people associated with it
[16:38:26] <thcipriani>	 has there been anyone supporting you deploying software? Or has it all been ad hoc?
[16:39:25] <physikerwelt>	 Most recently this was James_F
[16:41:00] <physikerwelt>	 Ok thank you both. I think this discussion can be better done in the respective change to not clutter this chat
[16:43:32] <wikibugs>	 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: [EPIC] Application Security Pipeline Components for Gitlab - Phase 2 Work - https://phabricator.wikimedia.org/T342177 (10sbassett)
[16:43:52] <wikibugs>	 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: Provide use case about specific workflow in documentation - https://phabricator.wikimedia.org/T348548 (10sbassett) 05Stalled→03In progress
[16:44:44] <thcipriani>	 physikerwelt: hrm, you might be the only service that can't self-deploy. Which puts you in the spot of figuring this out ad-hoc. Maybe a task for a standing mathoid window would be a good place to discuss.
[16:46:04] <thcipriani>	 (also would be good to know if other services are hurting for deployers)
[16:55:35] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10Metrics Platform Backlog, 10Data Products (Sprint 01), 10Patch-For-Review: Migrate metrics-platform repo to GitLab - https://phabricator.wikimedia.org/T344733 (10thcipriani) >>! In T344733#9198704, @phuedx wrote: >>>! In T344733#9192434, @Krinkle wrote: >> I've re...
[17:17:58] <James_F>	 I'm happy to deploy the service as needed ad-hoc, but I don't scale; for non-mathoid services which have a less good deployment testing plan, I'd be a bit leery, but mathoid is easy to test if it works or not.
[17:32:28] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10Metrics Platform Backlog, 10Data Products (Sprint 01), 10Patch-For-Review: Migrate metrics-platform repo to GitLab - https://phabricator.wikimedia.org/T344733 (10Reedy) >>! In T344733#9239777, @thcipriani wrote:  > So there are two paths: >  > If this repo benefit...
[18:01:40] <physikerwelt>	 James_F: Thank you. This is s good solution, and I hope the future of mathoid will be limited. We are currently already at 92% https://caniuse.com/mathml and I think it is ok for the rest to use client-side mathjax rendering. For keeping client-side mathjax up to date, we can then use the same mechanism as for any other client side JS library 
[18:29:32] <bvibber>	 hey all - quick q on composer/packagist hacks :D -- we're provisionally now using a local fork of the getid3 library for media metadata extraction to push bug fixes faster than upstream releases, but that's turning out to be harder to integrate with production because we have an extra step in vendor branch setup :D
[18:29:45] <bvibber>	 what's the best way for me to handle this? should i be pushing a package config to our local package repo?
[18:29:56] <bvibber>	 also i forgot to make a version branch i'm a fool ;)
[18:33:09] <Reedy>	 bvibber: As I mentioned on the task, I've already done https://packagist.org/packages/wikimedia/getid3
[18:33:44] <bvibber>	 Reedy: how do i go about updating it?
[18:33:53] <Reedy>	 Push a commit/tag/branch or whatever
[18:33:57] <Reedy>	 the rest is magic
[18:34:12] <bvibber>	 to ?
[18:34:23] <Reedy>	 the github repo, if that's the canonical
[18:34:40] <bvibber>	 of the project being packaged? there's no separate package metadata repo or control list?
[18:35:14] <bvibber>	 "in my day we didn't have these package managers" ;)
[18:35:21] <Reedy>	 composer uses packagist (primarily), packagist uses the repos registered to itself, and pulls any details it needs from the composer.json in the repo
[18:35:31] <bvibber>	 great
[18:35:53] <bvibber>	 so what was necessary to make this happen in the first place?
[18:36:10] <Reedy>	 https://packagist.org/packages/submit
[18:36:14] <Reedy>	 enter the repo url
[18:36:16] <bvibber>	 ah
[18:36:19] <Reedy>	 set up the webhook
[18:36:21] <bvibber>	 so there is a master list of repos?
[18:36:26] <bvibber>	 and someone maintains that?
[18:36:33] <Reedy>	 that's basically what packagist is, yeah
[18:36:36] <bvibber>	 ok
[18:36:52] <bvibber>	 can anyone add a wikimedia package or is there some specific account or permission needed?
[18:37:13] <Reedy>	 Anyone can, in theory
[18:37:26] <bvibber>	 if there's abuse, how do you remove a package from our namespace?
[18:37:28] <Reedy>	 it'll take whatever is in the name in composer.json
[18:37:42] <Reedy>	 "our" very much in quotes
[18:37:47] <Reedy>	 we don't own it, and it's not really a namespace
[18:37:47] <bvibber>	 heh
[18:37:49] <bvibber>	 heh
[18:37:57] <bvibber>	 ah so it's just "hopefully there won't be supply chain attacks"
[18:38:01] <bvibber>	 cool, cool
[18:38:22] <Reedy>	 I'd hope it doesn't let you override the entry unless you're an owner (of which there can be many)
[18:38:29] <bvibber>	 guess that's why we do the weird vendor branch acrhive :D
[18:38:44] <Reedy>	 pretty much, yeah
[18:38:55] <bvibber>	 is there anything special i need to do for that vendor archive or does it just go from all the composer.jsons too?
[18:38:55] <Reedy>	 for WMF production, we have a "good" idea of what we're deploying, and it can go through gerrit/CR
[18:39:14] <bvibber>	 (cause it was set up to pull from git and was able to do so locally in testing)
[18:39:36] <Reedy>	 James_F already made https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/964617
[18:39:42] <Reedy>	 https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/964617/2/composer.json
[18:39:51] <Reedy>	 TLDR being change composer.json, run `composer update --no-dev`
[18:40:13] <Reedy>	 git commit and git review
[18:40:15] <bvibber>	 right but if anything else similarly changes, i have to make a similar change yes?
[18:40:19] <bvibber>	 so i need to know how that works
[18:40:41] <bvibber>	 mediawiki/vendor has its own separate composer.json that's unrelated to the ones packaged with the extensions?
[18:40:45] <Reedy>	 yup
[18:40:50] <Reedy>	 yeah, if you want to deploy another version of anything in vendor, it requires a change like this to be made,  reviewed and merged
[18:40:51] <bvibber>	 jeez
[18:40:53] <Reedy>	 it'll then ride the train
[18:40:53] <bvibber>	 that's bad
[18:41:19] <Reedy>	 bad? have you seen composer-merge-plugin?
[18:41:20] * Reedy grins
[18:41:22] <bvibber>	 extra points of difference between dev and prod is bad
[18:41:43] <bvibber>	 leads to failures in production, or failures in dev, or both
[18:41:51] <bvibber>	 the first is problematic and can cause downtime
[18:42:02] <Reedy>	 in reality, we don't have much of an issue
[18:42:04] <bvibber>	 the latter leads to disillusionment and a slow wind-down of the life of an open-source project
[18:42:18] <bvibber>	 anyway that's enough old man yells at cloud :D
[18:42:23] <bvibber>	 most of the time it works ;)
[18:42:24] <Reedy>	 we pin versions in MW core etc, so the master branch of vendor won't merge unless core will ru with it
[18:42:37] <Reedy>	 same for gated extensions (of which we have some in production that aren't in the CI gate, but that's another issue)
[18:43:05] <bvibber>	 would it be possible to have ci just refuse to push if the versions don't match between vendor and the core/exts
[18:43:06] <bvibber>	 ?
[18:43:25] <Reedy>	 that's basically what it'll do
[18:43:32] <Reedy>	 but not for all extensions
[18:43:36] <bvibber>	 hm
[18:43:41] <bvibber>	 well, i guess, "but also thoes" :D
[18:43:52] <bvibber>	 there really shouldn't be two places to input the same thing that break 
[18:43:52] <Reedy>	 because not all extensions define their dependancies in mediawiki-vendor, because we don't care about 600+ of them that we host
[18:43:54] <bvibber>	 *shrug*
[18:44:16] <bvibber>	 ?
[18:44:17] <Reedy>	 well, there can be more than two places
[18:44:19] <bvibber>	 obviously only ones we use
[18:44:22] <Reedy>	 you start looking at our libraries
[18:44:27] <Reedy>	 where they depend on each other
[18:44:35] <Reedy>	 MW depends on them individually too
[18:44:51] <bvibber>	 right, so you need consistency of versions
[18:45:06] <bvibber>	 if one library said "i must have version 8" and core says "i must have version 25" you're gonna have problems installing version 8
[18:45:27] <Reedy>	 right, and CI won't let you merge that (even if somehow you got composer to install it)
[18:45:31] <bvibber>	 so why not go over all the manifests and make one from all the ones used?
[18:45:38] <bvibber>	 i'd assume the packagist tool can do this
[18:45:40] <bvibber>	 but i may be wrong :D
[18:45:53] <Reedy>	 https://github.com/wikimedia/composer-merge-plugin
[18:45:57] <Reedy>	 No it can't
[18:46:09] <Reedy>	 Which means <s>hacks</s>plugins like ^ exist
[18:46:18] <bvibber>	 hah
[18:46:25] <bvibber>	 ok, is that what's used in mediawiki-docker?
[18:46:31] <bvibber>	 or does it install things some other way?
[18:46:48] <Reedy>	 well, this is more fun for you
[18:46:57] <Reedy>	 you can run composer install/update individually in an extension folder
[18:47:22] <Reedy>	 MW will cope with that, as long as there's no duplicate dependancies (especially on different versions)
[18:47:34] <bvibber>	 ah, and that's what mediawiki-docker does
[18:47:44] <bvibber>	 key point of difference between dev and prod
[18:47:46] <Reedy>	 not 100% sure, but possibly
[18:47:58] <bvibber>	 it puts a separate 'vendor' dir in each ext
[18:48:47] <bvibber>	 one of these days i'm going to have to double-down and Reimagine The Small Dev Install Experience :D
[18:48:55] <bvibber>	 we've accumulated a lot of *weird stuff* over the years :D :D :D :D
[18:49:04] <bvibber>	 and there's always good reasons for the weird stuff 
[18:49:06] <bvibber>	 :D
[18:49:26] <bvibber>	 anyway thanks, i know which bits i need to poke for next time :D
[18:49:27] <Reedy>	 and there's the "fun" of installing skins and extensions using composer too
[18:49:31] <bvibber>	 heh
[18:49:32] <Reedy>	 but let's not get into that mess
[18:49:46] <bvibber>	 heheh
[18:59:16] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T347081 (10Jdlrobson)
[19:10:20] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T347081 (10matmarex)
[19:34:32] <James_F>	 physikerwelt: Deployed. Looks OK to me. Hope this helped!
[22:23:17] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Move 25% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T348122 (10matmarex) The Kubernetes work so far has caused problems with cross-wiki Echo notifications (see T223413, T342201). Please help...