[00:08:27] 10serviceops, 10Release Pipeline, 10Services, 10Patch-For-Review: Provide a node 12 production image (based on bullseye?) - https://phabricator.wikimedia.org/T284346 (10bd808) In my work on {T288542} I have found that there is indeed a rapidly growing number of libraries that have dropped node v10 support... [00:39:25] 10serviceops, 10SRE, 10Services, 10Toolhub, and 2 others: New Service Request Toolhub - https://phabricator.wikimedia.org/T280881 (10bd808) [01:56:06] 10serviceops, 10Performance-Team, 10SRE: Evaluate using igbinary for MW php-apcu at WMF (apc.serializer) - https://phabricator.wikimedia.org/T225074 (10Krinkle) [01:56:31] 10serviceops, 10Performance-Team, 10SRE: Evaluate using igbinary for MW php-apcu at WMF - https://phabricator.wikimedia.org/T225074 (10Krinkle) [01:58:58] 10serviceops, 10Performance-Team, 10SRE: Evaluate using igbinary for MW php-apcu at WMF - https://phabricator.wikimedia.org/T225074 (10Krinkle) Looking around on "the Internet", sources say igbinary is by default slower than php by default, but when setting `igbinary.compact_strings=Off` it should be faster.... [02:38:13] legoktm: pretty cool to see that break down of connections to shellbox/score [06:59:22] 10serviceops, 10MW-on-K8s, 10SRE: Create a gateway in kubernetes for the execution of our "lambdas" - https://phabricator.wikimedia.org/T261277 (10Legoktm) >>! In T261277#7245477, @Legoktm wrote: >> ...given there could be several dozens of such very small services > > I [[https://logstash.wikimedia.org/got... [11:07:05] 10serviceops, 10observability, 10GitLab (Initialization), 10Patch-For-Review: Define monitoring for gitlab - https://phabricator.wikimedia.org/T275170 (10Jelto) The puma workers get killed either if they hit memory limit (which is `puma['per_worker_max_memory_mb'] = 1024` ~1gb by default) or automatically... [13:32:42] hey folks, just got the first ORES score on ml-serve-prod \o/ [13:33:08] still very WIP, a ton of things to do, but the whole thing finally shows some progress [13:33:16] thanks a lot for all the support :) [13:38:33] <_joe_> :)) [14:12:26] 10serviceops, 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10BTullis) Excuse my jumping in on this ticket, but I have a query. Have you considered using the built-in mechanism for uploading the ba... [15:00:37] 10serviceops, 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10jcrespo) > Have you considered using the built-in mechanism for uploading the backup tar file to object storage, immediately after it h... [17:05:47] 10serviceops, 10MW-on-K8s, 10SRE, 10Release-Engineering-Team (Radar): The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10dancy) This doesn't work: `curl -v -H 'X-Wikimedia-Debug: backend=k8s-experimental' https://en.wikipedia.org/favi... [17:08:52] 10serviceops, 10MW-on-K8s, 10Platform Engineering, 10Scap, and 4 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Jdforrester-WMF) a:05Jdforrester-WMF→03None [17:55:56] 10serviceops, 10MW-on-K8s, 10SRE, 10Release-Engineering-Team (Radar): The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10Krinkle) `/favicon.ico` is (or should be) written to `w/favicon.php`. [17:56:57] 10serviceops, 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10BTullis) Thanks @jcrespo - That's a really considered and informative reply. [19:29:11] 10serviceops, 10MW-on-K8s, 10SRE, 10Release-Engineering-Team (Radar): The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10dancy) I think the issue is that the php-fpm container processing the request is trying to make an outgoing HTTP r... [22:21:37] 10serviceops, 10SRE: mcrouter crashing on mwmaint2002 - https://phabricator.wikimedia.org/T288787 (10RLazarus)