[05:41:28] 10serviceops, 10MW-on-K8s, 10SRE, 10Release-Engineering-Team (Radar): The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10Joe) So after discussion yesterday, it appears we've come to a consensus that given we're now building incremental... [06:36:44] 10serviceops, 10MW-on-K8s, 10SRE, 10Release-Engineering-Team (Radar): The restricted/mediawiki-webserver image should include skins and resources - https://phabricator.wikimedia.org/T285232 (10Joe) Things that get served statically include: * Favicons (like https://en.wikipedia.org/static/favicon/wikipedi... [06:57:54] elukey: feel free to do so. I wanted a second pair of eyes on it before removing my prefix [06:59:07] ack! LGTM as far as I can see it [07:00:28] I am still investigating the latencies/cpu-usage of k8s ml masters after deploying the kubelets (slowly and steadily increasing) and I remembered that we moved away from the drdb disk template for the k8s etcd hosts [07:00:41] so now I am wondering if I should do the same for the k8s ml masters [07:01:00] since drdb may not be the best with the docker partition [07:02:58] (lemme know if it makes sense or not) [07:33:04] <_joe_> why does a ml master have docker? [07:33:37] _joe_ for calico [07:34:12] we had to add it to make routing working for the webhook stuff [07:34:20] <_joe_> oh right [07:34:25] <_joe_> yeah it seems possible [07:34:54] perfect, I'll try to turn one node to disk_template plain to see how it goes [07:54:04] 10serviceops, 10MW-on-K8s, 10Platform Engineering, 10Scap, and 4 others: Define variant Wikimedia production config in compiled, static files - https://phabricator.wikimedia.org/T223602 (10Joe) This would indeed be quite important for mediawiki on kubernetes. If we moved the yaml files to a separate reposi... [08:45:56] 10serviceops, 10SRE, 10Patch-For-Review: bring 43 new mediawiki appserver in eqiad into production - https://phabricator.wikimedia.org/T279309 (10Dzahn) [08:53:23] 10serviceops, 10SRE, 10Patch-For-Review: bring 43 new mediawiki appserver in eqiad into production - https://phabricator.wikimedia.org/T279309 (10Dzahn) [09:48:29] could I get a review of https://gerrit.wikimedia.org/r/c/operations/puppet/+/702117 ? [09:52:56] 10serviceops, 10SRE, 10Patch-For-Review: bring 43 new mediawiki appserver in eqiad into production - https://phabricator.wikimedia.org/T279309 (10Dzahn) [10:03:28] 10serviceops, 10SRE, 10Patch-For-Review: bring 43 new mediawiki appserver in eqiad into production - https://phabricator.wikimedia.org/T279309 (10Dzahn) [10:05:11] 10serviceops, 10SRE, 10Patch-For-Review: bring 43 new mediawiki appserver in eqiad into production - https://phabricator.wikimedia.org/T279309 (10Dzahn) [11:18:01] 10serviceops, 10SRE, 10Patch-For-Review: bring 43 new mediawiki appserver in eqiad into production - https://phabricator.wikimedia.org/T279309 (10Dzahn) [12:14:29] 10serviceops, 10SRE, 10Patch-For-Review: bring 43 new mediawiki appserver in eqiad into production - https://phabricator.wikimedia.org/T279309 (10Jelto) [12:54:55] 10serviceops, 10SRE, 10Patch-For-Review: bring 43 new mediawiki appserver in eqiad into production - https://phabricator.wikimedia.org/T279309 (10Jelto) [13:10:07] hello folks, https://gerrit.wikimedia.org/r/c/operations/puppet/+/707235 is needed to collect calico metrics for ml-serve-ctrl nodes. It is a no-op for all cluster except the ML one, lemme know if you want to see a different approach [13:10:20] (otherwise I'd like to merge to start collecting metrics_ [13:57:33] 10serviceops, 10GitLab, 10Patch-For-Review: GitLab replica in codfw - https://phabricator.wikimedia.org/T285867 (10Jelto) I ran the install script using the `--check` (dry-run) against `gitlab2001`. Looks good, there are two errors due to check mode usage. I would like to roll out the ansible playbook on `g... [14:14:37] (calico felix metrics showing up now for controllers as well!) [14:31:40] I'd also like to merge the knative-servince chart if people agree, with basic config in admin_ng [14:31:43] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/699380 [14:32:10] just to start testing it, I am pretty sure it will require some swearing to make it work :D [14:39:27] <_joe_> oh this is for the CRDs [14:41:16] 10serviceops: Decide on librdkafka deployment model for k8s services - https://phabricator.wikimedia.org/T242810 (10Aklapper) 05Open→03Resolved >>! In T242810#6087104, @Ottomata wrote: > Can we resolve this? No reply for a year. Boldly doing so. Please reopen if not. [16:27:46] 10serviceops, 10Infrastructure-Foundations, 10MediaWiki-extensions-Score, 10Packaging: Update Lilypond in Shellbox container to >= 2.22.0, - https://phabricator.wikimedia.org/T287212 (10Legoktm) I spent a while yesterday trying to package 2.22.1 and eventually gave up after fighting the way guile is embedd... [17:06:02] 10serviceops, 10Infrastructure-Foundations, 10MediaWiki-extensions-Score, 10Packaging, 10Patch-For-Review: Update Lilypond in Shellbox container to >= 2.22.0, - https://phabricator.wikimedia.org/T287212 (10Legoktm) [18:25:02] 10serviceops, 10Technical-blog-posts, 10Datacenter-Switchover: Story idea for Blog: June 2021 DC Switchover - https://phabricator.wikimedia.org/T286080 (10srodlund) This has been posted! https://techblog.wikimedia.org/2021/07/23/june-2021-data-center-switchover/ Let me know if it looks good to you, and I'll... [18:47:51] 10serviceops, 10Technical-blog-posts, 10Datacenter-Switchover: Story idea for Blog: June 2021 DC Switchover - https://phabricator.wikimedia.org/T286080 (10Legoktm) Is it possible to stack the Citoid graphs instead of putting them side-by-side? Everything else looks great, thank you! [18:52:08] 10serviceops, 10Infrastructure-Foundations, 10MediaWiki-extensions-Score, 10Packaging: Update Lilypond in Shellbox container to >= 2.22.0, - https://phabricator.wikimedia.org/T287212 (10Legoktm) 05Open→03Resolved {F34561617} [18:55:30] 10serviceops, 10Technical-blog-posts, 10Datacenter-Switchover: Story idea for Blog: June 2021 DC Switchover - https://phabricator.wikimedia.org/T286080 (10srodlund) These are stacked now! [18:56:34] 10serviceops, 10Infrastructure-Foundations, 10MediaWiki-extensions-Score, 10Packaging: Update Lilypond in Shellbox container to >= 2.22.0, - https://phabricator.wikimedia.org/T287212 (10Legoktm) [19:10:21] 10serviceops, 10Technical-blog-posts, 10Datacenter-Switchover: Story idea for Blog: June 2021 DC Switchover - https://phabricator.wikimedia.org/T286080 (10srodlund) 05Open→03Resolved