[01:00:26] 10serviceops, 10Technical-blog-posts, 10Datacenter-Switchover: Story idea for Blog: June 2021 DC Switchover - https://phabricator.wikimedia.org/T286080 (10Legoktm) >>! In T286080#7218285, @srodlund wrote: > @Legoktm I decided to move this into a Google doc for editing before moving this over to the blog. Ca... [08:03:19] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team (Next): Perform l10n cache rebuild using initContainers instead of including it in the image - https://phabricator.wikimedia.org/T286952 (10JMeybohm) >>! In T286952#7228919, @dduvall wrote: > It is computationally expensive only during the first deploymen... [09:28:43] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Patch-For-Review: Evaluate Dragonfly for distribution of docker images - https://phabricator.wikimedia.org/T286054 (10JMeybohm) I did ran the ramp up test like I did for plain registry pulls with dragonfly on default settings (more or less) yesterday (see [[... [09:30:28] hello folks, if anybody has time for a review of the knative-serving chart I'd be super grateful - https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/699380 (even a simple "this is horrible -2" or "this is acceptable but you need to redo half of it" :D) [10:07:08] this is horrible -2 [10:19:19] 10serviceops, 10SRE, 10Patch-For-Review: bring 43 new mediawiki appserver in eqiad into production - https://phabricator.wikimedia.org/T279309 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw1421.eqiad.wmnet ` The log can be found in `/var/log/wmf-... [10:24:24] 10serviceops, 10SRE, 10Patch-For-Review: bring 43 new mediawiki appserver in eqiad into production - https://phabricator.wikimedia.org/T279309 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by dzahn on cumin1001.eqiad.wmnet for hosts: ` mw1422.eqiad.wmnet ` The log can be found in `/var/log/wmf-... [10:57:25] 10serviceops, 10SRE, 10Patch-For-Review: bring 43 new mediawiki appserver in eqiad into production - https://phabricator.wikimedia.org/T279309 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1421.eqiad.wmnet'] ` and were **ALL** successful. [11:10:59] 10serviceops, 10SRE, 10Patch-For-Review: bring 43 new mediawiki appserver in eqiad into production - https://phabricator.wikimedia.org/T279309 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['mw1422.eqiad.wmnet'] ` and were **ALL** successful. [12:04:02] hey _joe_ and other serviceops folks, cpt is doing their virtual offsite this week and that completely overruns the timeslot for the cpt service ops syncup, just a fyi [12:05:26] effie: thanks for the honest feedback :D [12:06:14] apergos: can you ask will to cancel the meeting? [12:06:20] or whoever can [12:06:24] checking [12:06:30] tx tx [12:17:52] <_joe_> apergos: urandom can cancel it [12:18:01] ah great [12:18:56] I am told Eric will cancel it. [15:18:35] 10serviceops, 10Technical-blog-posts, 10Datacenter-Switchover: Story idea for Blog: June 2021 DC Switchover - https://phabricator.wikimedia.org/T286080 (10srodlund) Awesome! I will move this over to the blog and will let you know when it is published (likely tomorrow, 23 July, or Monday 26 July. [15:23:04] jayme: o/ should we move this nice dashboard to a more canonical name ? https://grafana-rw.wikimedia.org/d/Ku6V7QYGz/jayme-etcd3 [15:51:11] 10serviceops, 10MW-on-K8s, 10SRE: Create a gateway in kubernetes for the execution of our "lambdas" - https://phabricator.wikimedia.org/T261277 (10Joe) [15:51:28] 10serviceops, 10MW-on-K8s, 10SRE: Evaluate istio as an ingress for production usage - https://phabricator.wikimedia.org/T287007 (10Joe) 05Open→03Resolved Overall istio looks a lot like the other envoy-based ingress I evaluated, being just quite a bit more complicated because istio can do much, much more... [15:51:59] <_joe_> elukey: ^^ basically I'm suggesting we will use istio as an ingress because you're maintaing it [15:52:09] <_joe_> so thanks for doing our work for us [16:00:28] _joe_ you are implicitly trusting my work and its reliability so I'll remind you this when you'll discover horrible things done by me :D [16:00:35] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team (Next): Perform l10n cache rebuild using initContainers instead of including it in the image - https://phabricator.wikimedia.org/T286952 (10dduvall) >>! In T286952#7229317, @JMeybohm wrote: > As we're already actively working on the network I/O side of th... [17:01:55] <_joe_> elukey: you worked on apache httpd, it can't be worse than that [19:09:35] 10serviceops, 10SRE, 10Patch-For-Review: bring 43 new mediawiki appserver in eqiad into production - https://phabricator.wikimedia.org/T279309 (10Dzahn) [19:11:49] 10serviceops, 10MW-on-K8s, 10Release-Engineering-Team (Next): Perform l10n cache rebuild using initContainers instead of including it in the image - https://phabricator.wikimedia.org/T286952 (10dduvall) 05Open→03Declined Closing this out with relevant updates from the discussion during our RelEng/SRE IC... [21:30:19] 10serviceops, 10Infrastructure-Foundations, 10MediaWiki-extensions-Score, 10Packaging: Update Lilypond in Shellbox container to >= 2.22.0, - https://phabricator.wikimedia.org/T287212 (10Legoktm) a:03Legoktm