[00:06:15] 10serviceops, 10Datacenter-Switchover: Split switchdc.mediawiki step 08 into a 09 - https://phabricator.wikimedia.org/T290677 (10Legoktm) [00:06:17] 10serviceops, 10MW-on-K8s, 10Datacenter-Switchover: Update switchdc cookbooks for mwdebug service - https://phabricator.wikimedia.org/T290676 (10Legoktm) [06:05:11] effie: o/ can I ack the mc1026.mgmt icinga alarm as WIP - DECOM ? [06:05:26] I did it yesterday didn't I ? [06:05:57] oh no, I added expiration 10/9 instead of 10/10 [06:06:05] I will do so, thank you [06:06:08] <3 [06:06:19] thank you! [06:43:39] 10serviceops, 10Observability-Logging, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Kubernetes logs (container stderr,strout) do not show up in Elasticsearch/Kibana - https://phabricator.wikimedia.org/T289766 (10JMeybohm) The patch needs to be ported to the buster package as well, T277739 /cc @el... [08:11:33] effie: is it ok to re-enable puppet on deploy1002 or still wip? [08:23:16] pleased dont:) [08:23:43] I haven't, this is why I asked :) [08:24:09] I will be on my laptop soon, I will run it for you [08:24:32] deal? [09:16:55] effie: np! I don't really need it, feel free to keep it disabled for more :) [09:17:05] jelto: o/ if you are around, I'd have a question for you [09:17:45] I noticed from the diff in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/719128/9 that tiller is added, and from helmfile_namespace.yaml it seems (at least, afaics) that there is no way to skip it for the moment [09:18:28] is there a way to bypass it or do we need some more code to helmfile_namespace? Or maybe something else [09:41:52] it would be nice to have that as a per cluster/environment setting rather than a per namespace one [09:56:51] I assume we need some code change for that, I can take a closer look after lunch. [09:59:41] jelto: yes yes no hurry, we can do even next week, it was just to confirm [14:43:36] Hi folks, dropping https://gerrit.wikimedia.org/r/c/operations/puppet/+/720048 in here to get consensus about how to keep going with the main vs ml cluster split. In the above patch the idea is to keep the hiera settings separate (will require a change in puppet private of course) but it seems to be cleaner in the long term [14:43:51] IIUC we could even split the admin toke [14:43:53] *token [15:37:44] 10serviceops, 10Prod-Kubernetes, 10Toolhub, 10Kubernetes: Maintenance environment needed for running one-off commands - https://phabricator.wikimedia.org/T290357 (10akosiaris) Tangentially related: T290360 I understand the need for this, having experience from Heroku in the past, the one off dynos functio... [15:59:05] have a good weekend folks :) [16:54:29] 10serviceops, 10SRE: Migrate node-based services in production to node12 - https://phabricator.wikimedia.org/T290750 (10Jdforrester-WMF) [17:00:13] 10serviceops, 10SRE: Migrate node-based services in production to node12 - https://phabricator.wikimedia.org/T290750 (10Jdforrester-WMF) [17:01:03] 10serviceops, 10SRE, 10Patch-For-Review, 10Platform Team Initiatives (Containerise Services): Migrate node-based services in production to node10 - https://phabricator.wikimedia.org/T210704 (10Jdforrester-WMF) [17:01:20] 10serviceops, 10SRE, 10Patch-For-Review, 10Platform Team Initiatives (Containerise Services): Migrate node-based services in production to node10 - https://phabricator.wikimedia.org/T210704 (10Jdforrester-WMF) [17:01:49] 10serviceops, 10SRE, 10Patch-For-Review, 10Platform Team Initiatives (Containerise Services): Migrate node-based services in production to node10 - https://phabricator.wikimedia.org/T210704 (10Jdforrester-WMF) [17:02:37] 10serviceops, 10SRE, 10Patch-For-Review, 10Platform Team Initiatives (Containerise Services): Migrate node-based services in production to node10 - https://phabricator.wikimedia.org/T210704 (10Jdforrester-WMF) Has Thumbor been upgraded, or is this waiting on {T216815}? [21:07:17] 10serviceops: Release a 1.16 tag of docker-registry.wikimedia.org/golang - https://phabricator.wikimedia.org/T283425 (10Addshore) 05Open→03Invalid