[06:54:14] 10serviceops, 10Infrastructure-Foundations, 10Release-Engineering-Team, 10Scap: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10Joe) >>! In T303559#7768785, @dancy wrote: > Hi @MoritzMuehlenhoff and @Volans. Can you comment on how this part of the proposal could be... [09:00:41] 10serviceops, 10Observability-Logging, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Kubernetes logs (container stderr,strout) do not show up in Elasticsearch/Kibana - https://phabricator.wikimedia.org/T289766 (10JMeybohm) ` Mar 11 08:45:01 kubernetes2011 rsyslogd: mmkubernetes: server returned un... [09:27:44] 10serviceops, 10Release-Engineering-Team, 10SRE-Access-Requests: Add some users to the docker group on deployment servers - https://phabricator.wikimedia.org/T303450 (10Joe) p:05Triage→03High @thcipriani I guess we need your approval for this. [10:08:42] hello folks, going to reimage kubernetes2011 in a bit and then 2012 [10:27:33] <_joe_> elukey: <3 [10:37:48] :) [10:37:56] me and jayme this morning were watching [10:37:56] https://grafana.wikimedia.org/d/BjBG0MYMz/jayme-kubernetes-operations?orgId=1&var-prometheus=k8s&var-site=All&viewPanel=33&from=now-30d&to=now [10:38:02] that looks great [10:38:47] (all steps down correspond to new hosts with bullseye being added) [10:38:52] (or reimaged) [11:04:05] elukey: --^ that's great. Is the switch from devicemapper to overlayfs2 the reason why listing images is so much quicker now? [11:05:12] btullis: could be also the new docker version, we are jumping from 18.x to 20.x, or a combination of both :) [11:05:40] gotcha. [11:05:59] (we haven't done any test yet but it would be nice to) [11:28:48] 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, 10Service-deployment-requests: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10BTullis) I've updated the helm charts for datahub so that the secrets handling is compatible with our puppet based secret handling method. T... [11:35:41] 2011 and 2012 reimaged and uncordoned [11:42:21] 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, 10Service-deployment-requests: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10BTullis) [11:44:21] 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, 10Service-deployment-requests: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10BTullis) I've updated the diagram to clarify the way that traffic is intended to flow within the deployment - i.e. requests to the GMS do not... [13:34:32] 10serviceops, 10Infrastructure-Foundations, 10Release-Engineering-Team, 10Scap: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10Volans) >>! In T303559#7769250, @Joe wrote: > What do you need specifically? A list of all the servers where scap is installed? That can b... [14:06:59] folks I have some time to kick off a couple of reimages [14:07:03] https://gerrit.wikimedia.org/r/c/operations/puppet/+/769978/ and next [14:07:13] (2013 and 2014 ) [14:07:25] otherwise if you prefer I'll start next week [14:29:32] I'm fine with now [14:29:49] thanksss [14:33:06] 10serviceops, 10Performance-Team (Radar): Migrate WMF production from PHP 7.2 to PHP 7.4 - https://phabricator.wikimedia.org/T271736 (10JMeybohm) [14:33:07] 10serviceops, 10Patch-For-Review: Stop loading wddx PHP extension - https://phabricator.wikimedia.org/T295725 (10JMeybohm) [14:33:32] 10serviceops, 10Patch-For-Review: Stop loading wddx PHP extension - https://phabricator.wikimedia.org/T295725 (10JMeybohm) p:05Low→03Medium a:03JMeybohm [14:52:14] 10serviceops, 10Infrastructure-Foundations, 10Release-Engineering-Team, 10Scap: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10akosiaris) >>! In T303559#7769862, @Volans wrote: > To me the simpler option would be to have Puppet drop a configuration file on the dep... [14:53:31] elukey: I 'll do kubernetes10[18-22] https://phabricator.wikimedia.org/T293728 early next week btw. I think you pinged me for this, sorry for not replying earlier. I might ask for reviews if something feels unclear. [14:56:57] akosiaris: ack sure! I can take care of them too if you don't have time next week (and/or busy with other things) [14:57:36] let's see if I 'll make it next week and if I don't, they are all yours [15:16:31] kubernetes2013 seems good, even if the netbox step to update networking facts failed https://netbox.wikimedia.org/api/extras/job-results/2654017/ [15:16:45] but on netbox data seems good [15:17:31] so I am inclined to uncordon, all looks fine [15:49:25] <_joe_> elukey: btw we can finally merge the k8s spicerack module [15:49:43] <_joe_> and we can use that to operate on the clusters [15:49:47] ah nice! [15:58:26] 10serviceops, 10Infrastructure-Foundations, 10Release-Engineering-Team, 10Scap: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10dancy) [16:00:44] 2013 and 2014 back in service [16:01:08] 2017 is the last one missing, then there are the 4 vms [16:34:42] great! Thank you so much! [16:39:23] np! The vms may need some brainstorm, afaics they have two virtual disks and a dedicated partman recipe [16:39:50] if we reimage in place we could repurpose /dev/vdb as ext4 partition for overlay [16:40:13] (and mount say /var/lib/docker in there) [16:40:40] 10serviceops, 10Infrastructure-Foundations, 10Release-Engineering-Team, 10Scap: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10dancy) >>! In T303559#7770038, @akosiaris wrote: >>>! In T303559#7769862, @Volans wrote: > >> To me the simpler option would be to have P... [16:41:54] 10serviceops, 10Infrastructure-Foundations, 10Release-Engineering-Team, 10Scap: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10dancy) [16:46:05] I haven't looked yet, but that sounds good. As those nodes are dedicated to kask anyways, there is less risk of emptyDir's filling the disk etc. [16:47:30] 10serviceops, 10Release-Engineering-Team, 10SRE, 10SRE-Access-Requests: Add some users to the docker group on deployment servers - https://phabricator.wikimedia.org/T303450 (10thcipriani) >>! In T303450#7769389, @Joe wrote: > @thcipriani I guess we need your approval for this. Approved! Needed now that im... [16:59:29] 10serviceops, 10Infrastructure-Foundations, 10Release-Engineering-Team, 10Scap: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10dancy) >>! In T303559#7769250, @Joe wrote: > What do you need specifically? A list of all the servers where scap is installed? Confirming... [17:00:12] 10serviceops, 10Infrastructure-Foundations, 10Release-Engineering-Team, 10Scap: Use scap to deploy itself to scap targets - https://phabricator.wikimedia.org/T303559 (10dancy) [18:01:47] 10serviceops, 10SRE, 10Traffic, 10envoy, 10Patch-For-Review: Refactor envoy access_log_path to access loggers - https://phabricator.wikimedia.org/T303231 (10RLazarus)