[01:10:37] 10serviceops, 10Traffic, 10envoy: Upgrade Envoy to supported version - https://phabricator.wikimedia.org/T300324 (10RLazarus) p:05Triage→03Medium [07:09:23] 10serviceops, 10Patch-For-Review: Upgrade kafka-main nodes to buster - https://phabricator.wikimedia.org/T296641 (10elukey) [07:09:53] 10serviceops, 10Data-Engineering, 10Patch-For-Review: Move kafka clusters to fixed uid/gid - https://phabricator.wikimedia.org/T296982 (10elukey) 05Resolved→03Open Forgot a couple of things to do for a complete cleanup: 1) We should move deployment-prep's clusters as well to the fixed uid/gid. 2) The `p... [08:32:37] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: kube-apiserver need to reach webhooks running inside of the cluster - https://phabricator.wikimedia.org/T290967 (10ops-monitoring-bot) Host rebooted by jayme@cumin1001 with reason: cgroup_enable=memory after docker install [08:40:50] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: kube-apiserver need to reach webhooks running inside of the cluster - https://phabricator.wikimedia.org/T290967 (10ops-monitoring-bot) Host rebooted by jayme@cumin1001 with reason: cgroup_enable=memory after docker install [08:46:57] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.2.1 - https://phabricator.wikimedia.org/T300058 (10JMeybohm) 05Open→03Resolved scap was updated to 4.2.1 everywhere [08:49:04] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: kube-apiserver need to reach webhooks running inside of the cluster - https://phabricator.wikimedia.org/T290967 (10JMeybohm) [09:05:11] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Implement POC for istio ingress - https://phabricator.wikimedia.org/T290966 (10JMeybohm) [09:05:26] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: kube-apiserver need to reach webhooks running inside of the cluster - https://phabricator.wikimedia.org/T290967 (10JMeybohm) 05Open→03Resolved All control planes have been migrated, I've also updated the docs at https://wikitech.wikimed... [13:02:05] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Run helm test after deploy - https://phabricator.wikimedia.org/T276949 (10Jelto) @JMeybohm mentioned to me that it is possible to combine multiple "helm.sh/hook-delete-policy". So the proposed annotations in T276949#7655743 can be enhanced... [13:22:03] 10serviceops, 10SRE, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10Jelto) [13:30:26] 10serviceops, 10SRE, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10Jelto) 05In progress→03Resolved With the removal of deprecated `helm2` test annotations in https://gerrit.wikimedia.org/r/757877 all (known) cleanup steps are finished. There is some... [13:30:27] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Upgrade kubernetes clusters to a security supported (LTS) version - https://phabricator.wikimedia.org/T244335 (10Jelto) [15:40:12] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.2.1 - https://phabricator.wikimedia.org/T300058 (10dancy) [17:27:40] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.2.1 - https://phabricator.wikimedia.org/T300058 (10dancy) Thanks @JMeybohm ! [18:26:05] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.2.2 - https://phabricator.wikimedia.org/T300392 (10dancy) [18:26:36] 10serviceops, 10Release-Engineering-Team, 10Scap: Deploy Scap version 4.2.2 - https://phabricator.wikimedia.org/T300392 (10dancy) [22:06:44] 10serviceops, 10SRE, 10Patch-For-Review: Remove mediawiki::packages::fonts from non thumbor servers - https://phabricator.wikimedia.org/T294378 (10Dzahn) @Joe All the font* and xfont* packages are gone. from mw1, mw2, wtp, parse, labweb, scandium etc For example https://debmonitor.wikimedia.org/packages/fon... [22:08:19] 10serviceops, 10SRE, 10Patch-For-Review: Remove mediawiki::packages::fonts from non thumbor servers - https://phabricator.wikimedia.org/T294378 (10Dzahn) I did not see a change in MW exceptions over the last couple days either.