[01:25:11] 10serviceops, 10SRE, 10WMF-Legal, 10Patch-For-Review: Move old transparency report pages to historical URLs and setup redirect - https://phabricator.wikimedia.org/T230638 (10Dzahn) https://gerrit.wikimedia.org/r/c/operations/puppet/+/745637 as part of T218900 means in the unlikely event that you need futu... [06:49:39] 10serviceops, 10Patch-For-Review: Upgrade kafka-main nodes to buster - https://phabricator.wikimedia.org/T296641 (10elukey) 05Open→03Stalled This task is not actionable until the firmware of kafka-main2003 is updated, see task T297422 [06:50:12] hello folks, I'll schedule some downtime for kafka-main2003 with Papaul next week, we need to upgrade its firmare before re-attempting a reimage, PXE boot + d-i seems not working at the moment [06:50:46] if the update works, we'll need to do it to kafka-main[12]00[1-3] basically (the newer nodes shouldn't be affected) [06:50:54] my usual luck :D [09:03:40] <_joe_> elukey: should we upgrade the firmware *and* reimage? [09:03:52] <_joe_> seems ok to me! [09:05:53] _joe_ yeah my plan is to systemctl stop/mask kafka, let Papaul to upgrade the firmware and re-attempt the reimage [09:06:15] if it works on 2003 we'll need to do the same for the other nodes [13:30:45] I'm going to remove Tiller and some helm2 related RBAC objects soon. First in staging and later in codfw and eqiad. See https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/742989 [14:39:59] <_joe_> jelto: +1 -180, the best patches [14:42:58] joe: I'm also looking forward to have 40 pods less running for every cluster [15:02:06] \o/ [15:27:47] 10serviceops, 10SRE, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10Jelto) First cleanup task is finished: [x] remove tiller and tiller service accounts ([742989](https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/742989)) Tiller deploymen... [15:28:01] 10serviceops, 10SRE, 10Kubernetes, 10Patch-For-Review: Migrate to helm v3 - https://phabricator.wikimedia.org/T251305 (10Jelto) [17:28:40] 10serviceops, 10Observability-Logging, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Kubernetes logs (container stderr,strout) do not show up in Elasticsearch/Kibana - https://phabricator.wikimedia.org/T289766 (10JMeybohm) 05Resolved→03Open Unfortunately this does not seem to be enough. During... [19:59:58] 10serviceops, 10Wikimedia-production-error: wtp1025: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10dancy) [20:10:09] 10serviceops, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10dancy) [20:49:38] 10serviceops, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10dancy) [20:51:32] 10serviceops, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10dancy) This task is a blocker for the wmf.13 train (next week). The rate at which memory usage increases on mediawiki... [21:04:20] 10serviceops, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10dancy) See https://grafana.wikimedia.org/goto/TBsGlYhnk Click on "Show deployments". See the usual sawtoothish patter... [21:12:37] 10serviceops, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10RLazarus) 21:10:42 !log sudo cumin -b7 -s10 -p0 'A:mw-eqiad and not P{mw1414.eqiad.wmnet}' restart-php7.2-fpm [21:21:02] 10serviceops, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10Dzahn) {F34877416} [21:48:36] 10serviceops, 10GitLab (Infrastructure): Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 (10Dzahn) I created a new instance called "runner-bullseye" with the idea to put the gitlab_runner puppet class on it and see how it goes and do so on bullseye. But I did not get to actua... [22:13:09] 10serviceops, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10dancy) [22:16:52] 10serviceops, 10Wikimedia-production-error: wtp* hosts: Out of memory (allocated 39845888) (tried to allocate 131072 bytes) in OutputHandler.php - https://phabricator.wikimedia.org/T297517 (10Majavah) p:05Triage→03Unbreak!