[09:39:26] mwdebug100[12] are out of space, I'm guessing from an additional mw version in /srv/mediawiki [09:42:56] continuing in -operations, mw2272 also affected [09:43:38] 10serviceops, 10MediaWiki-DjVu, 10Shellbox, 10Structured-Data-Backlog, and 4 others: RuntimeException: firejail is enabled, but cannot be found - https://phabricator.wikimedia.org/T352515 (10hnowlan) 05Open→03Resolved [09:51:47] if anyone is around with knowledge on how to remove old mw versions please shout in -operations [10:04:51] 10serviceops, 10SRE: Too many mw versions caused out of disk space on ~30 mw hosts - https://phabricator.wikimedia.org/T355117 (10fgiunchedi) [10:22:40] that's the tracking task for the issue ^ I suspect an old partman recipe was used during the last reimage [12:39:36] 10serviceops, 10Prod-Kubernetes, 10Data-Platform-SRE (2023/24 Q3 Milestone 1), 10Kubernetes, 10Patch-For-Review: Improve how we address outside k8s infrastructure from within charts (e.g. network policies) - https://phabricator.wikimedia.org/T331894 (10brouberol) I was thinking about the chart migration... [14:07:12] 10serviceops, 10SRE: Too many mw versions caused out of disk space on ~30 mw hosts - https://phabricator.wikimedia.org/T355117 (10jnuche) The problem was caused by older MW versions being left over on the drive. For instance: ` mwdeploy@mw2272:/srv/mediawiki$ ls composer.json dblists-index.php errorpages l... [15:20:10] 10serviceops, 10Data-Platform-SRE, 10Prod-Kubernetes, 10Wikidata, and 2 others: Write and adapt Runbooks and cookbooks related to the WDQS Streaming Updater and kubernetes - https://phabricator.wikimedia.org/T293063 (10Gehel) [15:48:02] 10serviceops, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10ABran-WMF) [16:56:40] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw1460.eqiad.wmnet with OS bullseye [17:02:52] _joe_ are you the right person to ask about kafka-main config changes during e-lukey's time off? We were wondering if we could enable log compaction for some topics. Re: https://phabricator.wikimedia.org/T354794 [17:03:30] <_joe_> inflatador: probably, I can try to take a look in the coming days [17:03:38] brouberol has enabled these changes on kafka-jumbo already with no impact FWiW [17:03:41] <_joe_> I'm a bit clogged with backlog due to the abuse response [17:04:09] _joe_ ACK, no rush just wanted to make sure you were the right person. I'll CC you [17:32:32] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by hnowlan@cumin2002 for host mw1460.eqiad.wmnet with OS bullseye completed: - mw1460 (**PASS**) - Downt... [17:48:19] 10serviceops, 10MW-on-K8s, 10SRE, 10Patch-For-Review: Reclaim jobrunner hardware for k8s - https://phabricator.wikimedia.org/T354791 (10hnowlan) [18:50:12] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw1360.eqiad.wmnet with OS bullseye [18:50:48] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw1361.eqiad.wmnet with OS bullseye [18:51:25] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw1362.eqiad.wmnet with OS bullseye [18:52:06] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw1363.eqiad.wmnet with OS bullseye [19:06:53] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw1374.eqiad.wmnet with OS bullseye [19:07:21] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw1375.eqiad.wmnet with OS bullseye [19:07:59] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw1376.eqiad.wmnet with OS bullseye [19:27:45] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw1362.eqiad.wmnet with OS bullseye completed: - mw1362 (**WARN... [19:29:45] 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw1361.eqiad.wmnet with OS bullseye completed: - mw1361 (**PASS... [19:31:09] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw1363.eqiad.wmnet with OS bullseye completed: - mw1363 (**PASS**) - Downtimed on I... [19:31:12] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw2291.codfw.wmnet with OS bullseye [19:31:55] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw2292.codfw.wmnet with OS bullseye [19:32:36] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw2293.codfw.wmnet with OS bullseye [19:34:11] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw1360.eqiad.wmnet with OS bullseye completed: - mw1360 (**PASS**) - Downtimed on I... [19:42:16] 10serviceops, 10CirrusSearch, 10Discovery-Search, 10Data-Platform-SRE (2023/24 Q3 Milestone 1): Requesting permission to enable kafka log compaction for page_rerender on kafka-main - https://phabricator.wikimedia.org/T354794 (10bking) Per IRC conversation with @Joe , he is the right person to approve this... [19:45:34] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw1374.eqiad.wmnet with OS bullseye completed: - mw1374 (**PASS**) - Downtimed on I... [19:46:23] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw2294.codfw.wmnet with OS bullseye [19:47:29] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw1376.eqiad.wmnet with OS bullseye completed: - mw1376 (**PASS**) - Downtimed on I... [19:50:16] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw1375.eqiad.wmnet with OS bullseye completed: - mw1375 (**PASS**) - Downtimed on I... [19:52:02] 10serviceops, 10Content-Transform-Team-WIP, 10Page Content Service, 10RESTBase Sunsetting, 10Patch-For-Review: Update mobileapps k8s deployment chart for Cassandra credentials - https://phabricator.wikimedia.org/T350507 (10Eevans) > - [] Create cassandra user for mobileapps, add credentials in puppet T... [19:52:11] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw2295.codfw.wmnet with OS bullseye [19:52:15] 10serviceops, 10Content-Transform-Team-WIP, 10Page Content Service, 10RESTBase Sunsetting, 10Patch-For-Review: Update mobileapps k8s deployment chart for Cassandra credentials - https://phabricator.wikimedia.org/T350507 (10Eevans) [19:56:39] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw2296.codfw.wmnet with OS bullseye [20:03:36] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host mw2297.codfw.wmnet with OS bullseye [20:12:17] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw2291.codfw.wmnet with OS bullseye completed: - mw2291 (**PASS**) - Downtimed on I... [20:13:38] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw2293.codfw.wmnet with OS bullseye completed: - mw2293 (**PASS**) - Downtimed on I... [20:16:23] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw2292.codfw.wmnet with OS bullseye completed: - mw2292 (**PASS**) - Downtimed on I... [20:25:45] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw2294.codfw.wmnet with OS bullseye completed: - mw2294 (**PASS**) - Downtimed on I... [20:31:04] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw2295.codfw.wmnet with OS bullseye completed: - mw2295 (**PASS**) - Downtimed on I... [20:37:36] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw2296.codfw.wmnet with OS bullseye completed: - mw2296 (**PASS**) - Downtimed on I... [20:43:11] 10serviceops, 10MW-on-K8s: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host mw2297.codfw.wmnet with OS bullseye completed: - mw2297 (**PASS**) - Downtimed on I...