[09:40:04] <wikibugs>	 10serviceops, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff)
[10:25:16] <wikibugs>	 10serviceops, 10Maps: Repool maps primaries in Kartotherian - https://phabricator.wikimedia.org/T355892 (10hnowlan) 05Open→03Resolved
[11:14:22] <wikibugs>	 10serviceops, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619 (10MoritzMuehlenhoff)
[12:57:27] <wikibugs>	 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert)
[12:57:53] <wikibugs>	 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, 10Release-Engineering-Team (Seen): Move 40% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T355532 (10Clement_Goubert) 05Open→03Resolved
[12:58:46] <wikibugs>	 10serviceops, 10MW-on-K8s, 10SRE, 10Traffic, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert)
[13:08:25] <wikibugs>	 10serviceops, 10iPoid-Service: Empty traffic and error panels for ipoid grafana - https://phabricator.wikimedia.org/T356861 (10kostajh)
[14:59:43] <wikibugs>	 10serviceops, 10Prod-Kubernetes, 10observability, 10Kubernetes: Increase visibility of kubernetes network status - https://phabricator.wikimedia.org/T356877 (10kamila)
[15:01:17] <wikibugs>	 10serviceops, 10Prod-Kubernetes, 10observability, 10Kubernetes: Increase visibility of kubernetes network status - https://phabricator.wikimedia.org/T356877 (10kamila)
[15:04:40] <wikibugs>	 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw2377.codfw.wmnet with OS bullseye
[15:05:03] <wikibugs>	 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw2378.codfw.wmnet with OS bullseye
[15:05:40] <wikibugs>	 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw2406.codfw.wmnet with OS bullseye
[15:05:43] <wikibugs>	 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw2301.codfw.wmnet with OS bullseye
[15:05:45] <wikibugs>	 10serviceops, 10MW-on-K8s, 10Patch-For-Review: Move servers from the appserver/api cluster to kubernetes - https://phabricator.wikimedia.org/T351074 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by hnowlan@cumin2002 for host mw2310.codfw.wmnet with OS bullseye
[15:09:19] <wikibugs>	 10serviceops, 10Prod-Kubernetes, 10observability, 10Kubernetes: Increase visibility of kubernetes network status - https://phabricator.wikimedia.org/T356877 (10kamila)
[15:10:16] <wikibugs>	 10serviceops, 10Prod-Kubernetes, 10observability, 10Kubernetes: Increase visibility of kubernetes network status - https://phabricator.wikimedia.org/T356877 (10lmata) please let us know if we can assist :-)
[15:23:56] <wikibugs>	 10serviceops, 10iPoid-Service (iPoid 1.0): Determine cause of HTTP 503 errors for ~8% of MediaWiki requests to ipoid service - https://phabricator.wikimedia.org/T356766 (10jijiki) I can see in the appserver logs   [2024-02-07T14:37:02.754Z] "GET /feed/v1/ip/[REDUCTED] HTTP/1.1" 503 UC 0 95 0 - "-" "MediaWiki/1...
[18:18:45] <wikibugs>	 10serviceops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: httpbb needs to be setup on cumin1002 and removed from cumin1001 - https://phabricator.wikimedia.org/T356054 (10Scott_French) 05Open→03Resolved Timers are up and happy on cumin1002 and no longer running on cumin1001.
[20:04:27] <wikibugs>	 10serviceops, 10SRE: confd setup left without configuration doesn't stop confd - https://phabricator.wikimedia.org/T356296 (10Dzahn) Seems to me this has to do with the `profile::firewall` migration from iptables to nftables.  What these hosts have in common is `profile::firewall::provider: nftables` in hierad...
[20:04:45] <wikibugs>	 10serviceops, 10SRE: confd setup left without configuration doesn't stop confd - https://phabricator.wikimedia.org/T356296 (10Dzahn) cc: @Muehlenhoff
[20:17:52] <wikibugs>	 10serviceops, 10CX-cxserver, 10Citoid, 10Content-Transform-Team-WIP, and 10 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118 (10Sbailey)
[20:25:42] <wikibugs>	 10serviceops, 10CX-cxserver, 10Citoid, 10Content-Transform-Team-WIP, and 10 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118 (10Sbailey) Chromium render (Proton) upgraded from Node 12 to Node 18  [content transform] update to 2024-02-05-181957-producti...