[03:30:43] FIRING: [16x] NodeTextfileStale: Stale textfile for dns1004:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [04:38:49] 06Traffic, 10Phabricator, 06Release-Engineering-Team (Radar): Phabricator videos fail in Firefox ("Range" request gets 503 from Varnish) - https://phabricator.wikimedia.org/T397661#12301457 (10Aklapper) p:05Triage→03Low [07:30:43] FIRING: [16x] NodeTextfileStale: Stale textfile for dns1004:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [08:59:06] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12301906 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=bb00a99c-5cd5-4943-9771-6bfb08321711) set by cmooney@cumin1004 for 2:00:00 o... [09:01:24] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12301920 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=bf370135-d73c-4a25-91fa-c29757024f16) set by cmooney@cumin1004 for 2:00:00 o... [09:49:57] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12302079 (10cmooney) [09:50:19] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12302083 (10cmooney) 05Open→03Resolved a:03cmooney Closing this one out, all devices have been upgraded to 23.4R2 now. [09:58:40] 06Traffic, 06Commons, 10MediaWiki-File-management, 06SRE, and 2 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12302139 (10OttoG) Today, it took about an hour for the original SVG content of this file to be updated t... [11:30:43] FIRING: [16x] NodeTextfileStale: Stale textfile for dns1004:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [12:55:53] 06Traffic, 06SRE: Migrated busted container image to Trixie - https://phabricator.wikimedia.org/T437442 (10MoritzMuehlenhoff) 03NEW [13:20:28] FIRING: [16x] NodeTextfileStale: Stale textfile for dns1004:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [13:25:28] FIRING: [16x] NodeTextfileStale: Stale textfile for dns1004:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [13:30:28] FIRING: [16x] NodeTextfileStale: Stale textfile for dns1004:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [13:31:19] ^ those are clearing btw, down to 4 remaining [13:35:28] FIRING: [16x] NodeTextfileStale: Stale textfile for dns1004:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [13:38:54] 06Traffic, 06MediaWiki-API-Platform-Team, 06ServiceOps, 10ServiceOps-SharedInfra: map the /api/ prefix to /w/rest.php - https://phabricator.wikimedia.org/T364400#12303145 (10Clement_Goubert) [13:39:01] 06Traffic, 06MediaWiki-API-Platform-Team, 06ServiceOps, 10ServiceOps-SharedInfra: map the /api/ prefix to /w/rest.php - https://phabricator.wikimedia.org/T364400#12303147 (10Clement_Goubert) [13:45:28] RESOLVED: [6x] NodeTextfileStale: Stale textfile for dns3003:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [16:53:35] 06Traffic, 06DC-Ops, 10ops-drmrs: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12304366 (10BCornwall) 05Open→03Resolved Thanks for hanging in there, everyone, and thanks @RobH for taking care of the arduous task of getting the RAM over! [19:40:43] FIRING: HaproxyKafkaNoMessages: Unexpected rate of produced HaproxyKafka messages by cp3072 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaNoMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=esams&var-instance=cp3072 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaNoMessages [19:55:43] RESOLVED: HaproxyKafkaNoMessages: Unexpected rate of produced HaproxyKafka messages by cp3072 - https://wikitech.wikimedia.org/wiki/HAProxyKafka#HaproxyKafkaNoMessages - https://grafana.wikimedia.org/d/d3e4e37c-c1d9-47af-9aad-a08dae2b3fd5/haproxykafka?orgId=1&var-site=esams&var-instance=cp3072 - https://alerts.wikimedia.org/?q=alertname%3DHaproxyKafkaNoMessages