[01:59:07] 10Domains, 06Traffic: Redirect fil.wikipedia.org to tl.wikipedia.org - https://phabricator.wikimedia.org/T432946#12296092 (10Bugreporter) 05Stalled→03Open [09:00:56] 10netops, 06Traffic, 10Cloud-VPS, 06Data-Platform-SRE, and 4 others: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#12296522 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Calling this one done, dumps-nfs.w.o is live as a standard load-ba... [13:04:21] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Enable gNMI on SRX devices and fasw - https://phabricator.wikimedia.org/T390052#12297314 (10ayounsi) Junos recommended for the SRX300 is now 24.4 so hopefully next year for 25.4, unless we decide to upgrade sooner. [13:11:02] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, and 2 others: Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#12297327 (10Jelto) [13:50:20] FIRING: DnsboxServiceMismatch: Service ntp-a state mismatch on dns1004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqiad&var-instance=dns1004:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [13:55:20] RESOLVED: DnsboxServiceMismatch: Service ntp-a state mismatch on dns1004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqiad&var-instance=dns1004:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [13:55:34] that's fine, ntp restarts [14:21:05] FIRING: [2x] DnsboxServiceMismatch: Service ntp-b state mismatch on dns1005:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [14:22:50] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-b state mismatch on dns1005:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [14:35:50] FIRING: [2x] DnsboxServiceMismatch: Service ntp-c state mismatch on dns1006:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [14:36:05] RESOLVED: DnsboxServiceMismatch: Service ntp-c state mismatch on dns1006:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqiad&var-instance=dns1006:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [14:43:30] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12297757 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=2feb17c1-1370-41ec-bccd-8b3aadc69c13) set by cmooney@cumin1004 for 1:00:00 o... [15:03:00] FIRING: [8x] PurgedHighEventLag: High event process lag with purged on cp4037:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [15:09:15] FIRING: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from US) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [15:09:43] FIRING: DnsboxServiceMismatch: Service ntp-c state mismatch on dns2006:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=codfw&var-instance=dns2006:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [15:10:07] RESOLVED: [24x] PurgedHighEventLag: High event process lag with purged on cp4037:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [15:14:51] RESOLVED: NELByCountryHigh: Elevated Network Error Logging events (tcp.timed_out from US) - https://wikitech.wikimedia.org/wiki/Network_monitoring#NEL_alerts - https://logstash.wikimedia.org/goto/5c8f4ca1413eda33128e5c5a35da7e28 - https://alerts.wikimedia.org/?q=alertname%3DNELByCountryHigh [15:15:12] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-b state mismatch on dns2005:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [15:37:20] FIRING: DnsboxServiceMismatch: Service ntp-b state mismatch on dns3004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=esams&var-instance=dns3004:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [15:42:20] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns3003:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [15:51:20] FIRING: DnsboxServiceMismatch: Service ntp-a state mismatch on dns4003:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=ulsfo&var-instance=dns4003:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [15:56:20] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-b state mismatch on dns3004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [16:07:50] FIRING: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns4003:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [16:12:50] RESOLVED: DnsboxServiceMismatch: Service ntp-b state mismatch on dns4004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=ulsfo&var-instance=dns4004:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [16:23:20] FIRING: DnsboxServiceMismatch: Service ntp-a state mismatch on dns5003:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqsin&var-instance=dns5003:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [16:28:20] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-b state mismatch on dns4004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [16:38:20] FIRING: DnsboxServiceMismatch: Service ntp-b state mismatch on dns5004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=eqsin&var-instance=dns5004:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [16:43:20] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns5003:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [16:53:20] FIRING: DnsboxServiceMismatch: Service ntp-a state mismatch on dns6001:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=drmrs&var-instance=dns6001:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [16:58:20] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-b state mismatch on dns5004:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [17:07:50] FIRING: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns6001:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [17:12:50] RESOLVED: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns6001:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [17:39:50] FIRING: [2x] DnsboxServiceMismatch: Service ntp-a state mismatch on dns7001:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [17:44:50] RESOLVED: DnsboxServiceMismatch: Service ntp-b state mismatch on dns7002:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=magru&var-instance=dns7002:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch [18:46:40] 06Traffic, 06Data-Engineering, 06Data-Engineering-Radar: Add backend information to webrequests in data lake - https://phabricator.wikimedia.org/T436223#12299535 (10ssingh) This is something that Traffic would undertake in theory, correct. I will need to check with the team and particularly @Fabfur so will f... [18:57:02] 06Traffic, 06DC-Ops, 10ops-drmrs: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12299590 (10ssingh) Hi @RobH. Any update from their end? [18:59:12] 06Traffic, 06DC-Ops, 10ops-drmrs: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12299596 (10RobH) This had remote hands update that it was done, double checking the work now. [19:05:05] 06Traffic, 06DC-Ops, 10ops-drmrs: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12299637 (10RobH) [19:06:15] 06Traffic, 06DC-Ops, 10ops-drmrs: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12299642 (10RobH) a:05RobH→03ssingh @ssingh: The new memory is dectected, same speed as the old just I can compare the serial output to the photos, throwing the photos h... [19:09:20] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12299657 (10cmooney) [21:27:27] brett: looks like I'm picking up some portion of your netbox hiera changes for cp6008. [21:27:37] just the locality information AFAICT [21:27:37] O.o [21:27:45] how so? [21:28:27] brett: this is what I'm seeing https://www.irccloud.com/pastebin/lbPL3pq5/ [21:28:49] swfrench-wmf: ugh, it's been so long I forgot that's what I need to do [21:29:02] thanks for doing that! [21:29:24] no worries at all! it was just triggered by the change of status from failed -> active :) [21:29:49] just wanted to flag in case you were puzzled that the diff _didn't_ appear in a subsequent run [21:31:19] oh totally, I was *definitely* looking for that and wondering. Definitely not thankful that you did it for me :) [21:33:56] 06Traffic, 06DC-Ops, 10ops-drmrs: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12300442 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2003 for host cp6008.drmrs.wmnet with OS trixie [22:27:23] 06Traffic, 06DC-Ops, 10ops-drmrs: hw troubleshooting: Memory failure for cp6008.drmrs.wmnet - https://phabricator.wikimedia.org/T431651#12300565 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2003 for host cp6008.drmrs.wmnet with OS trixie completed: - cp6008 (**PASS**) -... [23:30:28] FIRING: [16x] NodeTextfileStale: Stale textfile for dns1004:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale