[09:00:21] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750#12166353 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=a746a25c-706b-4c61-bdf4-214a1c78ff90) set by ayounsi@cumin1003 for 1:00:00 on 3 host(s) and their servi...
[09:12:04] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750#12166372 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=cce20762-fa19-4ada-b944-1a93ea8ed59f) set by ayounsi@cumin1003 for 2:00:00 on 3 host(s) and their servi...
[10:11:54] <wikibugs>	 10netops, 06Infrastructure-Foundations: Junos: investigate BGP rib sharding - https://phabricator.wikimedia.org/T320264#12166586 (10ayounsi)
[10:14:42] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: magru: upgrade core routers (2026 #2) - https://phabricator.wikimedia.org/T431750#12166601 (10ayounsi) 05Open→03Resolved a:03ayounsi Done.
[11:00:52] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: QFX5120 reporting 0 for neighbor bgp prefix counters on 23.4R2-S8.7 - https://phabricator.wikimedia.org/T433490 (10cmooney) 03NEW p:05Triage→03Medium
[11:03:34] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06SRE: QFX5120 reporting 0 for neighbor bgp prefix counters on 23.4R2-S8.7 - https://phabricator.wikimedia.org/T433490#12166714 (10cmooney) Case 2026-0729-825879 created
[11:31:06] <wikibugs>	 06Traffic, 06ServiceOps new: Receiving traffic from origin without sending it back to users (21st Jul) - https://phabricator.wikimedia.org/T432699#12166785 (10JMeybohm) In envoy land HTTP response status 0 means the client closed the connection before the server could send the full reply. Not sure if/where tha...
[12:00:17] <wikibugs>	 06Traffic, 10hCaptcha, 06Product Safety and Integrity, 06ServiceOps new, and 2 others: memcached errors seen in hCaptcha health checks - https://phabricator.wikimedia.org/T430340#12166872 (10JMeybohm) Hey @kostajh could you please elaborate what the impact of this is to the hCaptcha service and what you wo...
[12:28:16] <wikibugs>	 06Traffic, 13Patch-For-Review, 07Sustainability (Incident Followup): Experiment with single backend CDN nodes - https://phabricator.wikimedia.org/T288106#12166920 (10BBlack) Looks "reasonable"?  The biggest diff is in the loss in revalidation hits (which makes sense!), and probably causes some long tail incr...
[12:58:46] <wikibugs>	 06Traffic, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Standardize on a single web proxy hostname - https://phabricator.wikimedia.org/T429338#12167057 (10brouberol) We've now harmonized all DAGs defined in `airlfow-dags` to use url-downloader as proxies, whether the task runs on Kubernetes or YARN (through...
[12:58:54] <wikibugs>	 06Traffic, 06Data-Platform-SRE (2026-07-03 - 2026-07-31): Standardize on a single web proxy hostname - https://phabricator.wikimedia.org/T429338#12167062 (10brouberol) 05In progress→03Resolved
[13:31:29] <wikibugs>	 10netops, 06Infrastructure-Foundations, 07Sustainability (Incident Followup): Review thresholds for paging on DDoS alerts - https://phabricator.wikimedia.org/T431683#12167158 (10MLechvien-WMF)
[13:39:05] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: codfw: pod AB switches upgrade (2026) - https://phabricator.wikimedia.org/T426197#12167187 (10cmooney)
[13:47:20] <jinxer-wm>	 FIRING: DnsboxServiceMismatch: Service ntp-a state mismatch on dns3003:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=esams&var-instance=dns3003:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch
[13:47:37] <sukhe>	 ^ restarted ntpsec
[13:52:20] <jinxer-wm>	 RESOLVED: DnsboxServiceMismatch: Service ntp-a state mismatch on dns3003:9100 - https://wikitech.wikimedia.org/wiki/DNS#DnsboxServiceMismatch - https://grafana.wikimedia.org/d/96fb573c-0f3c-456a-886c-e50c29f3ed48/dns-box-service-state?var-site=esams&var-instance=dns3003:9100 - https://alerts.wikimedia.org/?q=alertname%3DDnsboxServiceMismatch
[14:11:32] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:New switch setup/configuration - https://phabricator.wikimedia.org/T418439#12167393 (10Papaul)
[15:17:44] <wikibugs>	 06Traffic, 10decommission-hardware: decommission lvs10[13-15] - https://phabricator.wikimedia.org/T433508 (10ssingh) 03NEW
[15:22:35] <wikibugs>	 10Domains, 06Traffic: Redirect fil.wikipedia.org to tl.wikipedia.org - https://phabricator.wikimedia.org/T432946#12167775 (10Sky_Harbor) >>! In T432946#12149114, @Aklapper wrote: >> The Language committee [has been informed of this proposal](https://meta.wikimedia.org/wiki/Talk:Language_committee#Redirecting_f...
[17:15:42] <wikibugs>	 10Domains, 06Traffic: Redirect fil.wikipedia.org to tl.wikipedia.org - https://phabricator.wikimedia.org/T432946#12168606 (10Aklapper) No problem. :)
[17:19:34] <swfrench-wmf>	 hello traffic friends - I believe we've reached the point where we're good to repool etcd client traffic in codfw following the reimages there (see scrollback from yesterday). in practice that means merging:
[17:19:34] <swfrench-wmf>	 * https://gerrit.wikimedia.org/r/c/operations/dns/+/1319128
[17:19:34] <swfrench-wmf>	 * https://gerrit.wikimedia.org/r/c/operations/puppet/+/1319126
[17:19:34] <swfrench-wmf>	 ... the former requires liberica restarts in eqsin and ulsfo [0] and the latter pybal restarts in codfw [1].
[17:19:34] <swfrench-wmf>	 would you all be able to assist with that part, or alternatively be available for me to drive and escalate to you if something goes wrong?
[17:19:35] <swfrench-wmf>	 [0] https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Move_other_clients
[17:19:35] <swfrench-wmf>	 [1] https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Move_PyBals
[17:23:21] <swfrench-wmf>	 ah, I see s.ukhe is away. brett: is this something you might be able to help with in some form?
[17:31:57] <wikibugs>	 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-urldownloader04 in project deployment-prep - https://phabricator.wikimedia.org/T433510#12168675 (10bd808) `lang=shell-session,counterexample bd808@deployment-urldownloader04:~$ sudo -i puppet agent -tv Info: Using en...
[17:32:20] <wikibugs>	 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-urldownloader04 in project deployment-prep - https://phabricator.wikimedia.org/T433510#12168680 (10bd808)
[17:32:25] <wikibugs>	 06Traffic, 06Infrastructure-Foundations, 06ServiceOps new, 06SRE, 13Patch-For-Review: Scaling urldownloaders by adding redundancy and load balancing - https://phabricator.wikimedia.org/T429175#12168681 (10bd808)
[17:36:49] <brett>	 swfrench-wmf: sure, let me catch up
[17:39:32] <swfrench-wmf>	 thanks! oh, and thanks for the reviews too :)
[17:39:41] <brett>	 swfrench-wmf: Gotcha. I'm happy to merge/deploy. Are there any other items you want before this happens?
[17:42:42] <swfrench-wmf>	 brett: awesome, thanks! no other prep required, no. what I would propose is that:
[17:42:42] <swfrench-wmf>	 1. (parallel) I'll take the DNS change and actuating that for non-liberica clients (e.g., confd) while you take the pybal side
[17:42:42] <swfrench-wmf>	 2. when pybal is done, let's sync and I'll give you the go-ahead to do liberica
[17:42:46] <swfrench-wmf>	 how does that sound?
[17:43:05] <brett>	 sounds good
[17:43:22] <swfrench-wmf>	 awesome - and starting now works?
[17:43:33] <brett>	 yep!
[17:43:46] <swfrench-wmf>	 fantastic - I'll hop over to -operations
[17:43:49] <brett>	 ack
[17:44:08] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad row A/B switch refresh prep - https://phabricator.wikimedia.org/T418012#12168699 (10cmooney) @VRiley to confirm what the next steps are here of most importance to unblock my side of the work:  # Cable the mgmt port on each of the...
[18:09:43] <wikibugs>	 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-urldownloader04 in project deployment-prep - https://phabricator.wikimedia.org/T433510#12168754 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/639532e722ab8c72d172217972913e3096f856a...
[18:13:07] <wikibugs>	 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-urldownloader04 in project deployment-prep - https://phabricator.wikimedia.org/T433510#12168761 (10Dzahn) Left a comment on https://gerrit.wikimedia.org/r/c/operations/puppet/+/1319102 - that looks indeed like the cu...
[18:15:59] <wikibugs>	 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-urldownloader04 in project deployment-prep - https://phabricator.wikimedia.org/T433510#12168768 (10bd808) https://gerrit.wikimedia.org/r/plugins/gitiles/cloud/instance-puppet/+/c6da354cddb150ee9b6fe1232e2c2b94b333b92...
[18:34:16] <wikibugs>	 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-urldownloader04 in project deployment-prep - https://phabricator.wikimedia.org/T433510#12168854 (10Dzahn) >>! In T433510#12168768, @bd808 wrote: > What is a reasonably test to see if the node is still working as expe...
[20:25:50] <wikibugs>	 06Traffic, 10Beta-Cluster-Infrastructure: Puppet agent failure detected on instance deployment-urldownloader04 in project deployment-prep - https://phabricator.wikimedia.org/T433510#12169216 (10bd808) 05Open→03Resolved a:03bd808 `lang=shell-session root@deployment-mediawiki14:~# curl --proxy http://d...
[20:33:41] <wikibugs>	 06Traffic: Relax request limits for upload cluster requests with JWT - https://phabricator.wikimedia.org/T431627#12169261 (10BCornwall) 05Open→03Resolved a:03BCornwall
[20:49:22] <wikibugs>	 06Traffic, 10MediaViewer, 10Thumbor, 07Browser-Support-Firefox, and 3 others: 429 too many requests when trying to view .webp image in MediaViewer in Firefox - https://phabricator.wikimedia.org/T418346#12169309 (10Krinkle) 05Open→03Resolved a:03Krinkle This has been fixed through a combination of...
[23:57:52] <wikibugs>	 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqsin, 06SRE: EQSIN:Switch refresh diagram and wiring - https://phabricator.wikimedia.org/T423724#12169750 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=0fbc87b8-270e-465d-9d10-b229a17d4554) set by pt1979@cumin1003 for 2:00:00...