[00:29:40] FIRING: [7x] VarnishHighThreadCount: cp3066:0's Varnish thread count nearly maxed out - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [00:34:40] FIRING: [8x] VarnishHighThreadCount: cp3066:0's Varnish thread count nearly maxed out - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [00:44:40] RESOLVED: [8x] VarnishHighThreadCount: cp3066:0's Varnish thread count nearly maxed out - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [07:51:34] 06Traffic, 06Commons, 10MediaWiki-File-management, 06SRE, and 2 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12284293 (10Tgr) Possibly Varnish normalizes thumb URLs but doesn't normalize original URLs (which until... [08:02:11] 06Traffic: Review fetch_external_clouds_vendors_nets script to improve Azure fetcher - https://phabricator.wikimedia.org/T395236#12284300 (10Fabfur) 05Open→03Resolved [08:02:57] 06Traffic, 10bot-traffic-requests: FY 25/26 WE 5.4.6 Classify the top 30 spiders by traffic as known bots - https://phabricator.wikimedia.org/T408061#12284302 (10Fabfur) 05Open→03Resolved [08:04:14] 06Traffic, 13Patch-For-Review: Enable QoS for upload video files - https://phabricator.wikimedia.org/T412785#12284305 (10Fabfur) 05Open→03Resolved [08:04:34] 06Traffic, 06SRE: Benthos - remove the kafka output module - https://phabricator.wikimedia.org/T398916#12284308 (10Fabfur) 05Open→03Declined [08:05:31] 06Traffic, 10observability, 06SRE Observability, 13Patch-For-Review: Benthos dependencies should be vendored - https://phabricator.wikimedia.org/T388261#12284310 (10Fabfur) 05Open→03Declined [08:06:36] 10netops, 06Infrastructure-Foundations, 10Observability-Metrics, 06SRE: Expand blackbox icmp probes to ping specific router interfaces/circuits - https://phabricator.wikimedia.org/T435855#12284322 (10cmooney) 05Open→03Resolved >>! In T435855#12274375, @ayounsi wrote: > Max RTT doesn't show when usi... [08:06:37] 06Traffic: Return 403 to non HEAD|GET requests in HAProxy tls frontend - https://phabricator.wikimedia.org/T360766#12284324 (10Fabfur) 05Open→03Declined [08:53:59] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, and 2 others: Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#12284435 (10Jelto) @ssingh reported in `wikimedia-traffic` a pending pybal restart and manually restarted pybal. I used the `sre.loadbalance... [09:14:29] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12284516 (10ayounsi) [09:15:32] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12284517 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by ayounsi@cumin1004 for host netflow2003.codfw.wmnet with OS trixie [09:40:26] 06Traffic, 10Liberica, 10Prod-Kubernetes, 06ServiceOps, and 2 others: Migrate Wikikube k8s apiserver and services to IPIP - https://phabricator.wikimedia.org/T420436#12284715 (10Jelto) [09:43:35] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Create a 3rd Netflow instance in eqiad/codfw - https://phabricator.wikimedia.org/T436517#12284723 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by ayounsi@cumin1003 for host netflow2005.codfw.wmnet with OS trixie [09:55:49] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12284786 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by ayounsi@cumin1004 for host netflow2003.codfw.wmnet with OS trixie completed: - netflow2003 (**PASS**) - Downtimed... [10:24:47] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Power alert for cr1-eqiad old line cards - https://phabricator.wikimedia.org/T436814#12284886 (10Jclark-ctr) Removed FPC 4 and 5 and installed blanking panels [10:29:33] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Create a 3rd Netflow instance in eqiad/codfw - https://phabricator.wikimedia.org/T436517#12284900 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by ayounsi@cumin1003 for host netflow2005.codfw.wmnet with OS trixie completed: - n... [11:34:46] 06Traffic, 06Data-Engineering: Add backend information to webrequests in data lake - https://phabricator.wikimedia.org/T436223#12285273 (10GGoncalves-WMF) This looks like something that Traffic SRE would initiate, ensuring this information is included in events produced at the edge, and then Data Engineering c... [11:35:02] 06Traffic, 06Data-Engineering, 06Data-Engineering-Radar: Add backend information to webrequests in data lake - https://phabricator.wikimedia.org/T436223#12285275 (10GGoncalves-WMF) [12:23:49] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Power alert for cr1-eqiad old line cards - https://phabricator.wikimedia.org/T436814#12285481 (10Jclark-ctr) 05Open→03Resolved [12:34:19] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 4 others: Pywikibot reports maxlag retry error on Wikidata - https://phabricator.wikimedia.org/T421642#12285533 (10DL2204) Over the last days, I constantly get maxlag errors, mostly when attempting to get items using wikibaseintegrator, but also (to a lesser e... [13:14:21] 06Traffic, 06Machine-Learning-Team (Q1 FY2026-27): CDN caching request for TTS v1 audio files - https://phabricator.wikimedia.org/T434046#12285798 (10BTullis) As I understand it (from T435952 and T436758) , the back-end service hosting these files for the period of the experiment will be: [[https://wikitech.wi... [13:23:34] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12285828 (10ayounsi) [13:23:57] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12285830 (10ayounsi) [13:24:36] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12285838 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by ayounsi@cumin1004 for host netflow2004.codfw.wmnet with OS trixie [13:29:48] fabfur: I'll have to go in a meeting now, druid is just back, I havent had the time to see if I manage to identify the traffic coming from CI that was rejected because of its lack of UA headers [13:30:10] arnaudb: the "best thing" to do would be to set the UA [13:30:46] yeah I sent a MR in that direction sukhe but fabfur asked for a turnilo query that highlighted the issue to see if there was another way around this [13:31:06] fabfur: what do you have in mind? [13:31:30] arnaudb: consider that given turnilo is downsampled, there won't be many of these, don't sweat too much over it :) [13:31:45] ack noted [13:31:46] sukhe: I'll answer in pvt [13:31:52] ok! [13:32:58] arnaudb: fabfur's point makes sense, let's talk more in private if it comes to that [13:33:20] (it still requires UA set, anyway, or it won't work) [13:33:29] ack [13:33:38] Hi! Is there something happening to a docker registry? I'm getting `https://registry.cloud.releng.team/v2/repos/data-engineering/airflow-dags/cache/build-test-image/blobs/uploads/: 500 Internal Server Error` from a gitlab worker [13:34:13] atsukoito: the only thing that comes to mind is the work that arnaudb is doing on the gitlab behind CDN thing [13:34:16] from our end, no [13:35:23] I'll ping #wikimedia-releng [13:37:04] sukhe: it is probably not related, since it is responding but failing [13:38:42] atsukoito: looking at response headers is also useful in these cases, like that service is not behind the CDN [13:38:54] 06Traffic, 06SRE, 07Wikimedia-Incident: Wikipedia and Wiktionary pages returning "upstream connect error or disconnect/reset before headers" - https://phabricator.wikimedia.org/T436004#12285930 (10Bewfip) →14Duplicate dup:03T436001 [13:39:00] but for other services, usually the `backend` header helps [13:39:02] 06Traffic, 06SRE, 07Wikimedia-Incident: Wikipedia and Wiktionary pages returning "upstream connect error or disconnect/reset before headers" - https://phabricator.wikimedia.org/T436004#12285935 (10Bewfip) [13:48:36] fabfur: I can't look at response headers, since it is a gitlab runner [13:50:55] atsukoito: so the issue is more likely to be on the backend server, or the access from a gitlab worker, than the CDN [13:51:59] I say this because: 1) it seems that we can access the URL just fine from purely the CDN, 2) there were no changes made on our end to effect this specific backend, 3) the 500 in that case therefore indicates that the origin is more likely to be the issue than the CDN [13:53:56] > https://registry.cloud.releng.team/ is also not behind the CDN [13:54:16] but I am thinking now that I am missing something fairly obvious, so please let me know :) [13:54:18] sukhe: tl;dr it was the gitlab migration. registry is going back to gitlab [13:54:32] ok! glad to hear it got resolved [13:54:40] https://www.irccloud.com/pastebin/jVdrg2hi/ [13:55:13] it is not yet, but releng is aware [13:55:22] > jnuche: hi, that is most likely T425441 and https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/113 [13:55:23] T425441: gitlab behind CDN - https://phabricator.wikimedia.org/T425441 [13:55:36] (for posterity, docker-registery.wikimedia.org is behind the CDN, but not registry.cloud.releng) [13:57:15] yup, I was initially confused too, because there are 2 registries [14:11:59] I'll move Gitlab out of the CDN to simplify the fixes, sukhe thanks for the +1 on the backend change, do you want me to merge it or do you prefer traffic folks behind the wheel? [14:12:24] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12286112 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by ayounsi@cumin1004 for host netflow2004.codfw.wmnet with OS trixie completed: - netflow2004 (**FAIL**) - Downtimed... [14:12:25] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12286113 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by ayounsi@cumin1004 for host netflow2004.codfw.wmnet with OS trixie executed with errors: - netflow2004 (**FAIL**)... [14:26:33] arnaudb: was in a meeting sorry [14:26:38] whatever works, let us know [14:26:44] I can do it now if you want [14:27:31] sukhe: sgtm! that would be perfect :) [14:27:41] thanks! [14:27:44] ok on it [14:37:21] arnaudb: rolling it out [14:48:11] arnaudb: all done on A:cp-text [15:45:40] 10netops, 06Infrastructure-Foundations: Upgrade netflow hosts to Trixie - https://phabricator.wikimedia.org/T424478#12286593 (10ayounsi) [15:48:48] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Create a 3rd Netflow instance in eqiad/codfw - https://phabricator.wikimedia.org/T436517#12286602 (10ayounsi) 05Open→03Resolved a:03ayounsi All done. [16:52:40] 06Traffic, 06Commons, 10MediaWiki-File-management, 06SRE, and 2 others: Varnish serving outdated version at original/non-thumb URL of overwritten file upload - https://phabricator.wikimedia.org/T425216#12286788 (10Bawolff) >>! In T425216#12284293, @Tgr wrote: > Possibly Varnish normalizes thumb URLs but do... [20:24:12] 06Traffic, 06SRE Observability, 13Patch-For-Review: Redirect http(s?)://wikimediastatus.net to https://www.wikimediastatus.net - https://phabricator.wikimedia.org/T419887#12287636 (10BCornwall) 05In progress→03Resolved We've kept the domain registration in markmonitor but moved the NS to porkbun, whi...