[00:16:16] 06Traffic, 10Hiddenparma: Introduce known-client identity objects and integrate with requestctl - https://phabricator.wikimedia.org/T403220#11165799 (10Scott_French) [06:38:00] FIRING: [2x] PurgedHighBacklogQueue: Large backlog queue for purged on cp5026:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [06:40:40] FIRING: VarnishHighThreadCount: Varnish's thread count on cp5026:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://grafana.wikimedia.org/d/wiU3SdEWk/cache-host-drilldown?viewPanel=99&var-site=eqsin&var-instance=cp5026 - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [06:43:00] RESOLVED: [2x] PurgedHighBacklogQueue: Large backlog queue for purged on cp5026:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighBacklogQueue [06:45:40] FIRING: [2x] VarnishHighThreadCount: Varnish's thread count on cp5026:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [06:50:40] FIRING: [2x] VarnishHighThreadCount: Varnish's thread count on cp5026:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [07:35:40] FIRING: [3x] VarnishHighThreadCount: Varnish's thread count on cp5026:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [07:50:40] FIRING: [4x] VarnishHighThreadCount: Varnish's thread count on cp5026:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [07:55:40] FIRING: [3x] VarnishHighThreadCount: Varnish's thread count on cp5026:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [08:10:40] RESOLVED: [2x] VarnishHighThreadCount: Varnish's thread count on cp5026:0 is high - https://wikitech.wikimedia.org/wiki/Varnish - https://alerts.wikimedia.org/?q=alertname%3DVarnishHighThreadCount [08:25:09] 10netops, 06Traffic, 06Infrastructure-Foundations: Upgrade End Of Support Junos - https://phabricator.wikimedia.org/T390813#11166256 (10ayounsi) There is no particular rush, let's say before the end of 2025 ? [09:25:27] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: Updates for Nokia switch support - https://phabricator.wikimedia.org/T404146#11166358 (10ayounsi) > We need to allow port number 48 on the Nokias, but not port number 0 as they start from 1 We already (and lazily) do : `min_value=0, max_value=48` which... [13:00:01] 10netops, 06Infrastructure-Foundations, 06SRE: Netbox: Updates for Nokia switch support - https://phabricator.wikimedia.org/T404146#11167203 (10ayounsi) [13:10:02] 10netops, 06Traffic, 06Infrastructure-Foundations: Upgrade End Of Support Junos - https://phabricator.wikimedia.org/T390813#11167264 (10ssingh) >>! In T390813#11166256, @ayounsi wrote: > There is no particular rush, let's say before the end of 2025 ? OK thanks for sharing, we will triage and plan accordingly! [13:52:20] 06Traffic, 06MediaWiki-Platform-Team, 10MobileFrontend, 06Reader Experience Team: Toggling desktop view doesn't toggle user back into mobile mode - https://phabricator.wikimedia.org/T403866#11167523 (10Etonkovidova) Checked the steps on iPhone 11 (iOS 18) - the Chrome option "Request desktop site" doesn't... [14:38:09] 06Traffic, 10DNS, 06FR-donorrelations, 06SRE: Custom URL for survey pop-up - https://phabricator.wikimedia.org/T400278#11167815 (10ssingh) Hi @EBrill-WMF: Happy to set up a time to talk about this; please let me know and I can set that up. Thanks. [15:22:46] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Management routers: use BGP instead of OSPF - https://phabricator.wikimedia.org/T294845#11168027 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=eb9214cb-2708-4519-b3c6-38d4f0f7cf7d) set by pt1979@cumin1002 for 1:00:00 o... [15:39:33] 06Traffic, 06serviceops, 10WE4.2 Bot detection (WE4.2 hCaptcha account creation trial): Investigate options for per-wiki, percentage-based rollout of hCaptcha - https://phabricator.wikimedia.org/T404184#11168134 (10jijiki) [15:40:57] 06Traffic, 06serviceops, 10WE4.2 Bot detection (WE4.2 hCaptcha account creation trial): Investigate options for per-wiki, percentage-based rollout of hCaptcha - https://phabricator.wikimedia.org/T404184#11168146 (10jijiki) [16:05:55] 06Traffic, 06Data-Engineering, 06Infrastructure-Foundations: Export development_network_probe data to Puppet servers for CDN deployment - https://phabricator.wikimedia.org/T402512#11168272 (10Ottomata) Let us know how/when we can help, we are grooming tasks and are going to put this in on our radar. [16:18:08] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Management routers: use BGP instead of OSPF - https://phabricator.wikimedia.org/T294845#11168372 (10Papaul) 05Open→03Resolved mr1-eqsin and cr2/3-eqsin are now running BGP for the management network. Resolving this task. Thanks @ay... [16:24:39] 06Traffic, 06Data-Engineering, 06SRE: Add pageview information to turnilo's webrequest_sampled_live (is_pageview is always "-") - https://phabricator.wikimedia.org/T402612#11168439 (10Ottomata) @JAllemandou can you weigh in here? @CDanis def possible to reuse the logic in Java, but it would probably be a la... [16:28:26] 06Traffic, 06Data-Engineering, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11168454 (10Ottomata) There is probably a reason why not, but could we just put dumps.wm.org behind the CDN? Then requests to it w... [16:31:55] 06Traffic: 403 errors with user-agent - https://phabricator.wikimedia.org/T404219#11168527 (10Pppery) [16:34:39] 06Traffic: 403 errors with user-agent - https://phabricator.wikimedia.org/T404219#11168542 (10Aklapper) Hi @Lupascriptix, thanks for taking the time to report this! Per https://x.com/OpenRefine/status/1963195708619591913, OpenRefine 3.9.5 should have fixed this. Do you run that latest version? [16:37:54] 06Traffic: 403 errors with user-agent - https://phabricator.wikimedia.org/T404219#11168566 (10ssingh) Thanks for reporting @Lupascriptix; this is indeed the right place. Replying on behalf of the SRE/Traffic team: can you confirm what @Aklapper said and follow up? We will try looking it at then. [16:39:47] 06Traffic, 06Data-Engineering, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11168570 (10taavi) >>! In T403159#11168454, @Ottomata wrote: > There is probably a reason why not, but could we just put dumps.wm.o... [16:48:59] 06Traffic: 403 errors with user-agent - https://phabricator.wikimedia.org/T404219#11168662 (10Lupascriptix) Just updated OpenRefine and that seems to have helped, but it's still throwing quite a few 403 errors at me. (If it matters, I'm also using SPARQL queries within OR). Just checked the python scripts and th... [16:53:39] 06Traffic, 06Data-Engineering, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11168700 (10Ottomata) TIL you can point rsync at dumps.wikmedia.org (if we allow you to). https://phabricator.wikimedia.org/T30655... [17:04:02] 06Traffic, 06Data-Engineering, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11168760 (10ssingh) This has been brought up recently in other discussions as well and Traffic is happy to triage it if required bu... [17:06:30] 06Traffic: 403 errors with user-agent - https://phabricator.wikimedia.org/T404219#11168779 (10ssingh) >>! In T404219#11168662, @Lupascriptix wrote: > Just updated OpenRefine and that seems to have helped, but it's still throwing quite a few 403 errors at me. (If it matters, I'm also using SPARQL queries within O... [17:09:25] 06Traffic, 06Data-Engineering, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11168807 (10BTullis) How about if we did something like this? ` (1) nginx -> (2) rsyslog -> (3) kafka-logging (5) kaf... [17:13:28] 06Traffic, 06Data-Engineering, 06Data-Platform-SRE, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11168825 (10BTullis) [17:15:38] 06Traffic: 403 errors with user-agent - https://phabricator.wikimedia.org/T404219#11168833 (10Lupascriptix) >>! In T404219#11168779, @ssingh wrote: >>>! In T404219#11168662, @Lupascriptix wrote: >> Just updated OpenRefine and that seems to have helped, but it's still throwing quite a few 403 errors at me. (If it... [17:17:25] 06Traffic: 403 errors with user-agent - https://phabricator.wikimedia.org/T404219#11168836 (10ssingh) If it helps, you can perhaps detail what you are doing on the task and someone from Wikidata can chime in. Let us know here again if that doesn't work. [17:19:08] 06Traffic, 06Data-Engineering, 06Data-Platform-SRE, 10Wikidata, 10Wikidata Analytics: Airflow processes to import dump logs and generate monthly metrics - https://phabricator.wikimedia.org/T403159#11168854 (10BTullis) Alternatively, we could just use an `hdfs-rsync` (or similar) process on the clouddumps... [18:02:54] FYI, I'm going to start moving the rest-gateway DNS changes in [0] forward. this should be largely uneventful, but wanted to mention that this will start soon. [18:02:54] [0] https://phabricator.wikimedia.org/T400131 [18:19:51] * sukhe trusts the swfrench-wmf force [18:35:52] 10netops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: Netbox: Updates for Nokia switch support - https://phabricator.wikimedia.org/T404146#11169333 (10cmooney) >>! In T404146#11166358, @ayounsi wrote: > We already (and lazily) do : `min_value=0, max_value=48` which was to accommodate both SON... [18:58:15] 10netops, 06Infrastructure-Foundations, 06SRE: Renumber link addressing to lsw1-e1-eqiad and lsw1-f1-eqiad - https://phabricator.wikimedia.org/T404248 (10cmooney) 03NEW p:05Triage→03Low [19:48:56] 06Traffic, 06Data-Engineering, 06SRE: Add pageview information to turnilo's webrequest_sampled_live (is_pageview is always "-") - https://phabricator.wikimedia.org/T402612#11169688 (10JAllemandou) Building a JAR containing the `is_pageview` logic with a few dependencies as possible is easy. My little researc... [21:53:16] 06Traffic, 06MediaWiki-Platform-Team, 10MobileFrontend, 06Reader Experience Team: Toggling desktop view doesn't toggle user back into mobile mode - https://phabricator.wikimedia.org/T403866#11170106 (10Jdlrobson-WMF) [22:02:43] 06Traffic, 06MediaWiki-Platform-Team: Mobile domain removal: Random page doesn't work with mobile/desktop switching - https://phabricator.wikimedia.org/T404023#11170163 (10Jdlrobson-WMF) [22:02:46] 06Traffic, 06MediaWiki-Platform-Team, 10MobileFrontend, 06Reader Experience Team: Toggling desktop view doesn't toggle user back into mobile mode - https://phabricator.wikimedia.org/T403866#11170164 (10Jdlrobson-WMF) [22:02:50] 06Traffic, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review, 07User-notice: [Rollout Phase 3] Enable unified mobile routing on remaining wikis - https://phabricator.wikimedia.org/T403510#11170162 (10Jdlrobson-WMF) [22:30:37] 06Traffic, 06MediaWiki-Platform-Team, 06Reader Experience Team, 10MobileFrontend (Core): Toggling desktop view doesn't toggle user back into mobile mode - https://phabricator.wikimedia.org/T403866#11170219 (10Jdlrobson-WMF) [22:39:56] 06Traffic, 10MediaWiki-Platform-Team (Radar), 07patch-welcome: Desktop view link on private wiki login required pages is broken - https://phabricator.wikimedia.org/T352798#11170297 (10Jdlrobson-WMF) Does not seem related to MobileFrontend. [22:41:51] 06Traffic, 06MediaWiki-Platform-Team, 06Reader Experience Team, 10MobileFrontend (Core PHP): Toggling desktop view doesn't toggle user back into mobile mode - https://phabricator.wikimedia.org/T403866#11170310 (10Jdlrobson-WMF) p:05Triage→03High [22:43:43] 06Traffic, 10MediaWiki-Action-API, 10MobileFrontend, 06MW-Interfaces-Team, 07JavaScript: Requests fail with Access-Control-Allow-Origin errors when using ForeignApi on iOS Safari - https://phabricator.wikimedia.org/T399674#11170316 (10Jdlrobson-WMF) [22:43:50] 06Traffic, 10MediaWiki-Action-API, 06MW-Interfaces-Team, 07JavaScript, 10MobileFrontend (Tracking): Requests fail with Access-Control-Allow-Origin errors when using ForeignApi on iOS Safari - https://phabricator.wikimedia.org/T399674#11170318 (10Jdlrobson-WMF)