[07:56:37] 10netops, 06Infrastructure-Foundations, 06SRE, 06Data-Platform-SRE (2026.01.05 - 2026.01.23), 07Essential-Work: Socket leaking on some dse-k8s row C & D hosts - https://phabricator.wikimedia.org/T414460#11544162 (10JAllemandou) >>! In T414460#11542216, @ops-monitoring-bot wrote: > Roll-reboot of nodes in... [10:43:36] 10netops, 06Infrastructure-Foundations, 06SRE, 06Data-Platform-SRE (2026.01.05 - 2026.01.23), 07Essential-Work: Socket leaking on some dse-k8s row C & D hosts - https://phabricator.wikimedia.org/T414460#11544445 (10ops-monitoring-bot) Roll-reboot of nodes in dse-eqiad cluster started by btullis: * dse-k8... [10:46:13] 10netops, 06Infrastructure-Foundations, 06SRE, 06Data-Platform-SRE (2026.01.05 - 2026.01.23), 07Essential-Work: Socket leaking on some dse-k8s row C & D hosts - https://phabricator.wikimedia.org/T414460#11544459 (10BTullis) >>! In T414460#11544162, @JAllemandou wrote: >>>! In T414460#11542216, @ops-monit... [10:57:57] 06Traffic, 06SRE: Offer AuthDNS service over IPv6 - https://phabricator.wikimedia.org/T81605#11544493 (10cmooney) Just want to add my two cents on the problem we hit trying to make the IPv6 IPs live. * I personally think it's cleaner if the dns servers only have configured, and only listen on, the IPs that ar... [11:19:17] 06Traffic, 06serviceops, 06MediaWiki-Platform-Team (Q3 Kanban Board), 07OKR-Work: api-gateway helm chart: rest routes should return retry-after when a rate limit applies. - https://phabricator.wikimedia.org/T405636#11544549 (10daniel) 05In progress→03Resolved [13:23:06] 06Traffic, 10Incident Tooling, 06ServiceOps new: Proof of Concept: SquareOne CDN Dashboards - https://phabricator.wikimedia.org/T414665#11544971 (10jijiki) [13:58:50] 06Traffic, 06MW-Interfaces-Team, 06ServiceOps new, 10ServiceOps-SharedInfra, and 4 others: Epic: API Rate Limiting Architecture - https://phabricator.wikimedia.org/T399291#11545094 (10MLechvien-WMF) [14:49:46] 06Traffic, 06SRE, 13Patch-For-Review: Offer AuthDNS service over IPv6 - https://phabricator.wikimedia.org/T81605#11545351 (10ssingh) https://puppet-compiler.wmflabs.org/output/1230351/7941/ ` Interface::Ip[ns1-v6] Exec[ip addr add 2620:0:861:53::1/128 label lo:anycast dev lo] Augeas[lo_2620:0:8... [15:05:06] 10netops, 06DBA, 06Infrastructure-Foundations, 10observability: librenms.syslog table is 800GB - https://phabricator.wikimedia.org/T415270 (10Marostegui) 03NEW [15:06:35] 10netops, 06DBA, 06Infrastructure-Foundations, 10observability: librenms.syslog table is 800GB - https://phabricator.wikimedia.org/T415270#11545478 (10Marostegui) p:05Triage→03High [15:09:02] 10netops, 06DBA, 06Infrastructure-Foundations, 10observability, 06SRE: librenms.syslog table size - https://phabricator.wikimedia.org/T349362#11545494 (10Marostegui) [15:09:18] 10netops, 06DBA, 06Infrastructure-Foundations, 10observability: librenms.syslog table is 800GB - https://phabricator.wikimedia.org/T415270#11545495 (10Marostegui) [15:30:15] 10netops, 06DBA, 06Infrastructure-Foundations, 10observability: librenms.syslog table is 800GB - https://phabricator.wikimedia.org/T415270#11545586 (10cmooney) > At T349362#9267648 we discussed having just 15 days of retention, is that still the case? If it is, maybe we need to reduce it a bit further? T... [15:32:25] 10netops, 06DBA, 06Infrastructure-Foundations, 10observability: librenms.syslog table is 800GB - https://phabricator.wikimedia.org/T415270#11545610 (10Marostegui) Thank you! [15:37:14] 06Traffic, 07Essential-Work, 13Patch-For-Review, 06Test Kitchen (Test Kitchen (Experiment Platform Sprint 18)): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11545639 (10Sfaci) Hi again @ssingh! We are going to double the tra... [16:02:09] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: MediaSearch should stop relying on render map config - https://phabricator.wikimedia.org/T415282 (10MatthewVernon) 03NEW [16:02:41] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: FY 25/26 WE 5.4.10 Standard Thumbnail Sizes Only - https://phabricator.wikimedia.org/T414805#11545791 (10MatthewVernon) p:05Triage→03High [16:02:44] 06Traffic, 06Data-Persistence, 10MediaViewer, 10SRE-swift-storage, 10Thumbor: MediaSearch should stop relying on render map config - https://phabricator.wikimedia.org/T415282#11545792 (10MatthewVernon) p:05Triage→03High [16:04:33] 06Traffic, 06Data-Persistence, 10MediaSearch, 10SRE-swift-storage, 10Thumbor: MediaSearch should stop relying on render map config - https://phabricator.wikimedia.org/T415282#11545794 (10Ladsgroup) To emphasize: MediaSearch does respond with standard sizes but the js uses a config that should be used. [16:41:47] 06Traffic, 07Essential-Work, 13Patch-For-Review, 06Test Kitchen (Test Kitchen (Experiment Platform Sprint 18)): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11545960 (10ssingh) >>! In T407570#11545639, @Sfaci wrote: > Hi agai... [18:50:44] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Decom Asw Switches in Rows C & D - https://phabricator.wikimedia.org/T412525#11546581 (10cmooney) >>! In T412525#11528293, @Jclark-ctr wrote: > @cmooney i have disconnected all the switches @Jclark-ctr I'm ha... [19:22:03] 06Traffic, 10API Platform, 10MediaWiki-User-login-and-signup, 06MediaWiki-Platform-Team (Q3 Kanban Board), 13Patch-For-Review: Login with `action=login` and bot password does not create a JWT session cookie - https://phabricator.wikimedia.org/T415007#11546633 (10DAlangi_WMF) 05Open→03In progress a:0... [19:23:38] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Decom Asw Switches in Rows C & D - https://phabricator.wikimedia.org/T412525#11546641 (10VRiley-WMF) Hey @cmooney, I was able to check C2 and confirm there is a cable from the managment port on the switch to th... [19:26:03] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Decom Asw Switches in Rows C & D - https://phabricator.wikimedia.org/T412525#11546643 (10VRiley-WMF) Please disreguard, I thought we were talking about lsw1-c2 [19:36:04] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Decom Asw Switches in Rows C & D - https://phabricator.wikimedia.org/T412525#11546665 (10Reedy) [20:17:55] 06Traffic, 07Essential-Work, 13Patch-For-Review, 06Test Kitchen (Test Kitchen (Experiment Platform Sprint 18)): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11546721 (10KReid-WMF) @ssingh The test-kitchen entry is: https://te... [20:19:52] 06Traffic, 07Essential-Work, 13Patch-For-Review, 06Test Kitchen (Test Kitchen (Experiment Platform Sprint 18)): Test the impact of incremental increase in traffic for cache splitting experiments - https://phabricator.wikimedia.org/T407570#11546724 (10ssingh) >>! In T407570#11546721, @KReid-WMF wrote: > @ss... [20:46:30] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Decom Asw Switches in Rows C & D - https://phabricator.wikimedia.org/T412525#11546770 (10cmooney) Thanks for the help with this one guys! All the switches have been reset to factory defaults. So they can be re... [22:47:21] 06Traffic, 10Huggle: Huggle is getting rate-limited when working on multiple wikis in parallel - https://phabricator.wikimedia.org/T415141#11547086 (10Framawiki) Indeed I don't see Cookie header in requests Huggle is sending, @Petrb if you have a moment to take a look around Cookiejar in ApiQuery ?