[04:00:38] 10Domains, 10WMF-Annual-Report, 10wikimediafoundation.org: Update redirects for annual.wikimedia.org - https://phabricator.wikimedia.org/T336217 (10Varnent) [04:21:59] ^ dont worry about that one, that's my team taking care of it and i'm on clinic duty [04:22:05] removed the DNS tag [08:48:05] 10Traffic, 10RESTBase, 10RESTBase-API, 10SRE: REST API is not invalidating caches after template and/or module changes - https://phabricator.wikimedia.org/T335770 (10akosiaris) 05Open→03Resolved a:03akosiaris I am gonna tentatively resolve this in the interest of not leaving a ticket lingering open l... [09:00:42] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops: Add configuration file support to mw-on-k8s.lua ATS script - https://phabricator.wikimedia.org/T336037 (10Joe) 05Open→03In progress p:05Triage→03High [09:00:53] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Joe) [10:03:43] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops: Add traffic sampling support to mw-on-k8s.lua ATS script - https://phabricator.wikimedia.org/T336038 (10Joe) 05Open→03In progress p:05Triage→03High [10:03:55] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Joe) [10:25:06] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Migrate row E/F network aggregation to dedicated Spine switches - https://phabricator.wikimedia.org/T322937 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=12105eb2-e5ac-4f19-9896-9ba53e1acd48) set by cmooney@cumin1001 f... [13:19:45] 10Traffic, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/decom codfw unified decommission task - https://phabricator.wikimedia.org/T335777 (10Papaul) a:03ssingh [14:45:51] 10Traffic, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/decom codfw unified decommission task - https://phabricator.wikimedia.org/T335777 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by sukhe@cumin2002 for hosts: `lvs2008.codfw.wmnet` - lvs2008.codfw.wmnet (**WARN**) - Downtimed ho... [14:51:37] 10Traffic, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/decom codfw unified decommission task - https://phabricator.wikimedia.org/T335777 (10ssingh) [15:05:21] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10Jhancock.wm) @Papaul onboard port 1 cable ID: 12110 onboard port 2 cable ID: 12109 NIC port 1 cable ID: 12108 NIC port 2 cable ID: 12174 [15:20:20] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10Papaul) [16:29:23] 10Wikimedia-Apache-configuration, 10DNS, 10SRE, 10Traffic-Icebox, 10MW-1.40-notes (1.40.0-wmf.24; 2023-02-20): Remove aliases `minnan` and `zh-cfr` for the Min Nan Wikipedia - https://phabricator.wikimedia.org/T230382 (10RLazarus) Disregard those last two SAL entries, wrong task number. :) [16:30:47] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10Papaul) [16:35:28] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin2002 for host lvs2012.codfw.wmnet with OS bullseye [17:13:54] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin2002 for host lvs2012.codfw.wmnet with OS bullseye completed... [17:19:34] 10Traffic, 10netops, 10DBA, 10Data-Platform-SRE, and 9 others: codfw row D switches upgrade - https://phabricator.wikimedia.org/T335042 (10BTullis) [17:59:18] 10Traffic, 10Content-Transform-Team-WIP, 10RESTBase, 10SRE, and 5 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365 (10Kappakayala) @KOfori, Could you please have someone from your team to help with consultation. Based on my chat with Frantz,... [18:02:09] 10Traffic, 10Content-Transform-Team-WIP, 10RESTBase, 10SRE, and 5 others: PCS caching and pregeneration when restbase is decommissioned - https://phabricator.wikimedia.org/T319365 (10KOfori) @Kappakayala indeed. Had a quick chat earlier with @FJoseph-WMF and briefly with the team. We'll set something up to... [18:17:26] 10Traffic, 10RESTBase, 10RESTBase-API, 10SRE: REST API is not invalidating caches after template and/or module changes - https://phabricator.wikimedia.org/T335770 (10Brycehughes) @akosiaris Fair enough. I also highly doubt it's anything MITM related. There is something weird going on, but it's spurious, so... [18:29:16] 10netops, 10Infrastructure-Foundations, 10SRE: Homer unable to commit config to cloudsw1-b1-codfw (QFX5120 21.4R3.16) - https://phabricator.wikimedia.org/T333316 (10cmooney) 05Open→03Resolved Applied to all devices now. [18:29:22] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10cmooney) [18:35:25] 10Traffic, 10DC-Ops, 10SRE, 10ops-codfw, 10Patch-For-Review: Q4:rack/setup/install lvs2011, lvs2012, lvs2013, lvs2014 - https://phabricator.wikimedia.org/T326767 (10Papaul) [19:29:24] 10Traffic, 10netops, 10DBA, 10Data-Platform-SRE, and 9 others: codfw row D switches upgrade - https://phabricator.wikimedia.org/T335042 (10bking) [19:31:38] 10Traffic, 10netops, 10DBA, 10Data-Platform-SRE, and 9 others: codfw row D switches upgrade - https://phabricator.wikimedia.org/T335042 (10bking) [19:33:03] 10Traffic, 10netops, 10DBA, 10Data-Platform-SRE, and 9 others: codfw row D switches upgrade - https://phabricator.wikimedia.org/T335042 (10bking) [20:38:53] 10Traffic, 10RESTBase-API, 10Documentation: I am hitting a rate limit on REST API endpoint - https://phabricator.wikimedia.org/T307610 (10BCornwall) Based on @BBlack's [[ #7918604 | previous comment ]], the 100/s is intentional. I would guess that running parallel requests during the 1000/10s would hit the l... [20:42:20] 10Traffic, 10RESTBase-API, 10Documentation: I am hitting a rate limit on REST API endpoint - https://phabricator.wikimedia.org/T307610 (10Mitar) Yes, great summary. [22:30:35] 10Traffic, 10DNS, 10SRE, 10WMF-Legal, 10wikimediafoundation.org: Update redirect for transparency.wikimedia.org - https://phabricator.wikimedia.org/T336301 (10Varnent) >>! In T336301#8838531, @Herald wrote: > As the #WMF-Legal project tag was added to this task, some general information to avoid wrong ex... [23:42:08] 10Traffic, 10DNS, 10SRE, 10WMF-Legal, 10wikimediafoundation.org: Update redirect for transparency.wikimedia.org - https://phabricator.wikimedia.org/T336301 (10Dzahn) Hi @Varnent this is also my team, like the ticket about redirects for annual.wikimedia.org. That is the #serviceops-collab tag nowadays. T...