[07:58:56] 06Traffic, 06Data-Engineering, 10Observability-Logging, 10Event-Platform, 13Patch-For-Review: Remove extra fields currently sent to Kafka - https://phabricator.wikimedia.org/T360642#9653079 (10gmodena) [08:12:26] 06Traffic, 06Data-Engineering, 10Observability-Logging, 10Event-Platform, 13Patch-For-Review: Remove extra fields currently sent to Kafka - https://phabricator.wikimedia.org/T360642#9653099 (10gmodena) > These are the fields that are sent from Benthos that aren't present in the current webrequest stream:... [08:56:09] 06Traffic, 06Data-Engineering, 10Observability-Logging, 10Event-Platform, 13Patch-For-Review: Remove extra fields currently sent to Kafka - https://phabricator.wikimedia.org/T360642#9653275 (10Fabfur) >>! In T360642#9653099, @gmodena wrote: >> These are the fields that are sent from Benthos that aren't p... [11:21:54] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE, 10Release-Engineering-Team (Seen): Move 70% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T360763 (10Clement_Goubert) 03NEW [11:22:12] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#9653618 (10Clement_Goubert) [11:23:36] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE, 10Release-Engineering-Team (Seen): Move 70% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T360763#9653616 (10Clement_Goubert) 05Open→03In progress p:05Triage→03High [11:25:02] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE, 10Release-Engineering-Team (Seen): Move 70% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T360763#9653621 (10Clement_Goubert) Waiting on `codfw` repool as part of {T357547} before moving forward with this increase. [12:52:09] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536#9653629 (10Clement_Goubert) [12:57:11] 06Traffic: Return 403 to non HEAD|GET requests in HAProxy tls frontend - https://phabricator.wikimedia.org/T360766 (10Fabfur) 03NEW [12:57:15] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE, and 2 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120#9653700 (10Clement_Goubert) [12:58:48] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE, 10Release-Engineering-Team (Seen): Migrate changeprop to mw-api-int - https://phabricator.wikimedia.org/T360767 (10Clement_Goubert) 03NEW [12:59:08] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE, 10Release-Engineering-Team (Seen): Migrate changeprop to mw-api-int - https://phabricator.wikimedia.org/T360767#9653739 (10Clement_Goubert) p:05Triage→03High [12:59:08] hey folks juts a heads up the below task may be of interest [12:59:10] https://phabricator.wikimedia.org/T360772 [12:59:38] It relates to moving the BGP sessions from the dns hosts in codfw to the top-of-rack switches instead of the CRs [13:00:19] TL;DR I believe for now (with half of codfw migrated to L3 top-of-racks) it's not worth the complexity to support the hybrid setup [13:00:40] When we move rows C and D to routed top-of-racks we can revisit without additional complications [13:01:25] 06Traffic, 10Automoderator, 06Data Products, 06Product-Analytics, and 2 others: 14Add revision ID to X-Analytics header - 14https://phabricator.wikimedia.org/T346350#9653797 (10Samwalton9-WMF) 14@phuedx I wondered if you (or any other subscribers here) had any insight on how Flagged Revisions would im... [13:02:04] 06Traffic, 06Security-Team, 10WMF-General-or-Unknown, 07ContentSecurityPolicy, 13Patch-Needs-Improvement: Add restrictive CSP to upload.wikimedia.org - https://phabricator.wikimedia.org/T117618#9653824 (10TheDJ) [13:13:05] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE, and 2 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120#9653741 (10Clement_Goubert) [13:13:13] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE, and 2 others: Migrate internal traffic to k8s - https://phabricator.wikimedia.org/T333120#9653763 (10Clement_Goubert) [13:14:13] 06Traffic, 10MW-on-K8s, 06serviceops, 06SRE, 10Release-Engineering-Team (Seen): Move 70% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T360763#9653845 (10Clement_Goubert) Given we have increased `mw-web` and `mw-api-ext` by respectively 53 and 10 replicas to cope with ha... [13:15:41] 10netops, 06Infrastructure-Foundations, 06SRE: Move public-vlan host BGP peerings from CRs to top-of-rack switches in codfw - https://phabricator.wikimedia.org/T360772 (10cmooney) 03NEW p:05Triage→03Low [13:15:49] 10netops, 06Infrastructure-Foundations, 06SRE: Re-IP hosts on codfw row A and B to new per-rack vlans/subnets - https://phabricator.wikimedia.org/T354869#9653918 (10cmooney) [13:15:53] 10netops, 06Infrastructure-Foundations, 06SRE: Move public-vlan host BGP peerings from CRs to top-of-rack switches in codfw - https://phabricator.wikimedia.org/T360772#9653917 (10cmooney) [13:17:41] 10netops, 06Infrastructure-Foundations, 06SRE: Move public-vlan host BGP peerings from CRs to top-of-rack switches in codfw - https://phabricator.wikimedia.org/T360772#9653941 (10cmooney) [13:33:09] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Decom asw-b-codfw switch stack - https://phabricator.wikimedia.org/T360776 (10cmooney) 03NEW p:05Triage→03Medium [13:34:41] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Decom asw-b-codfw switch stack - https://phabricator.wikimedia.org/T360776#9654011 (10Papaul) @cmooney what works for you works for me as well [13:35:12] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Decom asw-b-codfw switch stack - https://phabricator.wikimedia.org/T360776#9654012 (10Papaul) [13:35:34] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Decom asw-b-codfw switch stack - https://phabricator.wikimedia.org/T360776#9654013 (10Papaul) [14:07:59] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE, 13Patch-For-Review: Decom asw-b-codfw switch stack - https://phabricator.wikimedia.org/T360776#9654083 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=79f10d11-133e-477b-be4d-b326d7e4bcf9) set by cmooney@cumin1002 for 4:00:00... [14:18:19] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE, 13Patch-For-Review: Decom asw-b-codfw switch stack - https://phabricator.wikimedia.org/T360776#9654094 (10cmooney) [14:57:00] 06Traffic, 06SRE, 10SRE-swift-storage, 10Thumbor: Cache thumbs in our caching infrastructure (e.g. ATS) - https://phabricator.wikimedia.org/T345334#9654167 (10MatthewVernon) One thing that was discussed at the SRE meeting in Warsaw was looking at turnilo data (which IIRC is the last 90 days' requests) to e... [15:16:28] 06Traffic, 06SRE, 10SRE-swift-storage, 10Thumbor: Cache thumbs in our caching infrastructure (e.g. ATS) - https://phabricator.wikimedia.org/T345334#9654275 (10Ladsgroup) So I looked at some numbers for February: ` ladsgroup@stat1005:~$ spark3-sql --master yarn --executor-memory 8G --executor-cores 4 --dri... [15:29:47] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789 (10RobH) 03NEW [15:30:19] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw: codfw row C/D upgrade racking task - https://phabricator.wikimedia.org/T360789#9654382 (10RobH) [16:02:12] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE, 13Patch-For-Review: Decom asw-b-codfw switch stack - https://phabricator.wikimedia.org/T360776#9654489 (10cmooney) [16:26:50] 10netops, 06Infrastructure-Foundations, 06SRE: 14Migrate IP gateway for public1-a-codfw to spine switches - 14https://phabricator.wikimedia.org/T351532#9654574 (10cmooney) 05Open→03Resolved [16:27:28] 10netops, 06Infrastructure-Foundations, 06SRE: 14Migrate IP gateway for private1-b-codfw to spine switches - 14https://phabricator.wikimedia.org/T351534#9654580 (10cmooney) 05Open→03Resolved [16:28:35] 10netops, 06Infrastructure-Foundations, 06SRE: Codfw row A/B top-of-rack switch refresh - https://phabricator.wikimedia.org/T327938#9654586 (10cmooney) [16:28:48] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: 14Bring codfw row A-B EVPN switches live and make them gateway for existing Vlans - 14https://phabricator.wikimedia.org/T347191#9654584 (10cmooney) 05Open→03Resolved 14Closing this task, everything now completed. For future rows we can b... [16:28:59] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: 14Upgrade new codfw switches to Juniper recommended - 14https://phabricator.wikimedia.org/T341670#9654588 (10cmooney) [16:29:15] 10netops, 06Infrastructure-Foundations, 06SRE, 10SRE-tools, 13Patch-For-Review: Setup zero touch provisioning (ZTP) for network devices - https://phabricator.wikimedia.org/T336485#9654587 (10cmooney) [16:40:30] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Connect two hosts in codfw row A/B for switch migration testing - https://phabricator.wikimedia.org/T345803#9654614 (10cmooney) >>! In T345803#9479281, @Papaul wrote: > @cmooney can we get those 2 hosts back in decom? Thanks @papaul I'm done wit... [16:41:53] 10netops, 06Infrastructure-Foundations, 06SRE: 14Codfw row A/B top-of-rack switch refresh - 14https://phabricator.wikimedia.org/T327938#9654617 (10cmooney) 05Open→03Resolved a:03cmooney 14Closing this one, I've made some notes on wikitech below about how to approach these for future rows. https:/... [16:43:22] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Connect two hosts in codfw row A/B for switch migration testing - https://phabricator.wikimedia.org/T345803#9654622 (10cmooney) [17:09:27] 06Traffic, 06SRE, 10SRE-swift-storage, 10Thumbor: Cache thumbs in our caching infrastructure (e.g. ATS) - https://phabricator.wikimedia.org/T345334#9654752 (10Ladsgroup) So for "miss" (=swift/thumbor hits). The top hitter gets 750 in the whole month. Quickly it settles to ~130 a month. This results to any... [17:12:28] 10netops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Connect two hosts in codfw row A/B for switch migration testing - https://phabricator.wikimedia.org/T345803#9654809 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by cmooney@cumin1002 for hosts: `sretest2003.codfw.wmnet` - sretes... [18:52:29] 06Traffic, 10PageImages, 10WMF-General-or-Unknown, 07Regression: Miniature images from og:image not loading in social media links - https://phabricator.wikimedia.org/T359413#9655126 (10Jdlrobson) [19:41:57] 06Traffic, 06Data-Engineering, 10Observability-Logging, 10Event-Platform, 13Patch-For-Review: Remove extra fields currently sent to Kafka - https://phabricator.wikimedia.org/T360642#9655231 (10Ottomata) > meta.id and meta.request_id `meta.id` is used to uniquely identify an event, and it is usually used... [23:09:23] 06Traffic, 10Automoderator, 06Data Products, 06Product-Analytics, and 2 others: 14Add revision ID to X-Analytics header - 14https://phabricator.wikimedia.org/T346350#9655625 (10mpopov) 14Thank you so much for looking into it, @phuedx!!! So if I'm interpreting that table correctly, we can trust `rev_i...