[08:19:18] 10Traffic, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10fgiunchedi) [08:22:28] 10Traffic, 10Cloud-Services, 10SRE, 10cloud-services-team: Horizon/lvs alerts the wrong people (and also is generally too sensitive) - https://phabricator.wikimedia.org/T331197 (10fgiunchedi) The easiest thing to do ATM I think is set `page: false` in `service::catalog` for the labweb service(s), this way... [08:45:57] 10Traffic, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10MoritzMuehlenhoff) [09:25:34] 10Traffic, 10SRE: haproxy: work on systemd unit hardening (cp hosts) - https://phabricator.wikimedia.org/T323944 (10Vgutierrez) I've disabled the systemd hardening after confirming issues in ulsfo: `counterexample vgutierrez@cp4041:~$ ps auxww |grep haproxy |wc -l 49 ` HAProxy is unable to terminate old proce... [09:42:38] 10Traffic, 10MW-on-K8s, 10Wikidata, 10serviceops, and 2 others: Migrate testwikidata to Kubernetes - https://phabricator.wikimedia.org/T331268 (10Clement_Goubert) [10:31:43] 10Traffic, 10SRE: haproxy: work on systemd unit hardening (cp hosts) - https://phabricator.wikimedia.org/T323944 (10Vgutierrez) @ssingh this could be as easy to fix as granting `CAP_KILL`, I'm currently testing that on cp4045 [12:01:09] 10Traffic, 10serviceops, 10Datacenter-Switchover: March 2023 Traffic Repool checklist - https://phabricator.wikimedia.org/T331285 (10Clement_Goubert) [12:01:58] 10Traffic, 10SRE, 10serviceops, 10Datacenter-Switchover: March 2023 Traffic Repool checklist - https://phabricator.wikimedia.org/T331285 (10Clement_Goubert) p:05Triage→03High [12:04:07] 10Traffic, 10SRE, 10serviceops, 10Datacenter-Switchover: March 2023 Traffic Repool checklist - https://phabricator.wikimedia.org/T331285 (10Clement_Goubert) [12:05:33] 10Traffic, 10SRE, 10serviceops, 10Datacenter-Switchover: March 2023 Traffic Repool checklist - https://phabricator.wikimedia.org/T331285 (10Clement_Goubert) [12:16:45] 10Traffic, 10SRE, 10serviceops, 10Datacenter-Switchover: March 2023 Traffic Repool checklist - https://phabricator.wikimedia.org/T331285 (10Clement_Goubert) [13:24:21] 10Traffic, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10BTullis) [14:34:41] 10Traffic, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10BTullis) [14:42:18] 10Traffic, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10MatthewVernon) [15:14:08] 10Traffic, 10SRE, 10serviceops, 10Datacenter-Switchover: March 2023 Traffic Repool checklist - https://phabricator.wikimedia.org/T331285 (10Clement_Goubert) [15:23:32] I would love it if someone can confirm that https://gerrit.wikimedia.org/r/c/operations/puppet/+/894664 will prevent a repeat of Saturday's unplanned fire drill. [15:37:07] 10Traffic, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Jelto) [15:50:12] I'm not sure we actually want that since iirc labweb hosts wikitech [17:21:58] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech: HTTP URIs do not resolve from NL and DE? - https://phabricator.wikimedia.org/T330906 (10BBlack) 05Resolved→03Open >>! In T330906#8661013, @Ennomeijers wrote: > As I already mentioned earlier, the SPARQL endpoint and the RDF serialized data all use the HTTP ver... [17:24:28] 10Traffic, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10herron) [17:34:22] 10Traffic, 10MW-on-K8s, 10serviceops: Insert a header for specific domains at the first ATS layer to redirect traffic to mw-on-k8s - https://phabricator.wikimedia.org/T331318 (10Clement_Goubert) [17:34:50] 10Traffic, 10MW-on-K8s, 10serviceops: Insert a header for specific domains at the first ATS layer to redirect traffic to mw-on-k8s - https://phabricator.wikimedia.org/T331318 (10Clement_Goubert) [17:35:02] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 3 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [17:35:29] 10Traffic, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10ssingh) [17:35:51] 10Traffic, 10MW-on-K8s, 10serviceops: Insert a header for specific domains at the first ATS layer to redirect traffic to mw-on-k8s - https://phabricator.wikimedia.org/T331318 (10Clement_Goubert) p:05Triage→03Medium [18:58:02] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech: HTTP URIs do not resolve from NL and DE? - https://phabricator.wikimedia.org/T330906 (10Ennomeijers) Ok, I see your point. As long as the concept/canonical URIs for all entities are being published as http:// URIs there is no other way than following the 301 redir... [19:19:59] 10HTTPS, 10Traffic, 10SRE, 10Traffic-Icebox: Enable HSTS on store.wikimedia.org for HTTPS - https://phabricator.wikimedia.org/T128559 (10BCornwall) I suspect we generate little revenue for them and I don't see any sort of "Businesses that rely on Shopify" section on their site (they seem to prefer showing... [20:02:11] 10Traffic, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10herron) [21:01:29] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech: HTTP URIs do not resolve from NL and DE? - https://phabricator.wikimedia.org/T330906 (10TheDJ) The problem is identifiers vs urls. An identifier is stable. A url might not be. If you start using locators as identifiers.... things become gray. Then again. The spec... [21:04:02] 10Traffic, 10SRE, 10Sustainability (Incident Followup): cp3050 seemd more affected then otheres in recent incident - https://phabricator.wikimedia.org/T330682 (10BCornwall) p:05Triage→03High [21:25:34] 10Traffic, 10Wikidata, 10wdwb-tech: Wikidata seems to still be utilizing insecure HTTP URIs - https://phabricator.wikimedia.org/T331356 (10BBlack) p:05Triage→03High [21:26:42] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech: HTTP URIs do not resolve from NL and DE? - https://phabricator.wikimedia.org/T330906 (10BBlack) 05Open→03Resolved The redirects are neither //good// nor //bad//, they're instead both necessary (although that necessity is waning) and insecure. We thought we ha... [22:53:04] 10Traffic, 10SRE: ATS: origins server response data accounting issues - https://phabricator.wikimedia.org/T284290 (10BCornwall) 05Open→03Invalid Considering that over two years this doesn't seem to have cropped up, I don't think it's worth keeping open unless this becomes a problem again. The grafana metri... [22:53:09] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Automate EVPN switch underlay BGP neighbor peerings - https://phabricator.wikimedia.org/T327934 (10cmooney) [23:11:47] 10Traffic, 10SRE: Drop the VarnishTrafficDrop and HAProxyEdgeTrafficDrop alerts - https://phabricator.wikimedia.org/T322220 (10BCornwall) 05Open→03Resolved a:03BCornwall [23:17:58] 10Traffic, 10Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10User-Dereckson: Create /community-beacon alternative entry point - https://phabricator.wikimedia.org/T155929 (10BCornwall) 05Open→03Declined I'm BOLDly closing this as I came to the same conclusions as @Pcoombe. [23:19:59] 10Traffic, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=786ee8c7-4753-4e2d-96f9-8b55b691ff09) set by bking@cumin2002 for 1 day, 0:00:00 on 1... [23:21:04] 10Traffic, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=f9f1bd07-4af1-41e3-82b7-3ab0f2ff8672) set by bking@cumin2002 for 1 day, 0:00:00 on 5... [23:22:29] 10Traffic, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10bking) [23:25:20] 10Traffic, 10DBA, 10Data-Engineering-Planning, 10Data-Persistence, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10RKemper) [23:44:54] 10Traffic, 10Thumbor: Thumbor URLs are too permissive - https://phabricator.wikimedia.org/T310528 (10BCornwall) p:05Medium→03Low [23:50:41] 10Traffic, 10DNS: Central and South American countries in geo-maps - https://phabricator.wikimedia.org/T301605 (10BCornwall) p:05Triage→03Low