[08:19:03] 10Traffic, 10SRE, 10envoy, 10serviceops, 10Sustainability (Incident Followup): Raw "upstream connect error or disconnect/reset before headers. reset reason: overflow" error message shown to users during outage - https://phabricator.wikimedia.org/T287983 (10ema) >>! In T287983#7261627, @Legoktm wrote: > I... [08:51:59] 10Traffic, 10SRE: Experiment with single backend CDN nodes - https://phabricator.wikimedia.org/T288106 (10ema) p:05Triage→03Medium [08:52:56] (VarnishTrafficDrop) firing: 67% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [08:57:56] (VarnishTrafficDrop) resolved: 68% GET drop in text@ during the past 30 minutes - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org [09:04:17] 10Traffic, 10SRE: Experiment with single backend CDN nodes - https://phabricator.wikimedia.org/T288106 (10ema) [09:06:19] 10Traffic, 10Maps, 10Product-Infrastructure-Team-Backlog, 10SRE, and 2 others: Support maps serving for affiliate sites via an allow list - https://phabricator.wikimedia.org/T261694 (10MSantos) @Legoktm from #product-infrastructure-team-backlog which are the official maintainers of maps, this looks great.... [10:00:06] 10netops, 10Infrastructure-Foundations, 10SRE: Adjust egress buffer allocations on ToR switches - https://phabricator.wikimedia.org/T284592 (10cmooney) [10:01:36] 10netops, 10Infrastructure-Foundations: Lumen eqiad-codfw link down - https://phabricator.wikimedia.org/T288218 (10ayounsi) p:05Triage→03High [10:01:46] 10netops, 10Infrastructure-Foundations: Lumen eqiad-codfw link down - https://phabricator.wikimedia.org/T288218 (10ayounsi) [10:02:32] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): Switch buffer re-partition - cloudsw1-d5-eqiad - https://phabricator.wikimedia.org/T288037 (10cmooney) [10:02:53] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): Switch buffer re-partition - cloudsw1-c8-eqiad - https://phabricator.wikimedia.org/T288036 (10cmooney) [10:30:00] 10netops, 10Infrastructure-Foundations, 10SRE: Adjust egress buffer allocations on ToR switches - https://phabricator.wikimedia.org/T284592 (10cmooney) [10:30:20] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): Switch buffer re-partition - cloudsw1-c8-eqiad - https://phabricator.wikimedia.org/T288036 (10cmooney) 05Open→03Resolved [10:55:45] OK if I merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/709511/ right now? adjusts upload vcl [11:10:30] legoktm: yup [11:19:20] 10Traffic, 10Maps, 10Product-Infrastructure-Team-Backlog, 10SRE, and 2 others: Support maps serving for affiliate sites via an allow list - https://phabricator.wikimedia.org/T261694 (10Legoktm) >>! In T261694#7262470, @MSantos wrote: > @Legoktm from #product-infrastructure-team-backlog which are the offici... [11:19:44] 10Traffic, 10Maps, 10Product-Infrastructure-Team-Backlog, 10SRE: Limit maps serving to Wikimedia hosted sites only - https://phabricator.wikimedia.org/T261424 (10Legoktm) [11:19:54] 10Traffic, 10Maps, 10Product-Infrastructure-Team-Backlog, 10SRE, and 2 others: Support maps serving for affiliate sites via an allow list - https://phabricator.wikimedia.org/T261694 (10Legoktm) 05Open→03Resolved [11:23:48] 10netops, 10Infrastructure-Foundations, 10SRE: Adjust egress buffer allocations on ToR switches - https://phabricator.wikimedia.org/T284592 (10cmooney) [11:24:22] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): Switch buffer re-partition - cloudsw1-d5-eqiad - https://phabricator.wikimedia.org/T288037 (10cmooney) 05Open→03Resolved [11:26:00] 10netops, 10Infrastructure-Foundations, 10SRE: Adjust egress buffer allocations on ToR switches - https://phabricator.wikimedia.org/T284592 (10cmooney) [13:49:42] 10netops, 10DC-Ops, 10SRE, 10ops-codfw, 10Wikimedia-Incident: asw-a2-codfw unresponsive - https://phabricator.wikimedia.org/T286787 (10Papaul) Your replacement part associated with RMA R200361905 Item # 100 has been successfully shipped. Details of which are provided below. Replacement Serial Number: R... [14:54:04] 10Traffic, 10DC-Ops, 10SRE, 10Sustainability (Incident Followup): Audit eqiad & codfw LVS network links - https://phabricator.wikimedia.org/T286881 (10ops-monitoring-bot) Icinga downtime set by vgutierrez@cumin1001 for 1:00:00 1 host(s) and their services with reason: T286881 ` lvs2008.codfw.wmnet ` [15:11:28] 10Traffic, 10DC-Ops, 10SRE, 10Sustainability (Incident Followup): Audit eqiad & codfw LVS network links - https://phabricator.wikimedia.org/T286881 (10ops-monitoring-bot) Icinga downtime set by vgutierrez@cumin1001 for 1:00:00 1 host(s) and their services with reason: T286881 ` lvs2009.codfw.wmnet ` [15:25:33] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE: (Need By: TBD) rack/setup/install atlas-codfw.wikimedia.org - https://phabricator.wikimedia.org/T273114 (10cmooney) I've moved netbox details (console and ethernet connection, IP addressing) from old device to the replacement device now, reflecting t... [15:30:03] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE: (Need By: TBD) rack/setup/install atlas-codfw.wikimedia.org - https://phabricator.wikimedia.org/T273114 (10cmooney) [15:48:52] 10Traffic, 10Maps, 10Product-Infrastructure-Team-Backlog, 10SRE, and 2 others: Support maps serving for affiliate sites via an allow list - https://phabricator.wikimedia.org/T261694 (10Elitre) Just noting that the newly made page was pretty much "orphan" - most of the docs re: Maps live on mw.org, so I wen... [16:45:44] 10Traffic, 10SRE, 10vm-requests, 10Patch-For-Review: Please create two Ganeti VMs for Wikidough in eqsin - https://phabricator.wikimedia.org/T284246 (10Dzahn) I deleted the reserved IP mentioned above and then could run the cookbook again. VM has been created now, has been added to DHCP and OS installed.... [16:47:01] 10Traffic, 10SRE, 10vm-requests, 10Patch-For-Review: Please create two Ganeti VMs for Wikidough in eqsin - https://phabricator.wikimedia.org/T284246 (10Dzahn) 05Open→03Resolved [16:50:02] 10Traffic, 10SRE, 10vm-requests, 10Patch-For-Review: Please create two Ganeti VMs for Wikidough in eqsin - https://phabricator.wikimedia.org/T284246 (10ssingh) Thanks very much for the help, @Dzahn! [16:50:47] 10Traffic, 10Fundraising-Backlog, 10SRE, 10fr-donorservices, and 2 others: SSL cert for links.email.wikimedia.org - https://phabricator.wikimedia.org/T188561 (10DStrine) @JBennett @BBlack @Dwisehaupt @Jgreen I'm hearing that the email service provider (now branded acoustic) is getting higher ratings. What...