[08:37:12] 10Traffic, 10DC-Ops, 10SRE, 10decommission-hardware, 10ops-ulsfo: decommission bast4002.wikimedia.org - https://phabricator.wikimedia.org/T288579 (10MoritzMuehlenhoff) [08:37:35] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: (Need By: TBD) rack/setup/install ganeti4004 - https://phabricator.wikimedia.org/T289715 (10MoritzMuehlenhoff) 05Resolved→03Open The server doesn't have virtualisation enabled. I tried to enable it via the BIOS over the serial console, but I'm not getting a cons... [08:41:28] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: (Need By: TBD) rack/setup/install ganeti4004 - https://phabricator.wikimedia.org/T289715 (10MoritzMuehlenhoff) The server can be powered down any time, while it already has the ganeti role, it's not yet added to the cluster. [13:11:36] 10Traffic, 10SRE, 10observability, 10Upstream: flapping icinga Letsencrypt TLS cert alerts around renewal time - https://phabricator.wikimedia.org/T293826 (10RhinosF1) This just alerted again: > 14:09:04 <+icinga-wm> PROBLEM - mailman list info ssl expiry on lists1001 is CRITICAL: CRITICAL - Certificate li... [13:56:36] hi all! can someone please confirm that varnish will compress RESTbase responses on the fly, and in the process mess with the etags? [13:56:49] I'm trying to hunt down the source of "weak" etags in parsoid responses. They are breaking If-Match semantics... [13:57:57] The context is https://phabricator.wikimedia.org/T238849 - the workaround that led to that ticket being closed seems broken. [14:00:07] What I'm asking about is basically this: https://varnish-cache.org/docs/6.0/users-guide/compression.html [15:21:24] 10Traffic, 10DC-Ops, 10SRE, 10decommission-hardware, 10ops-ulsfo: decommission bast4002.wikimedia.org - https://phabricator.wikimedia.org/T288579 (10RobH) [15:21:48] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: (Need By: TBD) rack/setup/install ganeti4004 - https://phabricator.wikimedia.org/T289715 (10RobH) 05Open→03In progress >>! In T289715#8010871, @MoritzMuehlenhoff wrote: > The server doesn't have virtualisation enabled. I tried to enable it via the BIOS over the... [15:30:05] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: (Need By: TBD) rack/setup/install ganeti4004 - https://phabricator.wikimedia.org/T289715 (10RobH) 05In progress→03Open [15:30:08] 10Traffic, 10DC-Ops, 10SRE, 10decommission-hardware, 10ops-ulsfo: decommission bast4002.wikimedia.org - https://phabricator.wikimedia.org/T288579 (10RobH) [15:30:24] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: (Need By: TBD) rack/setup/install ganeti4004 - https://phabricator.wikimedia.org/T289715 (10RobH) a:05RobH→03MoritzMuehlenhoff virtualization is now enabled, yo ushould be able to push this into service as needed now [15:32:19] 10Traffic, 10DC-Ops, 10SRE, 10ops-ulsfo: (Need By: TBD) rack/setup/install ganeti4004 - https://phabricator.wikimedia.org/T289715 (10MoritzMuehlenhoff) Thanks! I'll do that on Tuesday [15:40:35] duesen: just a heads up, I think both v.gutierrez and b.black are OOO right now, and they're both the best at answering this question [16:06:50] cdanis: ok thanks, i'll come back later. It's not super urgent, but it is blocking the task of moving VE endpoints out of RESTbase. So it *will* be urgent some day :) [16:07:07] duesen: feel free to open a ticket with the question, that's probably best [16:07:26] on other days I would look myself but I'm in a last-minute crunch to finish some work before I go on vacation for two weeks :) [16:07:38] thanks, all goo [16:07:39] d [16:13:56] 10netops, 10Infrastructure-Foundations: Complete testing of SONiC NOS / Dell network gear and write up - https://phabricator.wikimedia.org/T310901 (10cmooney) p:05Triage→03Low [16:21:02] 10Traffic, 10API Platform, 10Parsoid, 10VisualEditor, 10Platform Team Workboards (MW Expedition): Find out if Varnish is messing with ETags, and what to do about it. - https://phabricator.wikimedia.org/T310904 (10daniel) [16:21:48] cdanis: https://phabricator.wikimedia.org/T310904 [16:21:50] 10Traffic, 10API Platform, 10Parsoid, 10VisualEditor, 10Platform Team Workboards (MW Expedition): Find out if Varnish is messing with ETags, and what to do about it. - https://phabricator.wikimedia.org/T310904 (10daniel) Pinging @BBlack and @Vgutierrez in the hope they can shed some light... [16:22:33] 👍 [16:22:51] duesen: vgutierrez: [16:23:03] er sorry, he will be back next week fwiw [16:23:20] (he has no internet right now so can safely ping:) [17:02:16] Re: Message-ID , Do we need to create tickets for any remaining Stretch VMs that are in danger of getting ripped out from under us? [17:03:26] brett: if you the mean the Traffic ones on cloud, given when they were last used, I think it is fine [17:03:45] or maybe we can email them to say we will get back to them once Valentin comes back next week (since they were being used by ema and valentin) [17:08:43] sukhe: Thanks for that. I'll email them and create a ticket for combing through the servers. Thanks! [17:11:11] thanks for following up on this [17:17:08] 10Traffic: Review Debian Buster VMs set for 2022-06 termination - https://phabricator.wikimedia.org/T310910 (10BCornwall) [17:50:22] 10Traffic, 10SRE: Review Debian Buster VMs set for 2022-06 termination - https://phabricator.wikimedia.org/T310910 (10RhinosF1) [17:52:46] 10Traffic, 10SRE: Review Debian Buster VMs set for 2022-06 termination - https://phabricator.wikimedia.org/T310910 (10RhinosF1) @BCornwall: See {T306101} which already exists and has some comments on regarding diff scan which has already been shut down via {T306245} and can likely just be deleted @ayounsi conf... [17:53:29] brett: see ^ for some more info [17:54:08] I'm not sure if that's best closed as a duplicate tbh [17:55:19] XioNoX: I pinged you there as you already have a diff scan replacement ^ [17:55:58] yeah, old one can be deleted [17:56:20] 10Traffic, 10SRE: Review Debian Buster VMs set for 2022-06 termination - https://phabricator.wikimedia.org/T310910 (10BCornwall) @RhinosF1: Ugh, sorry about that. My searches didn't manage to find those, so thanks for doing the dirty work for me... ._. [17:56:22] 10Traffic, 10SRE: Review Debian Buster VMs set for 2022-06 termination - https://phabricator.wikimedia.org/T310910 (10ayounsi) +1 to delete the old instance. [17:56:48] brett: no worries [17:56:50] 10Traffic, 10SRE: Review Debian Buster VMs set for 2022-06 termination - https://phabricator.wikimedia.org/T310910 (10BCornwall) 05Open→03Invalid [17:56:55] I know where a lot of stuff is [18:01:38] RhinosF1: Happy to re-open but it does seem those tickets pretty much covered everything I was thinking of already :/ [18:02:43] Especially since I confused Stretch and Buster [18:02:46] * brett facepalms [18:04:57] 10Traffic, 10SRE: Review Debian Stretch VMs set for 2022-06 termination - https://phabricator.wikimedia.org/T310910 (10BCornwall) [18:06:53] brett: it's friday :) [18:07:00] I think that's good now [18:08:11] Looks like you already have the powers to delete the diff scan VM [18:09:19] I'm checking twice before I cut :) [18:13:32] That's often sensible [18:30:47] 10Traffic, 10SRE, 10Patch-For-Review, 10SRE Observability (FY2021/2022-Q4), 10User-fgiunchedi: Migrate Traffic Prometheus alerts from Icinga to Alertmanager - https://phabricator.wikimedia.org/T300723 (10BCornwall) 05Open→03In progress [20:48:57] (HAProxyEdgeTrafficDrop) firing: 53% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [20:53:57] (HAProxyEdgeTrafficDrop) resolved: 69% request drop in text@eqsin during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://grafana.wikimedia.org/d/000000479/frontend-traffic?viewPanel=12&orgId=1&from=now-24h&to=now&var-site=eqsin&var-cache_type=text - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [21:30:37] 10Traffic, 10MediaWiki-General, 10SRE: Query canonicalization for MediaWiki - https://phabricator.wikimedia.org/T310087 (10Krinkle) Regarding parameter - One thing that comes to mind from a previous experiment long ago (I don't recall specifics and couldn't find any) is OAuth verification. OAuth is sensitiv... [22:53:56] (HAProxyEdgeTrafficDrop) firing: (2) 53% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop [22:58:56] (HAProxyEdgeTrafficDrop) resolved: (3) 67% request drop in text@drmrs during the past 30 minutes - https://wikitech.wikimedia.org/wiki/Monitoring/EdgeTrafficDrop - https://alerts.wikimedia.org/?q=alertname%3DHAProxyEdgeTrafficDrop