[06:45:52] <wikibugs>	 10Traffic, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: OpenSSL < 1.1.0 compatibility issues with new LE issuance chain - https://phabricator.wikimedia.org/T283165 (10Joe)
[09:13:56] <jinxer-wm>	 (VarnishTrafficDrop) firing: 53% GET drop in text@ulsfo during the past 30 minutes  - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org
[09:18:56] <jinxer-wm>	 (VarnishTrafficDrop) resolved: 56% GET drop in text@ulsfo during the past 30 minutes  - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org
[09:23:56] <jinxer-wm>	 (VarnishTrafficDrop) firing: 55% GET drop in text@ulsfo during the past 30 minutes  - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org
[09:37:07] <wikibugs>	 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10ayounsi) @aborrero is it possible to have more information on this new service? Design doc or similar. I can't find anything on Wikite...
[09:40:27] <wikibugs>	 10netops, 10Infrastructure-Foundations: Rebuild Routinator (rpki) VMs with larger disk - https://phabricator.wikimedia.org/T292503 (10cmooney)
[09:43:03] <wikibugs>	 10netops, 10Infrastructure-Foundations: Document our OOB - https://phabricator.wikimedia.org/T292504 (10ayounsi) p:05Triage→03Medium
[09:43:57] <jinxer-wm>	 (VarnishTrafficDrop) resolved: 55% GET drop in text@ulsfo during the past 30 minutes  - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org
[09:52:00] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Rebuild Routinator (rpki) VMs with larger disk - https://phabricator.wikimedia.org/T292503 (10MoritzMuehlenhoff) https://packages.nlnetlabs.nl/ also provides the routinator debs for bullseye (plus it's a static Go binary anyway), so if we're recreating the VMs a...
[09:54:53] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Rebuild Routinator (rpki) VMs with larger disk - https://phabricator.wikimedia.org/T292503 (10cmooney) @MoritzMuehlenhoff yes smart thinking we'll do that :)
[09:55:49] <wikibugs>	 10Traffic: Investigate cp5006 crash - https://phabricator.wikimedia.org/T292506 (10ema)
[09:56:16] <wikibugs>	 10Traffic, 10SRE Observability: Investigate cp5006 crash - https://phabricator.wikimedia.org/T292506 (10ema)
[09:56:19] <wikibugs>	 10Traffic, 10SRE Observability: Investigate cp5006 crash - https://phabricator.wikimedia.org/T292506 (10ema) p:05Triage→03Medium
[09:57:18] <ema>	 elukey, vgutierrez: FYI I've opened a task about the recent crash of cp5006 ^
[09:59:02] <vgutierrez>	 ack
[10:00:46] <ema>	 it's interesting that rsyslog stopped functioning pretty much immediately judging from what ended up in /var/log/syslog and friends
[10:07:23] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Rebuild Routinator (rpki) VMs with larger disk - https://phabricator.wikimedia.org/T292503 (10cmooney) @ayounsi Riccardo suggested maybe using a separate disk/partition for the routinator data?  That was partly to just do a quick dirty job and not rebuild, but w...
[10:10:19] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Rebuild Routinator (rpki) VMs with larger disk - https://phabricator.wikimedia.org/T292503 (10cmooney) p:05Triage→03Low
[10:18:19] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Rebuild Routinator (rpki) VMs with larger disk - https://phabricator.wikimedia.org/T292503 (10MoritzMuehlenhoff) >>! In T292503#7401527, @cmooney wrote: > @ayounsi Riccardo suggested maybe using a separate disk/partition for the routinator data?  That was partly...
[10:19:18] <wikibugs>	 10Traffic, 10SRE, 10Patch-For-Review, 10Performance-Team (Radar): Test haproxy as a WMF's CDN TLS terminator with real traffic - https://phabricator.wikimedia.org/T290005 (10Vgutierrez)
[10:19:55] <wikibugs>	 10Acme-chief, 10Traffic, 10SRE, 10Patch-For-Review: Support OCSP stapling from prefetched responses in HAProxy - https://phabricator.wikimedia.org/T290249 (10Vgutierrez) 05Open→03Resolved a:03Vgutierrez
[11:10:13] <wikibugs>	 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE, and 2 others: Q1:(Need By: TBD) rack/setup/install cloudswift100[12] - https://phabricator.wikimedia.org/T289882 (10aborrero) >>! In T289882#7401435, @ayounsi wrote: > @aborrero is it possible to have more information on this new service? Design doc...
[12:23:16] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10Documentation: Document our OOB - https://phabricator.wikimedia.org/T292504 (10Aklapper)
[12:27:26] <wikibugs>	 10Traffic, 10Beta-Cluster-Infrastructure: Figure out why deployment-cache-text06 keeps crashing - https://phabricator.wikimedia.org/T286502 (10ema)
[12:53:20] <wikibugs>	 10Traffic, 10SRE: Package and deploy Varnish 6.0.8 - https://phabricator.wikimedia.org/T292290 (10ema) Preliminary testing in beta looks good, uploading the package to the archive.
[13:09:10] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE, 10Documentation: Document our OOB - https://phabricator.wikimedia.org/T292504 (10ayounsi) 05Open→03Resolved https://wikitech.wikimedia.org/wiki/OOB
[14:46:41] <wikibugs>	 10Traffic, 10Fundraising-Backlog, 10SRE, 10fr-donorservices, and 2 others: SSL cert for links.email.wikimedia.org - https://phabricator.wikimedia.org/T188561 (10Jgreen) >>! In T188561#7264108, @DStrine wrote: > @JBennett @BBlack @Dwisehaupt @Jgreen I'm hearing that the email service provider (now branded a...
[14:51:33] <wikibugs>	 10Traffic, 10DNS, 10SRE: Additional DNS entries for Wikilearn project (Community Development) - https://phabricator.wikimedia.org/T292537 (10Vgutierrez) 05Open→03In progress p:05Triage→03Medium
[14:56:44] <wikibugs>	 10Traffic, 10DNS, 10SRE: Additional DNS entries for Wikilearn project (Community Development) - https://phabricator.wikimedia.org/T292537 (10Vgutierrez) @Ijon could you confirm that you want forum.dev.learn.wiki. pointing to a private class C IP (192.168.193.13)?
[14:58:04] <wikibugs>	 10Traffic, 10DNS, 10SRE: Additional DNS entries for Wikilearn project (Community Development) - https://phabricator.wikimedia.org/T292537 (10Vgutierrez) 05In progress→03Stalled
[15:05:38] <bblack>	 https://os-reports.wikimedia.org/stretch.html <- we have 2x pybaltest instances on the bad list here (stretch)
[15:05:53] <bblack>	 I doubt they're testing anything useful in current form, as prod LVS has moved on to buster
[15:06:09] <bblack>	 should we reinstall/upgrade these? I'm not sure who was using them for what last
[15:07:01] <vgutierrez>	 I'm probably the last user of those
[15:07:13] <vgutierrez>	 yeah, we can reinstall them
[16:24:15] <wikibugs>	 10Traffic, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: OpenSSL < 1.1.0 compatibility issues with new LE issuance chain - https://phabricator.wikimedia.org/T283165 (10akosiaris) >>! In T283165#7365637, @MoritzMuehlenhoff wrote: > For production:  > * OpenSSL in Buster and Bullseye is not affect...
[16:30:07] <wikibugs>	 10Traffic, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: OpenSSL < 1.1.0 compatibility issues with new LE issuance chain - https://phabricator.wikimedia.org/T283165 (10akosiaris) >>! In T283165#7402880, @akosiaris wrote: >>>! In T283165#7365637, @MoritzMuehlenhoff wrote: >> For production:  >> *...
[16:31:00] <wikibugs>	 10Traffic, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: OpenSSL < 1.1.0 compatibility issues with new LE issuance chain - https://phabricator.wikimedia.org/T283165 (10MoritzMuehlenhoff) > With T291458 done, I 've already rebuilt bullseye (which was not affected) and buster main images (with lib...
[16:35:51] <wikibugs>	 10netops, 10Infrastructure-Foundations, 10SRE: Rebuild Routinator (rpki) VMs with larger disk - https://phabricator.wikimedia.org/T292503 (10MoritzMuehlenhoff) I've added routinator to apt.wikimedia.org at "thirdparty/routinator" for bullseye-wikimedia and adapted the Puppet code, so that when the these get...
[17:28:56] <jinxer-wm>	 (VarnishTrafficDrop) firing: 68% GET drop in text@ulsfo during the past 30 minutes  - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org
[17:33:56] <jinxer-wm>	 (VarnishTrafficDrop) resolved: 68% GET drop in text@ulsfo during the past 30 minutes  - https://grafana.wikimedia.org/d/000000180/varnish-http-requests?viewPanel=6 - https://alerts.wikimedia.org
[18:30:18] <wikibugs>	 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Krinkle)
[18:50:57] <wikibugs>	 10Traffic, 10DNS, 10SRE: Additional DNS entries for Wikilearn project (Community Development) - https://phabricator.wikimedia.org/T292537 (10Ijon) Oh, thanks for catching this silly mistake!  Indeed, the dev record should be to 52.44.207.59, not the private IP.