[02:01:55] 06Traffic: varnish 7.1.1 crash - https://phabricator.wikimedia.org/T391334#10724399 (10BCornwall) Should we decide to go forward with downgrading, Varnish 6 and related packages have been imported into the bullseye-wikimedia component/varnish6 component. We'd perform the following: * re-introduce the Varnish s... [07:33:45] 06Traffic, 10Prod-Kubernetes, 06SRE, 10Wikidata, and 4 others: Frequent 500 Errors and Timeouts When Adding Statements to New Properties - https://phabricator.wikimedia.org/T374230#10724715 (10Ifrahkhanyaree_WMDE) [10:26:01] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: Link down between cr3-ulsfo and cr4-ulsfo - https://phabricator.wikimedia.org/T390731#10725338 (10ayounsi) For sure that's an odd one... Maybe we could try with a different port. For OSPF, +1 to do it for the troubleshooting window. D... [11:02:01] 06Traffic, 10Citoid, 06Editing QA, 06Editing-team, and 5 others: Switch from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10725430 (10Mvolz) [11:03:36] 06Traffic, 10Citoid, 06Editing QA, 06Editing-team, and 5 others: Switch from restbase to api gateway for Citoid - https://phabricator.wikimedia.org/T361576#10725432 (10Mvolz) >>! In T361576#10721517, @gerritbot wrote: > Change #1131008 **merged** by jenkins-bot: > %%%[mediawiki/extensions/Citoid@master] Us... [11:10:43] 06Traffic, 06Data-Persistence, 06SRE, 10SRE-swift-storage, and 6 others: Change default image thumbnail size - https://phabricator.wikimedia.org/T355914#10725471 (10Ladsgroup) >>! In T355914#10723774, @Jdforrester-WMF wrote: >>>! In T355914#10717142, @Ladsgroup wrote: >> It'd be nice to add this to next we... [11:10:44] 06Traffic, 13Patch-For-Review: varnish 7.1.1 crash - https://phabricator.wikimedia.org/T391334#10725473 (10Vgutierrez) FWIW varnish@cp3066 didn't report any thread creation failure before the crash: https://grafana.wikimedia.org/goto/K_XG9oANg?orgId=1 {F59024238} [13:02:04] 06Traffic, 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Unable to save Jupyter Notebooks or start IPython kernel on stat1008 - https://phabricator.wikimedia.org/T390959#10725934 (10Gehel) p:05Triage→03High [13:05:13] 06Traffic, 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Unable to save Jupyter Notebooks or start IPython kernel on stat1008 - https://phabricator.wikimedia.org/T390959#10725951 (10BTullis) a:03BTullis [13:30:46] 06Traffic, 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Unable to save Jupyter Notebooks or start IPython kernel on stat1008 - https://phabricator.wikimedia.org/T390959#10726067 (10BTullis) Hi @CDobbins - I'm starting to look into this for you, but I can't initially replicate the issue for myself. Also, I ca... [13:35:16] 06Traffic, 10Data-Platform-SRE (2025.03.22 - 2025.04.11): Unable to save Jupyter Notebooks or start IPython kernel on stat1008 - https://phabricator.wikimedia.org/T390959#10726093 (10BTullis) I think that the best thing to do first, seeing as you clearly have something running, would be for you to try to stop... [13:47:37] bblack vgutierrez I noted on Slack, but I got double booked for our slot, so I won't make it today. I _believe_ Sam can likely make it (he had some things to attend to, but I see he's back online today). [13:48:07] dr0ptp4kt: I've seen your slack message, thanks for reaching us also here :) [13:48:38] ty :) [13:56:56] +1 :) [14:32:34] 06Traffic, 06DC-Ops, 10ops-eqiad, 06SRE: Q3:test NIC for lvs1017 or lvs1018 - https://phabricator.wikimedia.org/T387145#10726404 (10Vgutierrez) >>! In T387145#10720903, @cmooney wrote: > (IMPORTANT) The obvious complication there is that lvs1016 has insufficient 10G ports to connect to everything that lvs1... [14:44:21] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: Decom eqiad row B <-> cloudsw links - https://phabricator.wikimedia.org/T391489 (10ayounsi) 03NEW [14:44:34] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad: Decom eqiad row B <-> cloudsw links - https://phabricator.wikimedia.org/T391489#10726454 (10ayounsi) [15:23:01] 06Traffic, 13Patch-For-Review: varnish 7.1.1 crash - https://phabricator.wikimedia.org/T391334#10726563 (10Vgutierrez) [15:27:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-codfw, 06SRE: Install and cable Nokia test devices and test servers in codfw - https://phabricator.wikimedia.org/T385217#10726607 (10cmooney) Hi @Jhancock.wm @papaul as discussed in our call if you could get an old Juniper QFX5100 switch racked in A... [16:46:43] 10netops, 06Infrastructure-Foundations, 10Observability-Alerting, 13Patch-For-Review: Migrate network icinga alerts to gNMI/prometheus - https://phabricator.wikimedia.org/T388641#10726881 (10cmooney) @ayounsi similar to Fillipos graph I put this dashboard together using the approach I had earlier, basicall... [19:08:15] 06Traffic, 10conftool, 10Hiddenparma: Requestctl needs to be able to check if a header is set, not just not set. - https://phabricator.wikimedia.org/T391368#10727410 (10Fabfur) >>! In T391368#10722559, @Volans wrote: > Wild suggestion, what if we merge both proposals? Make `header_value` to accept either a b... [19:12:55] FIRING: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [19:12:58] FIRING: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [19:17:55] RESOLVED: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent [19:17:58] RESOLVED: SLOMetricAbsent: - https://alerts.wikimedia.org/?q=alertname%3DSLOMetricAbsent