[07:14:17] 10netops, 10Infrastructure-Foundations, 10SRE: eqiad/codfw virtual-chassis upgrades - https://phabricator.wikimedia.org/T327248 (10ayounsi) [07:14:28] 10Traffic, 10netops, 10DBA, 10Data-Persistence, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10ayounsi) 05Open→03Resolved a:03ayounsi [07:43:33] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox: Netbox Juniper report - https://phabricator.wikimedia.org/T306238 (10ayounsi) From T311999#8594766 I see that there is progress! Yay! @jbond / @MoritzMuehlenhoff In Juniper's form the only information requested when selecting OIDC is `ID token (Ope... [09:52:30] 10Traffic, 10SRE, 10User-MoritzMuehlenhoff: Unexpected auditd service restart failure - https://phabricator.wikimedia.org/T287266 (10MoritzMuehlenhoff) Per the bug that should be fixed in the auditd package in Bullseye, we'll be able to confirm when we reimage the doh* servers to Bullseye. [09:54:13] 10Traffic: varnish-frontend-fetcherr sets incorrect level in logstash - https://phabricator.wikimedia.org/T330267 (10TheDJ) [10:00:14] 10Traffic: varnish-frontend-fetcherr sets incorrect level in logstash - https://phabricator.wikimedia.org/T330267 (10Vgutierrez) p:05Triage→03Medium [10:12:04] 10Traffic: varnish-frontend-fetcherr sets incorrect level in logstash - https://phabricator.wikimedia.org/T330267 (10Vgutierrez) @fgiunchedi I could use your help here, as reported, level is being populated with `INFO, %{SEVERITY_LABEL}` but it doesn't seem to be coming from varnish-frontend-fetcherr itself but... [10:12:38] godog: when you have the chance: https://phabricator.wikimedia.org/T330267#8636341 [10:14:31] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox: Netbox Juniper report - https://phabricator.wikimedia.org/T306238 (10SLyngshede-WMF) @ayounsi doesn't it need an URL as well, for the endpoint? [10:21:49] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox: Netbox Juniper report - https://phabricator.wikimedia.org/T306238 (10ayounsi) For the record: some doc on {F36864730} as well as https://jnprprod.devportal-aw-us.webmethods.io/portal/apis [10:24:32] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox: Netbox Juniper report - https://phabricator.wikimedia.org/T306238 (10ayounsi) >>! In T306238#8636356, @SLyngshede-WMF wrote: > @ayounsi doesn't it need an URL as well, for the endpoint? I guess they will give it to us later on in the onboarding proc... [10:30:53] 10Traffic, 10SRE, 10observability: varnish-frontend-fetcherr sets incorrect level in logstash - https://phabricator.wikimedia.org/T330267 (10Vgutierrez) [10:38:19] 10Traffic: Provide a cookbook to perform HAProxy upgrades on CDN nodes - https://phabricator.wikimedia.org/T330272 (10Vgutierrez) [10:38:37] 10Traffic: Provide a cookbook to perform HAProxy upgrades on CDN nodes - https://phabricator.wikimedia.org/T330272 (10Vgutierrez) p:05Triage→03Medium [10:43:50] 10Traffic, 10Sustainability (Incident Followup): Provide a cookbook to perform HAProxy upgrades on CDN nodes - https://phabricator.wikimedia.org/T330272 (10jcrespo) [10:48:47] 10Traffic, 10MediaWiki-File-management, 10SRE, 10Patch-For-Review, 10Technical-Debt: Remove IEContentAnalyzer - https://phabricator.wikimedia.org/T309787 (10Vgutierrez) I think so, I've took the liberty of amending the commit and adding a test for the new header as well [11:13:51] 10netops, 10Infrastructure-Foundations, 10Cloud-Services-Origin-Team, 10Cloud-Services-Worktype-Unplanned, and 2 others: [cloudvirt] Move to jumbo frames - https://phabricator.wikimedia.org/T330075 (10ayounsi) >>! In T330075#8629532, @aborrero wrote: > Questions for NetOps: they live in the cloud-hosts vla... [11:18:14] 10netops, 10Infrastructure-Foundations, 10Cloud-Services-Origin-Team, 10Cloud-Services-Worktype-Unplanned, and 2 others: [cloudvirt] Move to jumbo frames - https://phabricator.wikimedia.org/T330075 (10aborrero) Ok, thanks! Next question would be: Because {T319184} some cloudvirts have a single NIC trunk... [11:25:18] 10netops, 10Infrastructure-Foundations, 10Cloud-Services-Origin-Team, 10Cloud-Services-Worktype-Unplanned, and 2 others: [cloudvirt] Move to jumbo frames - https://phabricator.wikimedia.org/T330075 (10ayounsi) Switch side is already at 9192. Server TCP stack will define the MSS based on the interface used... [11:32:36] vgutierrez: ok! I'll take a look later [11:32:43] thanks [11:38:35] (PurgedHighEventLag) firing: High event process lag with purged on cp5026:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=eqsin%20prometheus/ops&var-instance=cp5026 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [11:43:35] (PurgedHighEventLag) resolved: High event process lag with purged on cp5026:2112 - https://wikitech.wikimedia.org/wiki/Purged#Alerts - https://grafana.wikimedia.org/d/RvscY1CZk/purged?var-datasource=eqsin%20prometheus/ops&var-instance=cp5026 - https://alerts.wikimedia.org/?q=alertname%3DPurgedHighEventLag [12:29:54] 10netops, 10Infrastructure-Foundations, 10SRE, 10netbox: Netbox Juniper report - https://phabricator.wikimedia.org/T306238 (10jbond) @ayounsi i have took another look at this. from the steps in the document above i have now configured * Register the Juniper API gateway app in the Customer/Partner's IdP.... [12:51:18] 10Traffic, 10Observability-Logging, 10SRE: varnish-frontend-fetcherr sets incorrect level in logstash - https://phabricator.wikimedia.org/T330267 (10fgiunchedi) I took a quick look at this and found the following: * the logger program seems to be `modules/varnish/files/varnishfetcherr.py` ran by `modules/va... [13:04:07] 10netops, 10Infrastructure-Foundations, 10SRE, 10IPv6, 10User-jbond: Fix IPv6 autoconf issues once and for all, across the fleet. - https://phabricator.wikimedia.org/T102099 (10jbond) a:05jbond→03None [14:38:51] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host dns4004.wikimedia.org with OS bullseye [16:42:57] could someone take a look https://gerrit.wikimedia.org/r/c/operations/dns/+/890908 ? [16:52:14] 10Traffic, 10SRE, 10User-MoritzMuehlenhoff: Unexpected auditd service restart failure - https://phabricator.wikimedia.org/T287266 (10BCornwall) Ah, my bad, I thought this *was* affecting bullseye. Oops. Sounds good then. [17:04:22] zabe: looking [17:11:02] sukhe: linked a task [17:11:33] thanks [17:18:49] zabe: done [17:19:13] thanks :) [17:24:30] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host dns4004.wikimedia.org with OS bullseye executed with errors: - dns4004 (**FAIL**) - Downtimed o... [18:05:20] 10Traffic, 10SRE: Let HAProxy handle port 80 - https://phabricator.wikimedia.org/T323557 (10BCornwall) 05Open→03Stalled [18:06:35] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10BCornwall) 05Open→03In progress [18:35:10] 10Traffic, 10DNS, 10SRE, 10Patch-For-Review, 10Software-Licensing: Add LICENSE to operations/dns scripts - https://phabricator.wikimedia.org/T291323 (10Ottomata) Also fine with my work being licensed at Apache 2.0. Thank you! [18:49:55] 10HTTPS, 10SRE, 10Traffic-Icebox: Enable HSTS on store.wikimedia.org for HTTPS - https://phabricator.wikimedia.org/T128559 (10SHust) Sharing Shopify's latest update below. If anyone has any ideas, please send them my way since I still have no clue what to do! I made a few tests and found the issue. The subd... [20:58:04] 10HTTPS, 10SRE, 10Traffic-Icebox: Enable HSTS on store.wikimedia.org for HTTPS - https://phabricator.wikimedia.org/T128559 (10BCornwall) Hi, @SHust. We appear to be running in circles here! What we're after has nothing to do with DNS/domain names/CNAME/A records, etc. This is entirely about adjusting a secur... [21:15:12] 10HTTPS, 10SRE, 10Traffic-Icebox: Enable HSTS on store.wikimedia.org for HTTPS - https://phabricator.wikimedia.org/T128559 (10Dzahn) >>! In T128559#8638381, @SHust wrote: > Sharing Shopify's latest update below. If anyone has any ideas, please send them my way since I still have no clue what to do! Hi, tha... [21:17:27] 10HTTPS, 10SRE, 10Traffic-Icebox: Enable HSTS on store.wikimedia.org for HTTPS - https://phabricator.wikimedia.org/T128559 (10Dzahn) P.S. Yea, just listen to what @BCornwall said above. That is going to make it less confusing. And thanks for doing this! [22:25:23] 10Traffic, 10SRE: create a puppetized abstraction for haproxy blocklist hysteresis - https://phabricator.wikimedia.org/T329331 (10BCornwall) p:05Triage→03Low [22:27:30] 10Traffic, 10Analytics-Radar, 10Data-Engineering-Icebox, 10SRE: Requests to (hard) redirect pages return their target's contents but are counted as pageviews to the redirect page - https://phabricator.wikimedia.org/T125015 (10BCornwall) p:05Medium→03Triage [22:29:26] 10Traffic, 10SRE, 10SRE-OnFire (FY2021/2022-Q4): ncredir redirects for status.wiki* --> status.wikimedia.org - https://phabricator.wikimedia.org/T318804 (10BCornwall) p:05Triage→03Low [22:56:08] 10Traffic, 10SRE, 10SRE-OnFire (FY2021/2022-Q4): ncredir redirects for status.wiki* --> status.wikimedia.org - https://phabricator.wikimedia.org/T318804 (10BCornwall) Looks like `interpret_wildcard()` in [[ https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/media... [22:57:34] 10Traffic, 10Data-Engineering, 10Data-Persistence, 10Discovery-Search, and 8 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10colewhite) [23:27:13] 10Traffic, 10SRE, 10SRE-OnFire (FY2021/2022-Q4): ncredir redirects for status.wiki* --> status.wikimedia.org - https://phabricator.wikimedia.org/T318804 (10BCornwall) Looking into it further, it seems this is a very possible change! nginx mappings/site names support wildcards. Pulling back a bit, does anyth... [23:59:08] 10Traffic, 10Beta-Cluster-Infrastructure, 10SRE: Rename deployment-cache-(text|upload)0x to deployment-cp0x - https://phabricator.wikimedia.org/T280393 (10BCornwall) cp hosts have now been updated to bullseye, FYI