[01:59:22] 10Traffic, 10MediaWiki-File-management, 10Patch-For-Review, 10Technical-Debt: Remove IEContentAnalyzer - https://phabricator.wikimedia.org/T309787 (10Legoktm) [09:40:11] 10Traffic, 10netops, 10DBA, 10Data-Persistence, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10JMeybohm) Global depool of a/a services from codfw is done. [11:10:41] 10Traffic, 10netops, 10DBA, 10Data-Persistence, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10jbond) [11:16:48] 10Traffic, 10netops, 10DBA, 10Data-Persistence, and 10 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10jbond) [11:18:19] 10Traffic, 10netops, 10DBA, 10Data-Persistence, and 10 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10fgiunchedi) [13:41:37] 10Traffic, 10netops, 10DBA, 10Data-Persistence, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=aec8ddda-9ad5-4b7f-8bca-c273e036a282) set by ayounsi@cumin1001 for 2:00:00 on 215 host(s) and their serv... [13:48:30] 10Traffic, 10Data-Engineering-Planning, 10Observability-Alerting, 10SRE, 10Shared-Data-Infrastructure (Shared-Data-Infra Sprint 09): Reduce/eliminate false positives for VarnishKafkaNoMessages alert - https://phabricator.wikimedia.org/T324522 (10JArguello-WMF) 05Open→03Resolved [13:55:30] 10Traffic, 10netops, 10DBA, 10Data-Persistence, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10Vgutierrez) [14:37:22] 10Traffic, 10netops, 10DBA, 10Data-Persistence, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10ayounsi) Upgrade went smoothly, less than 15min hard downtime here too. [14:38:00] 10Traffic, 10netops, 10DBA, 10Data-Persistence, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10fgiunchedi) [14:45:23] 10Traffic, 10netops, 10DBA, 10Data-Persistence, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10jbond) [14:50:54] 10netops, 10Infrastructure-Foundations, 10SRE: eqiad/codfw virtual-chassis upgrades - https://phabricator.wikimedia.org/T327248 (10ayounsi) [14:51:30] 10Traffic, 10netops, 10DBA, 10Data-Persistence, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10fgiunchedi) [15:06:04] 10Traffic, 10netops, 10DBA, 10Data-Persistence, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10jcrespo) I restarted es5 codfw backup job, the only backup-related thingy affected by the downtime. [15:06:46] 10Traffic, 10Data-Engineering, 10Data-Persistence, 10Discovery-Search, and 7 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10ayounsi) p:05Triage→03Medium [15:07:23] 10Traffic, 10Data-Engineering, 10Data-Persistence, 10Discovery-Search, and 7 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10ayounsi) [15:07:35] 10netops, 10Infrastructure-Foundations, 10SRE: eqiad/codfw virtual-chassis upgrades - https://phabricator.wikimedia.org/T327248 (10ayounsi) [15:11:43] 10Traffic, 10Data-Engineering, 10Data-Persistence, 10Discovery-Search, and 7 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10jcrespo) [15:13:22] 10Traffic, 10Data-Engineering, 10Data-Persistence, 10Discovery-Search, and 7 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10MoritzMuehlenhoff) [15:48:25] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: codfw: Bad power supply on cr1-codfw(PEM 0) - https://phabricator.wikimedia.org/T329943 (10Papaul) 05Open→03Resolved Replaced PEM0 everything looks good now . {F36864090} [15:51:17] 10Traffic, 10netops, 10DBA, 10Data-Persistence, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10Jelto) [15:58:02] 10Traffic, 10SRE, 10IPv6: Start a pure IPv6 web site for wikimedia services - https://phabricator.wikimedia.org/T330020 (10BCornwall) a:03BCornwall [16:00:33] 10HTTPS, 10Traffic, 10Diff-blog, 10SRE, 10Technical Blog: Send HSTS header on all Wordpress VIP-hosted domains - https://phabricator.wikimedia.org/T270034 (10BCornwall) 05Open→03Resolved Perfect! Thanks so much for your magic hands and making this a reality, @Sbenchagra. [16:22:24] 10HTTPS, 10Traffic, 10Diff-blog, 10SRE, 10Technical Blog: Send HSTS header on all Wordpress VIP-hosted domains - https://phabricator.wikimedia.org/T270034 (10Sbenchagra) You are welcome! I am curious @BCornwall, why did it take more than two years for this task to be completed? [16:27:15] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: asw-a-codfw management interface unreachable - https://phabricator.wikimedia.org/T330048 (10Papaul) 05Open→03Resolved Rebooting the mgmt switch fix the issue [17:14:26] 10netops, 10Infrastructure-Foundations, 10Cloud-Services-Origin-Team, 10Cloud-Services-Worktype-Unplanned, and 2 others: [cloudvirt] Move to jumbo frames - https://phabricator.wikimedia.org/T330075 (10aborrero) [17:19:10] 10HTTPS, 10Traffic, 10Diff-blog, 10SRE, 10Technical Blog: Send HSTS header on all Wordpress VIP-hosted domains - https://phabricator.wikimedia.org/T270034 (10BCornwall) Good question. I fear I'm not equipped to give an authoritative answer, but generally low priority combined with ownership doubts (who o... [17:22:54] 10HTTPS, 10Traffic, 10Diff-blog, 10SRE, 10Technical Blog: Send HSTS header on all Wordpress VIP-hosted domains - https://phabricator.wikimedia.org/T270034 (10Sbenchagra) Thank you @BCornwall! Same, please flag any tickets that need my attention. Three months ago, I started managing the [[ https://wikimed... [17:59:49] 10Traffic, 10DNS, 10SRE, 10Chinese-Sites: Let all requests from mainland China will be processed to codfw/esams/drmrs - https://phabricator.wikimedia.org/T330024 (10BCornwall) 05Open→03Declined Hi, @I. Thank you for reporting and for your detailed descriptions. The team's limited capacity prevents the... [18:00:09] 10Traffic, 10SRE, 10IPv6: Start a pure IPv6 web site for wikimedia services - https://phabricator.wikimedia.org/T330020 (10BCornwall) 05Open→03Declined p:05Triage→03Lowest Hi, @I. Thank you for reporting and for your detailed descriptions. The team's limited capacity prevents the maintenance work re... [18:11:27] 10Traffic, 10MediaWiki-File-management, 10SRE, 10Patch-For-Review, 10Technical-Debt: Remove IEContentAnalyzer - https://phabricator.wikimedia.org/T309787 (10BCornwall) p:05Triage→03Lowest a:03BCornwall [18:26:12] 10Traffic, 10MediaWiki-File-management, 10SRE, 10Patch-For-Review, 10Technical-Debt: Remove IEContentAnalyzer - https://phabricator.wikimedia.org/T309787 (10BCornwall) a:05BCornwall→03Legoktm [18:26:28] 10Traffic, 10MediaWiki-File-management, 10SRE, 10Patch-For-Review, 10Technical-Debt: Remove IEContentAnalyzer - https://phabricator.wikimedia.org/T309787 (10BCornwall) 05Open→03In progress [19:01:27] 10Traffic, 10API Platform, 10MediaWiki-Core-HTTP-Cache, 10MediaWiki-REST-API, and 4 others: Determine http cache control and active purging for REST endpoints serving parsoid output - https://phabricator.wikimedia.org/T308424 (10JArguello-WMF) [19:40:09] 10Traffic, 10MediaWiki-File-management, 10SRE, 10Patch-For-Review, 10Technical-Debt: Remove IEContentAnalyzer - https://phabricator.wikimedia.org/T309787 (10BCornwall) @bblack, @Vgutierrez: Is it reasonable to put this header into Varnish itself as per https://gerrit.wikimedia.org/r/c/890512? Seems sound... [20:04:11] 10HTTPS, 10Traffic, 10Diff-blog, 10SRE, 10Technical Blog: Send HSTS header on all Wordpress VIP-hosted domains - https://phabricator.wikimedia.org/T270034 (10Dzahn) @Sbenchagra and @BCornwall Thank you soooo much for resolving this. It's great to see long-standing tickets closed. @Sbenchagra regarding w... [20:04:26] 10Traffic, 10netops, 10DBA, 10Data-Persistence, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10colewhite) [21:54:25] 10Traffic, 10SRE: Wikidough: Support EDNS(0) Padding: RFC 7830 and RFC 8467 - https://phabricator.wikimedia.org/T274431 (10Aklapper) [21:55:14] 10Traffic, 10SRE: Deploy Wikidough: Experimental DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT) public resolver - https://phabricator.wikimedia.org/T252132 (10Aklapper) [22:14:27] 10netops, 10Infrastructure-Foundations, 10SRE, 10IPv6, 10User-jbond: Fix IPv6 autoconf issues once and for all, across the fleet. - https://phabricator.wikimedia.org/T102099 (10Aklapper) [22:49:17] 10Traffic, 10Privacy Engineering, 10SRE: Remove obsolete "Permissions-Policy: interest-cohort" header - https://phabricator.wikimedia.org/T312823 (10BCornwall) 05In progress→03Resolved Thanks @ssingh for that followup patch ._. [23:14:44] 10Traffic, 10SRE, 10User-MoritzMuehlenhoff: Unexpected auditd service restart failure - https://phabricator.wikimedia.org/T287266 (10BCornwall) AFAICT we aren't packaging auditd ourselves. It might be easiest to just notify a trigger to re-start the stupid service after install since it looks like Debian isn... [23:15:22] 10Traffic, 10SRE, 10Patch-For-Review: haproxy: work on systemd unit hardening (cp hosts) - https://phabricator.wikimedia.org/T323944 (10BCornwall) 05Open→03Stalled