[00:01:59] 10Traffic, 10DNS, 10Fundraising-Backlog, 10Infrastructure-Foundations, and 3 others: Consider if to support BIMI for wiki mail - https://phabricator.wikimedia.org/T311685 (10BCornwall) p:05Medium→03Low [05:44:27] 10Traffic, 10DNS, 10SRE, 10Patch-For-Review, 10Software-Licensing: Add LICENSE to operations/dns scripts - https://phabricator.wikimedia.org/T291323 (10BCornwall) 05In progress→03Resolved [08:19:58] 10Varnish, 10Continuous-Integration-Infrastructure, 10SRE, 10Traffic-Icebox: Make CI run Varnish VCL tests - https://phabricator.wikimedia.org/T128188 (10hashar) The task is from 2016, and I most probably filed it as a placeholder to track effort to enhance test coverage on various repositories. We were on... [08:53:21] 10Traffic, 10SRE, 10Patch-For-Review, 10Sustainability (Incident Followup): Provide a cookbook to perform HAProxy upgrades on CDN nodes - https://phabricator.wikimedia.org/T330272 (10Vgutierrez) 05Open→03Resolved [12:41:16] 10Traffic, 10SRE: eqsin hosts are not rebooting when running sre.hosts.reimage cookbook - https://phabricator.wikimedia.org/T327812 (10RobH) This seems to have stalled with an inconclusive assumption that the NIC firmware update solved it, but we have no confirmation of that (see @ssingh's last comment above).... [17:06:34] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host dns4003.wikimedia.org with OS bullseye [17:11:55] 10Traffic, 10MediaWiki-File-management, 10SRE, 10Patch-For-Review, 10Technical-Debt: Remove IEContentAnalyzer - https://phabricator.wikimedia.org/T309787 (10BBlack) Looks good to me, and appropriate at the Varnish layer in this case. [17:50:01] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host dns4003.wikimedia.org with OS bullseye completed: - dns4003 (**PASS**) - Downtimed on Icinga/Al... [18:09:31] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ssingh) [18:11:08] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ssingh) [18:15:06] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by sukhe@cumin2002 for host dns5003.wikimedia.org with OS bullseye [18:21:09] 10HTTPS, 10SRE, 10Traffic-Icebox: Enable HSTS on store.wikimedia.org for HTTPS - https://phabricator.wikimedia.org/T128559 (10SHust) **Latest Shopify update after escalation -->** Thanks, Sandra, I have a good direction to go on this with my team. Looks like we do have a hard no on the "includeSubDomains;... [18:59:09] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ssingh) [19:21:03] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by sukhe@cumin2002 for host dns5003.wikimedia.org with OS bullseye completed: - dns5003 (**PASS**) - Downtimed on Icinga/Al... [19:21:58] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ssingh) [20:38:04] 10Traffic, 10MediaWiki-File-management, 10SRE, 10Patch-For-Review, 10Technical-Debt: Remove IEContentAnalyzer - https://phabricator.wikimedia.org/T309787 (10BCornwall) Carrying over a conversation from https://gerrit.wikimedia.org/r/c/802592 in which @tstarling says: > Ideally I would like there to be... [20:59:35] 10HTTPS, 10SRE, 10Traffic-Icebox: Enable HSTS on store.wikimedia.org for HTTPS - https://phabricator.wikimedia.org/T128559 (10BCornwall) What a mess. So if I'm understanding this correctly: * They refuse to include `includeSubDomains` unless it "makes sense" (what does that even mean?) ** Which means that p... [21:04:36] 10HTTPS, 10SRE, 10Traffic-Icebox: Enable HSTS on store.wikimedia.org for HTTPS - https://phabricator.wikimedia.org/T128559 (10BBlack) I assume "makes sense" here is probably cases where shopify knows of or has configured actual subdomains of the domain in question, or something like that. In either case, ye... [21:09:09] 10HTTPS, 10SRE, 10Traffic-Icebox: Enable HSTS on store.wikimedia.org for HTTPS - https://phabricator.wikimedia.org/T128559 (10BBlack) Maybe worth pointing out (I had an old stale link to this years ago earlier in the ticket), if nothing else because it may cause whomever at shopify to actually reach out to a... [21:12:20] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host dns5004.wikimedia.org with OS bullseye [21:13:31] 10HTTPS, 10SRE, 10Traffic-Icebox: Enable HSTS on store.wikimedia.org for HTTPS - https://phabricator.wikimedia.org/T128559 (10BCornwall) My interpretation of the response was that they also refuse to set the `preload` value. [21:14:50] 10Traffic, 10MediaWiki-File-management, 10SRE, 10Patch-For-Review, 10Technical-Debt: Remove IEContentAnalyzer - https://phabricator.wikimedia.org/T309787 (10TheDJ) > check for the header with curl during install, and warn the user if it is not present. I guess we can do this for upgrades, but for fresh... [21:29:06] 10Traffic, 10MediaWiki-File-management, 10SRE, 10Patch-For-Review, 10Technical-Debt: Remove IEContentAnalyzer - https://phabricator.wikimedia.org/T309787 (10TheDJ) Ugh.. ok. i see that the envCheckUploadsDirectory of the installer is not even checking wgUploadPath... and wgUploadBaseUrl either. [22:17:22] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host dns5004.wikimedia.org with OS bullseye completed: - dns5004 (**PASS**) - Downtimed on Icinga/Al... [22:28:02] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10BCornwall) [22:32:53] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host dns6002.wikimedia.org with OS bullseye [22:43:15] 10Traffic, 10MediaWiki-File-management, 10SRE, 10Patch-For-Review, 10Technical-Debt: Remove IEContentAnalyzer - https://phabricator.wikimedia.org/T309787 (10TheDJ) whipped up a beginning for an installer check, but requires more work. (i'm about to go on vacation, so might be a while before i can get bac... [23:20:04] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host dns6002.wikimedia.org with OS bullseye completed: - dns6002 (**PASS**) - Downtimed on Icinga/Al... [23:21:28] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10BCornwall)