[00:00:02] 10HTTPS, 10SRE, 10Traffic-Icebox: Enable HSTS on store.wikimedia.org for HTTPS - https://phabricator.wikimedia.org/T128559 (10SHust) Final Shopify reply below. Sorry everyone, I really tried... Thank you for your patience while I checked on this HSTS inquiry with my team. My team who manages this dug really... [07:09:52] 10Traffic: Add unique error IDs to 4xx responses - https://phabricator.wikimedia.org/T330973 (10ayounsi) [07:58:59] 10Traffic, 10Data-Engineering, 10Data-Persistence, 10Infrastructure-Foundations, and 8 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Marostegui) [08:28:15] 10Traffic, 10SRE: Add unique error IDs to 4xx responses - https://phabricator.wikimedia.org/T330973 (10Volans) Surely being able to distinguish them from the message would help, but still relies on the user to report the exact verbatim message they are getting, and rely on external information. Given that our... [09:16:08] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech: HTTP URIs do not resolve from NL and DE? - https://phabricator.wikimedia.org/T330906 (10Lydia_Pintscher) [09:37:14] 10netops, 10Infrastructure-Foundations, 10SRE: cr2-esams:FPC0 Parity error - https://phabricator.wikimedia.org/T318783 (10ayounsi) 05Resolved→03Open The issue is back: > 2023-01-30 12:36:42 UTC Minor FPC 0 Minor Errors we need to follow up with JTAC for a replacement. [09:37:35] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech: HTTP URIs do not resolve from NL and DE? - https://phabricator.wikimedia.org/T330906 (10Vgutierrez) >>! In T330906#8657917, @Ennomeijers wrote: > Thanks for the replies! Advising to use HTTPS over HTTP makes sense. > > But not supporting redirection from HTTP to... [09:48:26] FYI mr1-ulsfo is currently unreachable, hence I suggest to avoid any action on ulsfo that might require mgmt access to debug/fix it ;) [and reimages won't work] [09:49:26] 10Traffic, 10WMF-JobQueue, 10Regression, 10Wikimedia-production-error: Image thumbnails aren't served correctly after image reupload - https://phabricator.wikimedia.org/T330942 (10Joe) This is an UBN issue. Adding traffic as this is either an issue with jobs not being enqueued to regenerate the thumbnail,... [09:50:24] 10Traffic, 10WMF-JobQueue, 10Regression, 10Wikimedia-production-error: Image thumbnails aren't served correctly after image reupload - https://phabricator.wikimedia.org/T330942 (10jcrespo) New files thumbnails work as intended: https://test.wikipedia.org/wiki/File:Testing_upload_2.jpeg [11:02:04] 10Traffic, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Marostegui) @ayounsi @akosiaris @Joe to confirm, we are going to depool eqiad before this maintenance like we've done in codfw right? [11:02:20] 10Traffic, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Marostegui) [11:59:23] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech: HTTP URIs do not resolve from NL and DE? - https://phabricator.wikimedia.org/T330906 (10Ennomeijers) I think this touches upon a fundamental question of how to model WD information as Linked Data. As currently stated in the [[ URL | Data Access article ]] the //co... [12:12:42] 10Traffic, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10akosiaris) >>! In T330165#8660042, @Marostegui wrote: > @ayounsi @akosiaris @Joe to confirm, we are going to depool eqiad before this maintenance lik... [12:12:55] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech: HTTP URIs do not resolve from NL and DE? - https://phabricator.wikimedia.org/T330906 (10Vgutierrez) Probably HSTS is only being implemented by browsers. There is any particular reason to target the HTTP version or it could be bumped to HTTPS? Considering that we d... [13:15:11] 10Traffic, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10BTullis) [13:18:38] 10Traffic, 10DBA, 10Data-Engineering, 10Infrastructure-Foundations, and 8 others: eqiad row B switches upgrade - https://phabricator.wikimedia.org/T330165 (10Marostegui) >>! In T330165#8660202, @akosiaris wrote: >>>! In T330165#8660042, @Marostegui wrote: >> @ayounsi @akosiaris @Joe to confirm, we are goin... [14:34:21] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech: HTTP URIs do not resolve from NL and DE? - https://phabricator.wikimedia.org/T330906 (10akosiaris) curl also implements HSTS. See https://curl.se/docs/hsts.html, but it is indeed primarily a mechanism to protect users of browsers. @Ennomeijers you are right abou... [14:44:50] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech: HTTP URIs do not resolve from NL and DE? - https://phabricator.wikimedia.org/T330906 (10BBlack) >>! In T330906#8657917, @Ennomeijers wrote: > Thanks for the replies! Advising to use HTTPS over HTTP makes sense. > > But not supporting redirection from HTTP to HTT... [14:54:42] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech: HTTP URIs do not resolve from NL and DE? - https://phabricator.wikimedia.org/T330906 (10Nikki) >>! In T330906#8659810, @Vgutierrez wrote: >>>! In T330906#8657917, @Ennomeijers wrote: >> But not supporting redirection from HTTP to HTTPS will in my opinion introduce... [15:07:29] 10Traffic, 10SRE, 10Wikidata, 10wdwb-tech: HTTP URIs do not resolve from NL and DE? - https://phabricator.wikimedia.org/T330906 (10Ennomeijers) 05Stalled→03Resolved a:03Ennomeijers As I already mentioned earlier, the SPARQL endpoint and the RDF serialized data all use the HTTP version as the canonica... [15:15:19] 10Traffic, 10SRE: Deprecating the dns::auth role and moving authdns[12]001 to dns[12]001. - https://phabricator.wikimedia.org/T330670 (10ssingh) >>! In T330670#8652102, @jbond wrote: > lgtm just some curiosity :) > >> After the above change, we will have three DNS boxes in the core DCs, with ns0 pointing to d... [15:16:47] 10Traffic, 10SRE: Deprecating the dns::auth role and moving authdns[12]001 to dns[12]001. - https://phabricator.wikimedia.org/T330670 (10jbond) >>! In T330670#8661042, @ssingh wrote: > Yes, that was the eventual plan: to do ns0 over all three dns rec boxes and similarly for ns1, just like we are doing for ns2.... [15:20:21] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (FY2022/2023-Q3): Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10aborrero) This ticket had little activity in the last month. Did something happen offline that wasn't r... [15:25:46] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (FY2022/2023-Q3): Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10cmooney) @aborrero I've been getting the cloudsw configured in the background, which is nearly done. M... [17:07:41] 10netops, 10Infrastructure-Foundations, 10SRE: Plan codfw row A/B top-of-rack switch refresh - https://phabricator.wikimedia.org/T327938 (10cmooney) @papaul in terms of the cables we will need to begin as follows. I'm assuming here we go with [[ https://www.fs.com/de-en/products/71644.html?attribute=675&id=... [17:07:59] 10Traffic, 10SRE: Add unique error IDs to 4xx responses - https://phabricator.wikimedia.org/T330973 (10RLazarus) Adding @CDanis as we were just talking about something along these lines. [18:13:45] 10HTTPS, 10SRE, 10Traffic-Icebox: Enable HSTS on store.wikimedia.org for HTTPS - https://phabricator.wikimedia.org/T128559 (10BCornwall) Unless I'm mistaken, the only recourse we have at this point is to throw our "top-ten website" gut around and demand the improvements for the security of our users. I don't... [19:10:56] 10HTTPS, 10Traffic, 10SRE, 10Traffic-Icebox: Enable HSTS on store.wikimedia.org for HTTPS - https://phabricator.wikimedia.org/T128559 (10BBlack) Is there a reasonable shopify alternative that meets policy? That would be my question. If there isn't, we're stuck with this policy violation, but shouldn't st... [19:24:19] 10HTTPS, 10Traffic, 10SRE, 10Traffic-Icebox: Enable HSTS on store.wikimedia.org for HTTPS - https://phabricator.wikimedia.org/T128559 (10BCornwall) There's [[ https://woocommerce.com/ | WooCommerce ]] which could be created alongside all other WordPress installations (VIP has unlimited sites, right?) with... [20:04:31] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host dns1001.wikimedia.org with OS bullseye [20:24:07] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by brett@cumin2002 for host dns2001.wikimedia.org with OS bullseye [20:43:40] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host dns1001.wikimedia.org with OS bullseye completed: - dns1001 (**PASS**) - Downtimed on Icinga/Al... [21:04:15] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cumin2002 for host dns2001.wikimedia.org with OS bullseye completed: - dns2001 (**PASS**) - Downtimed on Icinga/Al... [21:07:48] 10Traffic, 10SRE, 10Patch-For-Review: Upgrade Traffic hosts to bullseye - https://phabricator.wikimedia.org/T321309 (10BCornwall) [21:11:16] 10Traffic, 10SRE, 10SRE-OnFire, 10Sustainability (Incident Followup): (Re) evaluate effectiveness / usefulness of varnish/haproxy traffic drop alerts - https://phabricator.wikimedia.org/T310608 (10BCornwall) I know that I ignore them. Perhaps rather than removing them entirely, we could tweak the detection... [23:31:48] 10HTTPS, 10Traffic, 10SRE, 10Traffic-Icebox: Enable HSTS on store.wikimedia.org for HTTPS - https://phabricator.wikimedia.org/T128559 (10Dzahn) Maybe it would make shopify reconsider if you merely mention that you _might_ consider using an alternative, combined with pointing out that it's "top 10 website".