[01:12:17] inflatador: ok! thanks :) [02:48:39] Hi traffic team, [02:48:39] I'm currently in the process of upgrading our Grafana instances and I have a few questions regarding trafficserver map rules. [02:48:39] As part of the upgrade, I'll need to perform a failover between our two Grafana instances. For this I created a patch to route user traffic to the new active host. https://gerrit.wikimedia.org/r/c/operations/puppet/+/992719 [02:48:40] My question is whether it's necessary to run Puppet on a traffic server after merging this patch for the changes to take effect or if it's okay to wait for Puppet run? [02:48:41] I'm considering the potential scenario where waiting for Puppet to run on those hosts might result in the catchall rule directing traffic to the previously passive host initially, as the first request wouldn't match any rules while the active host is temporarily down. [02:48:42] Thanks in advance. :) [06:13:42] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE: Migrate servers in codfw rack B3 from asw-b3-codfw to lsw1-b3-codfw - https://phabricator.wikimedia.org/T355870 (10Marostegui) Once T355862 is done, es2021 needs to be switched back to be es4 slave (reverting all this T356064) [06:18:18] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE: Migrate servers in codfw rack B3 from asw-b3-codfw to lsw1-b3-codfw - https://phabricator.wikimedia.org/T355870 (10Marostegui) [10:42:27] 10Traffic, 10Data-Persistence, 10SRE, 10SRE-swift-storage, 10Wikimedia-Site-requests: Changing default image thumbnail size on English Wikipedia - https://phabricator.wikimedia.org/T355914 (10Joe) Given the chosen size is both non-standard (meaning it's not used on most large wikis) and not in the list o... [11:25:22] 10Traffic, 10Data-Persistence, 10SRE, 10SRE-swift-storage, and 2 others: Changing default image thumbnail size on English Wikipedia - https://phabricator.wikimedia.org/T355914 (10MatthewVernon) [11:42:10] 10Traffic, 10Data-Persistence, 10SRE, 10SRE-swift-storage, and 2 others: Changing default image thumbnail size on English Wikipedia - https://phabricator.wikimedia.org/T355914 (10MatthewVernon) I think the main issue is likely that we'll melt Thumbor if we just switch enwiki to 250, because 250 isn't a pre... [12:11:59] 10Traffic, 10Data-Persistence, 10SRE, 10SRE-swift-storage, and 2 others: Changing default image thumbnail size on English Wikipedia - https://phabricator.wikimedia.org/T355914 (10TheDJ) We could have MW detect thumbnail transclusions with the default size, have a configuration setting for a 1/x ratio, then... [12:23:33] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, and 2 others: Move 40% of mediawiki external requests to mw on k8s - https://phabricator.wikimedia.org/T355532 (10Clement_Goubert) [13:19:58] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10ayounsi) Current status, ignoring IPv6 for now. The cluster VIP is dynamically announced from the primary cluster node. Limitation from `isc-dhcp... [14:08:17] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10Volans) Indeed the `dhcrelay` not working as expected is a bit annoying also because if we run a dhcrelay for each VM, we'd need to hook also at VM... [14:15:36] 10Traffic, 10Data-Persistence, 10SRE, 10SRE-swift-storage, and 2 others: Changing default image thumbnail size on English Wikipedia - https://phabricator.wikimedia.org/T355914 (10Ladsgroup) This is much messier and much hairier than it looks. And yes, without proper preparation this will bring down everyth... [15:26:59] 10Traffic, 10Data-Persistence, 10SRE, 10SRE-swift-storage, and 2 others: Changing default image thumbnail size on English Wikipedia - https://phabricator.wikimedia.org/T355914 (10TheDJ) > Once that's done, I suggest getting rid of thumbsizes user preferences altogether and making the thumbsize wiki-wide in... [15:41:24] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by ayounsi@cumin2002 for hosts: `testvm2006.codfw.wmnet` - testvm2006.codfw.wmnet (**... [16:39:56] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Jdforrester-WMF) [16:53:13] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Jdforrester-WMF) [17:12:39] 10Traffic, 10Upstream: HAProxy 2.6.16/2.8.5 CPU spikes on cp3066 - https://phabricator.wikimedia.org/T354424 (10Vgutierrez) Fix already released on HAProxy 2.9: https://www.mail-archive.com/haproxy@formilux.org/msg44547.html [17:33:21] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10ayounsi) For the latter, some more debug: I added ` shared-network "test" { subnet 10.192.24.1 netmask 255.255.255.255 { opti... [17:33:57] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE, 10Patch-For-Review: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355833 (10jhathaway) @bcampbell patch is merged, if you want to give it a whirl! [17:47:52] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE, 10Patch-For-Review: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355833 (10bcampbell) Thanks @jhathaway . I just clicked the button on the Shopify admin co... [17:56:01] 10Traffic, 10Data-Persistence, 10SRE, 10SRE-swift-storage, and 2 others: Changing default image thumbnail size on English Wikipedia - https://phabricator.wikimedia.org/T355914 (10Ganesha811) Is there a pre-generated size around 250px that would be easier to switch to? We can attempt to develop a new consen...