[06:16:35] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10Marostegui) [07:06:09] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10Marostegui) [07:30:12] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10Marostegui) [07:52:35] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866 (10Marostegui) db2105 is no longer a master. This host can be done after being depooled [07:52:45] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866 (10Marostegui) [07:53:33] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10Marostegui) >>! In T355862#9494195, @Marostegui wrote: > db2142 - x2 master > db2103 - s1 master > es2020 - es4 master The... [08:00:51] 10Traffic, 10SRE, 10SRE-swift-storage, 10MediaWiki-Platform-Team (Radar), 10Performance Issue: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Midleading) Is it possible to keep prerendered thumbnails indefinitely? I don't want to see search results full o... [08:04:35] 10Traffic, 10SRE, 10SRE-swift-storage, 10MediaWiki-Platform-Team (Radar), 10Performance Issue: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Ladsgroup) >>! In T211661#9497396, @Midleading wrote: > Is it possible to keep prerendered thumbnails indefinitel... [08:23:42] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10ayounsi) To keep on the radar but non-blocking : {T309724} [08:27:14] 10Traffic, 10SRE, 10SRE-swift-storage, 10MediaWiki-Platform-Team (Radar), 10Performance Issue: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Midleading) Yes, the thumbnails can be regenerated. But with a page load time of 10+ seconds... This is unaccepta... [08:36:39] 10Wikimedia-Apache-configuration, 10SRE, 10Security-Team, 10WMF-General-or-Unknown, and 3 others: Add security.txt to Wikimedia sites? (2023 edition) - https://phabricator.wikimedia.org/T337949 (10LSobanski) AS the original task was declined without comment, it would be helpful to understand what the input... [08:52:46] 10Traffic, 10Patch-For-Review: Restarting fifo-log-demux should not restart nginx - https://phabricator.wikimedia.org/T355905 (10Vgutierrez) IIRC that was done to smooth the reimage process and first puppet run on various roles using fifo-log-demux. nginx needs something (fifo-log-demux) consuming log message... [09:33:46] 10Wikimedia-Apache-configuration, 10SRE, 10Security-Team, 10WMF-General-or-Unknown, and 3 others: Add security.txt to Wikimedia sites? (2023 edition) - https://phabricator.wikimedia.org/T337949 (10MoritzMuehlenhoff) >>! In T337949#9497483, @LSobanski wrote: > AS the original task was declined without comme... [09:49:20] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866 (10klausman) [10:06:35] 10Wikimedia-Apache-configuration, 10SRE, 10Security-Team, 10WMF-General-or-Unknown, and 3 others: Add security.txt to Wikimedia sites? (2023 edition) - https://phabricator.wikimedia.org/T337949 (10ABran-WMF) p:05Triage→03Medium [10:50:00] 10Traffic, 10SRE, 10SRE-swift-storage, 10MediaWiki-Platform-Team (Radar), 10Performance Issue: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Ladsgroup) I understand, we won't delete everything all at once. It'll be at least a slow rolling deletion. I wan... [11:11:12] 10Traffic, 10SRE, 10SRE-swift-storage, 10MediaWiki-Platform-Team (Radar), 10Performance Issue: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10Midleading) Keep your words, don't delete everything at once. I can easily imagine a situation when I would re-vi... [11:13:32] 10netops, 10Data-Persistence, 10Data-Persistence-Backup, 10Infrastructure-Foundations, and 3 others: Migrate servers in codfw rack B4 from asw-b4-codfw to lsw1-b4-codfw - https://phabricator.wikimedia.org/T355860 (10MatthewVernon) [11:14:16] 10netops, 10Data-Persistence, 10Data-Persistence-Backup, 10Infrastructure-Foundations, and 3 others: Migrate servers in codfw rack B4 from asw-b4-codfw to lsw1-b4-codfw - https://phabricator.wikimedia.org/T355860 (10MatthewVernon) [I'll want to check afterwards that the ms-be nodes are happy, but this shou... [11:16:56] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A2 from asw-a2-codfw to lsw1-a2-codfw - https://phabricator.wikimedia.org/T355861 (10MatthewVernon) [11:19:08] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A2 from asw-a2-codfw to lsw1-a2-codfw - https://phabricator.wikimedia.org/T355861 (10MatthewVernon) swift will need depooling in codfw before this work. Likewise the affected thanos-fe node. I... [11:19:13] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10MatthewVernon) [11:21:22] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, and 2 others: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10MatthewVernon) Once complete, I'll want to check the ms-be nodes are all happy (shouldn't be an issue). [11:21:56] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A7 from asw-a7-codfw to lsw1-a7-codfw - https://phabricator.wikimedia.org/T355867 (10MatthewVernon) [11:22:20] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack A7 from asw-a7-codfw to lsw1-a7-codfw - https://phabricator.wikimedia.org/T355867 (10MatthewVernon) Once complete I'll want to check the backends, but this shouldn't be an issue. [11:23:24] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack B2 from asw-b2-codfw to lsw1-b2-codfw - https://phabricator.wikimedia.org/T355868 (10MatthewVernon) [11:25:06] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack B2 from asw-b2-codfw to lsw1-b2-codfw - https://phabricator.wikimedia.org/T355868 (10MatthewVernon) The affected thanos frontend will need depooling. Similarly, swift in codfw will need depooling. [11:26:35] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack B7 from asw-b7-codfw to lsw1-b7-codfw - https://phabricator.wikimedia.org/T355872 (10MatthewVernon) [11:27:37] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Migrate servers in codfw rack B7 from asw-b7-codfw to lsw1-b7-codfw - https://phabricator.wikimedia.org/T355872 (10MatthewVernon) I'll want to check the backends once this work is complete, but it shouldn't be an issue. [11:31:10] 10Traffic, 10SRE, 10SRE-swift-storage, 10MediaWiki-Platform-Team (Radar), 10Performance Issue: Automatically clean up unused thumbnails in Swift - https://phabricator.wikimedia.org/T211661 (10MatthewVernon) Thumbs that are being used get cached in the CDN in any case. [13:45:28] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by ayounsi@cumin1002 for hosts: `srestest2005.codfw.wmnet` - srestest2005.codfw.wmnet (**WARN**) - //Host... [20:40:32] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355833 (10jhathaway) After reading this post on [[ https://www.reddit.com/r/sysadmin/comments/q40ure/comment/hf... [21:25:45] does anyone know if you can specify a port in hieradata/common/profile/trafficserver/backend.yaml (not the "replacement" part?) ref: https://phabricator.wikimedia.org/T355617 , migrating a public service to private IPs [21:44:27] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355833 (10bcampbell) @jhathaway All good, thanks for the breakdown. I also CCd you in the support interaction wi... [21:57:21] nm, doing things a different way [21:58:12] sukhe just a heads-up, I'm revising the private IP migration per t-aavi suggestion and will no longer need service discovery...so you're off the hook on that code review ;P [22:50:32] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE, 10Patch-For-Review: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355833 (10jhathaway) @bcampbell patch is out for review