[08:49:20] 10netops, 10Data-Persistence, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B5 from asw-b5-codfw to lsw1-b5-codfw - https://phabricator.wikimedia.org/T355549 (10Marostegui) [11:42:14] 10Traffic, 10MW-on-K8s, 10SRE, 10serviceops, 10Release-Engineering-Team (Seen): Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Lucas_Werkmeister_WMDE) [14:37:13] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [14:39:11] 10netops, 10Data-Persistence, 10Data-Persistence-Backup, 10Infrastructure-Foundations, and 2 others: Migrate servers in codfw rack B4 from asw-b4-codfw to lsw1-b4-codfw - https://phabricator.wikimedia.org/T355860 (10Marostegui) [14:39:25] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10Marostegui) [14:39:46] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10Marostegui) [14:41:32] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A5 from asw-a5-codfw to lsw1-a5-codfw - https://phabricator.wikimedia.org/T355864 (10Marostegui) [14:41:52] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866 (10Marostegui) [14:44:26] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A8 from asw-a8-codfw to lsw1-a8-codfw - https://phabricator.wikimedia.org/T355874 (10Marostegui) [14:44:38] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE: Migrate servers in codfw rack B3 from asw-b3-codfw to lsw1-b3-codfw - https://phabricator.wikimedia.org/T355870 (10Marostegui) [14:44:48] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B6 from asw-b6-codfw to lsw1-b6-codfw - https://phabricator.wikimedia.org/T355871 (10Marostegui) [14:45:01] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B8 from asw-b8-codfw to lsw1-b8-codfw - https://phabricator.wikimedia.org/T355873 (10Marostegui) [14:47:44] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10Marostegui) db2142 - x2 master db2103 - s1 master es2020 - es4 master [14:48:21] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A8 from asw-a8-codfw to lsw1-a8-codfw - https://phabricator.wikimedia.org/T355874 (10Marostegui) db2146 - slave db2106 - slave [14:48:43] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE: Migrate servers in codfw rack B3 from asw-b3-codfw to lsw1-b3-codfw - https://phabricator.wikimedia.org/T355870 (10Marostegui) db2108 - slave db2123 - slave es2021 - es4 master [14:50:19] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10Marostegui) db2183 - codfw backup master @jcrespo [14:51:53] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A5 from asw-a5-codfw to lsw1-a5-codfw - https://phabricator.wikimedia.org/T355864 (10Marostegui) db2121 - slave db2132 m1 master (not used) db2145 - slave db2104 - m2 master db2153 - slave db2154 - slave db2... [14:58:51] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866 (10Marostegui) db2155 - slave db2156 - slave db2097 - backups slave @jcrespo db2105 - s3 master db2122 - slave db2133 - m2 ma... [15:03:14] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B6 from asw-b6-codfw to lsw1-b6-codfw - https://phabricator.wikimedia.org/T355871 (10Marostegui) db2098 - backup slave @jcrespo db2110 - slave db2111 - slave db2124 - slave db2134 - m3 master (not used) db20... [15:04:57] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack B8 from asw-b8-codfw to lsw1-b8-codfw - https://phabricator.wikimedia.org/T355873 (10Marostegui) db2148 - slave db2163 - slave db2185 zarcillo dc master (nothing required) db2164 - slave db2189 - slave es2029... [15:05:07] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10jcrespo) Thank you, I will shutdown media backups anyway every time one host is affected, not just this one, to minimize fa... [15:08:06] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A3 from asw-a3-codfw to lsw1-a3-codfw - https://phabricator.wikimedia.org/T355862 (10Marostegui) [15:13:15] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by ayounsi@cumin2002 for hosts: `sretest1005.eqiad.wmnet` - sretest1005.eqiad.wmnet (... [15:40:20] 10Traffic, 10SRE: create a puppetized abstraction for haproxy blocklist hysteresis - https://phabricator.wikimedia.org/T329331 (10CDanis) @Fabfur just wanted to make sure you've seen this task, it is decent documentation of the existing mechanism and probably helpful for doing T353910 [15:41:09] thanks cdanis [15:41:21] 10Traffic, 10Infrastructure-Foundations, 10SRE: NEL: don't alert on domains we don't control - https://phabricator.wikimedia.org/T349807 (10CDanis) p:05Triage→03Medium [15:43:41] 10netops, 10Infrastructure-Foundations, 10SRE: Put Dell SONiC switches in production - https://phabricator.wikimedia.org/T335028 (10ayounsi) p:05Triage→03Medium [15:46:24] 10Wikimedia-Apache-configuration, 10SRE, 10Security-Team, 10WMF-General-or-Unknown, and 3 others: Add security.txt to Wikimedia sites? (2023 edition) - https://phabricator.wikimedia.org/T337949 (10joanna_borun) [15:48:09] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A4 from asw-a4-codfw to lsw1-a4-codfw - https://phabricator.wikimedia.org/T355863 (10cmooney) [15:48:51] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866 (10Marostegui) [15:59:46] 10Traffic, 10Data-Persistence, 10conftool, 10serviceops: Switch conftool to use the version 3 etcd datastore - https://phabricator.wikimedia.org/T350565 (10Volans) [16:00:59] fabfur: no no, apologies for not doing that earlier, I dug up that task this morning and then I found a draft message where I was going to cc you back in December [16:17:29] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Move lvs2012 from private1-b-codfw (row) to private1-b2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352918 (10cmooney) [16:18:54] 10Traffic, 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Move lvs2011 from private1-a-codfw (row) to private1-a2-codfw (rack) vlan - https://phabricator.wikimedia.org/T352920 (10cmooney) [16:19:14] 10Traffic, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: NetworkProbeLimit cookie should set samesite attribute - https://phabricator.wikimedia.org/T342624 (10CDanis) p:05Triage→03Low a:03CDanis [16:20:21] 10Traffic, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: GeoIP mapping experiments - https://phabricator.wikimedia.org/T332024 (10CDanis) p:05Triage→03Low a:05JameelKaisar→03CDanis [16:46:13] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355833 (10bcampbell) @jhathaway I do not know what CNAME record 4 is for. I can ask Sandra to connect me with Sh... [16:47:11] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE: Ensure that store.wikimedia.org complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355835 (10bcampbell) Thanks @ssingh. All looks good on the Shopify end for this instance. It says are domain is authenticating... [16:51:17] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE: Ensure that store.wikimedia.org complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355835 (10ssingh) 05Open→03Resolved a:03ssingh Thanks for letting us know @bcampbell. I am marking this as resolved; in... [16:52:32] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE: Ensure that store.wikimedia.org complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355835 (10bcampbell) Thanks @ssingh . The other Shopify instance still needs the CNAME records added it looks like, but we are... [16:58:25] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355833 (10jhathaway) It appears to point to an SPF record: ` u13504486.wl237.sendgrid.net. 1740 IN TXT "v=spf... [16:59:36] 10Traffic, 10DNS, 10Foundational Technology Requests, 10SRE: Ensure that wikimediafoundation.myshopify.com complies with Google's new email sender guidelines - https://phabricator.wikimedia.org/T355833 (10bcampbell) @jhathaway I reached out to Sandra requesting that I be connected with our Shopify rep for... [18:23:52] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [23:15:25] 10Traffic, 10Patch-For-Review: Restarting fifo-log-demux should not restart nginx - https://phabricator.wikimedia.org/T355905 (10BCornwall) @bblack, @ssingh, @Vgutierrez T284555 added in a BindsTo= so that fifo-log-demux and nginx were dependent on each other. Just want to make sure that we do, in fact, want f...