[01:46:06] 10Traffic, 10Data-Persistence, 10SRE, 10SRE-swift-storage, and 5 others: Change default image thumbnail size - https://phabricator.wikimedia.org/T355914 (10Jdlrobson) [02:27:08] 10Traffic, 10Data-Persistence, 10SRE, 10SRE-swift-storage, and 5 others: Change default image thumbnail size - https://phabricator.wikimedia.org/T355914 (10Superpes15) a:05Superpes15→03None [11:32:42] 10Traffic, 10Data Products, 10Data-Engineering, 10Observability-Logging, 10Patch-For-Review: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117 (10gmodena) > the currently suggested one is webrequest.frontend. @gmodena, the idea there is to group all webrequest topics... [11:48:58] 10netops, 10Infrastructure-Foundations, 10SRE: BGP peering from LSW to K8s hosts using loopback IP not IRB - https://phabricator.wikimedia.org/T357619 (10cmooney) Just a bit more background, I discovered this looking at a tcpdump, this is //lsw1-a4-codfw// trying to establish BGP to //mw2383//: ` 11:10:04.59... [11:49:51] 10Traffic, 10Data Products, 10Data-Engineering, 10Observability-Logging, 10Patch-For-Review: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117 (10gmodena) > Open question: do we want webrequest.frontent (or whatever we settle on) to be a versioned stream? https://wiki... [11:53:49] 10Traffic, 10Data Products, 10Data-Engineering, 10Observability-Logging, 10Patch-For-Review: Move analytics log from Varnish to HAProxy - https://phabricator.wikimedia.org/T351117 (10Fabfur) >>! In T351117#9545687, @gmodena wrote: >> the currently suggested one is webrequest.frontend. @gmodena, the idea... [12:11:48] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team, and 2 others: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by aborrero@cumin1002 for host cloudvirt1031.eqiad.wmnet with OS bookworm [13:01:15] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team, and 2 others: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by aborrero@cumin1002 for host cloudvirt1031.eqiad.wmnet with OS bookworm com... [13:46:32] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team, 10User-aborrero: openstack: nova refuses to admit a compute node after a reimage - https://phabricator.wikimedia.org/T357631 (10aborrero) [14:04:23] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team, 10User-aborrero: openstack: nova refuses to admit a compute node after a reimage - https://phabricator.wikimedia.org/T357631 (10aborrero) https://docs.openstack.org/nova/latest/admin/troubleshooting/orphaned-allocations.html [14:11:31] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [14:12:14] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A5 from asw-a5-codfw to lsw1-a5-codfw - https://phabricator.wikimedia.org/T355864 (10cmooney) 05Open→03Resolved a:03cmooney All looking good, closing task. Thanks everyone for their assistance. [14:57:41] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10ayounsi) In theory if all those patches are merged/deployed, the VM will be using /32 IPs from early_command.sh all the way to its final state and... [15:10:03] 10Traffic, 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 2 others: Create parsoid mediawiki deployment - https://phabricator.wikimedia.org/T357392 (10cscott) Seems fine, most parsoid traffic is probably happening on the main cluster these days anyway. We just want to make sure that scandium doesn't get... [15:49:50] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=dc8a2b8d-561d-404c-ac7f-f64637c16dd1) set by cmooney@cumin... [15:53:19] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team, and 2 others: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184 (10aborrero) [15:58:39] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=23a82a8c-672f-4105-8a05-0b7dbbb4cb97) set by cmooney@cumin... [16:08:38] 10netops, 10Ganeti, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Investigate Ganeti in routed mode - https://phabricator.wikimedia.org/T300152 (10ayounsi) Started to write the doc over there : https://wikitech.wikimedia.org/wiki/Ganeti#Routed_Ganeti [16:11:15] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866 (10cmooney) All moves now complete, ports up on new switch and all devices pinging ok! [16:11:58] 10netops, 10DBA, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate servers in codfw rack A6 from asw-a6-codfw to lsw1-a6-codfw - https://phabricator.wikimedia.org/T355866 (10ABran-WMF) amazing, thanks @cmooney! will start repooling [16:13:49] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Migrate hosts from codfw row A/B ASW to new LSW devices - https://phabricator.wikimedia.org/T355544 (10cmooney) [16:51:56] 10Traffic, 10Content-Transform-Team, 10MW-on-K8s, 10SRE, and 2 others: Create parsoid mediawiki deployment - https://phabricator.wikimedia.org/T357392 (10cscott) Also worth keeping in mind: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/965608 -- in order to avoid inadventently getting exte... [18:24:48] 10Acme-chief, 10Traffic: Puppetize deployment of ncmonitor - https://phabricator.wikimedia.org/T355190 (10BCornwall) 05In progress→03Resolved [18:24:50] 10Acme-chief, 10Traffic: Create automation for registered MarkMonitor DNS and acme-chief/ncredir - https://phabricator.wikimedia.org/T355189 (10BCornwall) [18:29:17] 10Wikimedia-Apache-configuration, 10Fundraising-Backlog, 10SRE, 10fundraising-tech-ops, and 2 others: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10Scott_French) Thanks to @Dwisehaupt for preparing the config patch and @RLazarus for assistance deploying it. The change is now l... [18:32:21] 10Wikimedia-Apache-configuration, 10Fundraising-Backlog, 10SRE, 10fundraising-tech-ops, 10serviceops: Request donatewiki redirect - https://phabricator.wikimedia.org/T357436 (10Scott_French) 05Open→03Resolved a:03Scott_French [19:54:30] 10Traffic, 10collaboration-services: Consider separating Gitlab code management and deb building management - https://phabricator.wikimedia.org/T357719 (10BCornwall)