[00:21:40] 10serviceops, 10SRE, 10Parsoid (Tracking): Maybe consider consolidating parsoid-* and restbase-* proxy services, respectively - https://phabricator.wikimedia.org/T285445 (10Arlolra) Also consider T279825#7174049 [01:44:54] 10serviceops, 10SRE, 10conftool, 10Datacenter-Switchover, 10Patch-For-Review: Disable maintenance scripts via conftool - https://phabricator.wikimedia.org/T266717 (10Legoktm) I updated https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/701053/ with a process that should address all the conc... [07:04:03] jayme: ping me when you're around so we can discuss the services switchover? [07:04:33] legoktm: ping [07:04:37] :D [07:05:43] so https://wikitech.wikimedia.org/wiki/Switch_Datacenter#Services https://gerrit.wikimedia.org/r/plugins/gitiles/operations/cookbooks/+/refs/heads/master/cookbooks/sre/switchdc/services/ is how the cookbook works [07:05:56] but you mentioned that the procedure for the recent upgrade was different? [07:07:05] we used a different cookbook which was/is not designed for switchdc during the k8s upgrade [07:07:43] but I would ignore that for now as that would def. need some love [07:08:12] lets just go with the documented one [07:08:39] ok, so you're good with using what we have right now? [07:10:12] yeah, the other one I had multiple issues with (during k8s upgrade). But ofc. I've not done any work on it or any sanity checking of the switchdc one [07:15:07] gotcha [08:32:43] !issync [08:32:44] Syncing #wikimedia-serviceops (requested by legoktm) [08:32:46] Set /cs flags #wikimedia-serviceops wikibugs +Vv [08:37:06] 10serviceops, 10SRE, 10vm-requests: codfw: 1 of VMs requested for gitlab - https://phabricator.wikimedia.org/T285456 (10Dzahn) [08:37:19] 10serviceops, 10GitLab, 10SRE, 10vm-requests: codfw: 1 of VMs requested for gitlab - https://phabricator.wikimedia.org/T285456 (10Dzahn) [08:42:42] 10serviceops, 10GitLab, 10SRE, 10vm-requests: codfw: 1 of VMs requested for gitlab - https://phabricator.wikimedia.org/T285456 (10MoritzMuehlenhoff) Looks fine, please create in row C or D to better balance our our capacity. In general let's avoid notes like "same as gitlab1001", if one looks back at the... [08:59:07] thanks for that lego [08:59:54] :) [09:24:13] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Performance-Team (Radar): June 2021 Datacenter switchover - https://phabricator.wikimedia.org/T281515 (10Marostegui) [12:04:12] 10serviceops, 10MW-on-K8s, 10Release Pipeline, 10Patch-For-Review: Run stress tests on docker images infrastructure - https://phabricator.wikimedia.org/T264209 (10JMeybohm) I made a comparison between Dragonfly and Kraken and while I do think Kraken is probably the better solution technically it's lack of... [13:21:11] 10serviceops, 10SRE, 10Parsoid (Tracking): Maybe consider consolidating parsoid-* and restbase-* proxy services, respectively - https://phabricator.wikimedia.org/T285445 (10jbond) p:05Triage→03Medium [13:38:45] 10serviceops, 10Maps, 10Patch-For-Review, 10User-jijiki: Deploy tegola-vector-tiles to kubernetes - https://phabricator.wikimedia.org/T283159 (10jijiki) [13:39:16] 10serviceops, 10User-jijiki: Productionise thumbor1005 and thumbor1006 - https://phabricator.wikimedia.org/T285477 (10jijiki) [13:40:13] 10serviceops, 10User-jijiki: Productionise thumbor1005 and thumbor1006 - https://phabricator.wikimedia.org/T285477 (10jijiki) [13:45:45] 10serviceops, 10decommission-hardware, 10User-jijiki: decommission thumbor1003 - https://phabricator.wikimedia.org/T285479 (10jijiki) [13:46:36] 10serviceops, 10decommission-hardware, 10User-jijiki: decommission thumbor1004.eqiad.wmnet - https://phabricator.wikimedia.org/T285480 (10jijiki) [13:47:00] 10serviceops, 10decommission-hardware, 10User-jijiki: decommission thumbor1003.eqiad.wmnet - https://phabricator.wikimedia.org/T285479 (10jijiki) [15:48:58] 10serviceops, 10MW-on-K8s, 10SRE, 10User-jijiki: Enable TLS termination on the mwdebug deployment. fix the service definition in the chart - https://phabricator.wikimedia.org/T284421 (10jijiki) a:03jijiki [15:49:39] 10serviceops, 10MW-on-K8s, 10SRE, 10User-jijiki: Create a mwdebug deployment for mediawiki on kubernetes - https://phabricator.wikimedia.org/T283056 (10jijiki) [17:46:42] 10serviceops, 10SRE, 10Datacenter-Switchover, 10Performance-Team (Radar): June 2021 Datacenter switchover - https://phabricator.wikimedia.org/T281515 (10wkandek) [18:31:28] 10serviceops, 10Performance-Team, 10SRE, 10MW-1.36-notes, and 3 others: Enable "/*/mw-with-onhost-tier/" route for MediaWiki where safe - https://phabricator.wikimedia.org/T264604 (10Krinkle) ### Background 12 local tabs with: ` $ watch -n0 "curl 'https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page' -H '... [23:24:14] 10serviceops, 10SRE, 10Datacenter-Switchover: Siteinfo timeout during switch datacenter - https://phabricator.wikimedia.org/T266618 (10Legoktm) [23:24:37] 10serviceops, 10SRE, 10Datacenter-Switchover: Siteinfo timeout during switch datacenter - https://phabricator.wikimedia.org/T266618 (10Legoktm) >>! In T266618#7170274, @Legoktm wrote: >> it's connecting to port 80 with the x-forwarded-proto header, and that should probably be updated. > > This is easy, I'll... [23:26:28] 10serviceops, 10SRE, 10Datacenter-Switchover: Various services hardcode api.svc.eqiad.wmnet - https://phabricator.wikimedia.org/T285518 (10Legoktm) p:05Triage→03High