[08:11:05] <volans>	 FYI thanos-be1003 is cron-spamming every 5 minutes with:
[08:11:07] <volans>	 Exception during recon-cron while accessing devices: [Errno 13] Permission denied: '/srv/swift-storage/sdf1'
[08:11:19] <volans>	 (cross posting in -observability cc go.dog too)
[08:12:02] <marostegui>	 volans: we do not own those services yet
[08:12:25] <volans>	 that's why I cross-posted, I'm not sure at which point we are on that ;)
[08:12:32] <marostegui>	 haha
[08:12:54] <marostegui>	 Hopefully after august we might start getting those :)
[10:12:57] <Amir1>	 marostegui: two updates, later this week I will start cleaning image table in test wikis and such, and next week for real in commons. The second: cleaning up logging table of dewiki, ruwiki, etc. will start today, can I have some numbers on what are the biggest logging tables are?
[10:13:09] <Amir1>	 so we would know the impact, etc.
[10:13:16] <marostegui>	 Amir1: let me check
[10:13:56] <marostegui>	 -rw-rw---- 1 mysql mysql 20G Jun 30 10:13 /srv/sqldata/ruwiki/logging.ibd
[10:14:18] <marostegui>	 -rw-rw---- 1 mysql mysql 54G Jun 30 10:13 /srv/sqldata/dewiki/logging.ibd
[10:14:29] <marostegui>	 54GB...jeez
[10:14:55] <marostegui>	 Amir1: keep in mind that god knows when were these tables last optimized, so those numbers might include blank spaces too
[10:15:29] <Amir1>	 haha, it'll be fun
[10:15:34] <marostegui>	 I can try to optimize them before you start the clean up, but that would need to be tomorrow, as we are not doing maintenance till tomorrow (so the dust can settle after the switch)
[10:16:40] <Amir1>	 I can wait, this is not urgent
[10:17:11] <Amir1>	 just keep it in mind, it's not just these two wikis, it's around 40 wikis
[10:17:27] <Amir1>	 huwiki, etc.
[10:17:50] <marostegui>	 I will optimize ruwiki and dewiki tomorrow in a host, so we can compare with those two
[10:18:22] <Amir1>	 https://github.com/wikimedia/operations-mediawiki-config/blob/master/dblists/flaggedrevs.dblist
[10:18:28] <Amir1>	 sure
[10:18:59] <marostegui>	 I can do enwiki too (it is 25G)
[10:19:17] <wikibugs>	 10DBA, 10SRE, 10Datacenter-Switchover: switchdc should automatically downtime "Read only" checks on DB masters being switched - https://phabricator.wikimedia.org/T285803 (10Kormat) To clarify, there's no way to only downtime a specific icinga check across multiple machines (that i know of). I used `sre.hosts...
[10:19:33] <Amir1>	 enwiki won't be much because it's on protect mode
[10:19:50] <marostegui>	 What is protect mode?
[10:20:11] <Amir1>	 flaggedrevs can be enabled on all of the wiki or protect mode (a small set of pages)
[10:20:31] <Amir1>	 when it's on all of the wiki, the flaggedrevs logging would be astronomical 
[10:20:33] <marostegui>	 Aaah! and it is fully enabled on dewiki and ruwiki?
[10:20:37] <Amir1>	 yup
[10:20:43] <Amir1>	 in forty of them
[10:20:44] <marostegui>	 Ah ok, let's try there then
[10:21:49] <Amir1>	 I assume basically any logging table that is big across all wikis has something to do with this
[10:22:00] <Amir1>	 (unless the wiki itself is really big like enwiki)
[10:22:07] <marostegui>	 yeah, makes sense
[10:22:14] <marostegui>	 let's see how de/ru wiki go
[10:22:20] <Amir1>	 <3
[11:42:58] <wikibugs>	 10DBA, 10SRE, 10Datacenter-Switchover: When switching DCs, update pc hosts in tendril - https://phabricator.wikimedia.org/T266723 (10Kormat) @Legoktm : great, thanks!
[12:03:07] <wikibugs>	 10DBA, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: Figure out how x2 should be handled in DC switchover - https://phabricator.wikimedia.org/T285519 (10Kormat) >>! In T285519#7182387, @Krinkle wrote: > Yes, this would be my preferred outcome as well. This would be least risky and easiest to reason a...
[12:03:17] <kormat>	 marostegui: FYI ^
[12:03:45] <marostegui>	 kormat: I will check in a sec, writing scorecards :)
[12:17:29] <Reedy>	 "you get a point"
[12:17:30] <Reedy>	 "you get a point"
[12:17:36] <Reedy>	 "you all get points!"
[12:25:14] <kormat>	 "some of them are negative"
[12:25:48] * Reedy coughs
[12:55:33] <wikibugs>	 10DBA, 10SRE, 10Datacenter-Switchover, 10Patch-For-Review: Figure out how x2 should be handled in DC switchover - https://phabricator.wikimedia.org/T285519 (10Marostegui) Ok, so we need to treat it like parsercache in that regard.
[15:39:50] <urbanecm>	 marostegui: kormat: fyi, creating few more databases for more wikis. Will update tasks once I'm done :).
[15:42:33] <marostegui>	 urbanecm: cheers
[15:46:50] <wikibugs>	 10DBA, 10Data-Services: Prepare and check storage layer for shiwiki - https://phabricator.wikimedia.org/T284928 (10Urbanecm) Database was created.
[15:47:48] <wikibugs>	 10DBA, 10Data-Services: Prepare and check storage layer for dagwiki - https://phabricator.wikimedia.org/T284456 (10Urbanecm) Database was created.
[16:14:54] <wikibugs>	 10DBA, 10Data-Services: Prepare and check storage layer for banwikisource - https://phabricator.wikimedia.org/T284390 (10Urbanecm) The database was created.
[16:33:18] <wikibugs>	 10DBA, 10MediaWiki-Parser, 10Performance-Team, 10MW-1.37-notes (1.37.0-wmf.11; 2021-06-21), and 2 others: purgeParserCache.php should not take over 24 hours for its daily run - https://phabricator.wikimedia.org/T282761 (10Krinkle) The manual run for pc1 and pc2, as well as the scheduled run that was at pc3...