[05:18:59] <marostegui>	 s3 has been switched
[05:19:04] <marostegui>	 primary
[05:58:43] <marostegui>	 btullis: https://phabricator.wikimedia.org/T370122
[08:28:06] <btullis>	 marostegui: I'm happy for you re-clone it, if you like. I could do it, if you would rather.
[08:28:26] <marostegui>	 btullis: I'll take care of it, as I want to upgrade dbstore too
[08:28:39] <btullis>	 Ack, thanks.
[10:51:48] <marostegui>	 btullis: dbstore1008:3317 back
[10:52:06] <btullis>	 Great! Many thanks.
[11:51:44] <marostegui>	 altering revision on enwiki takes around 30h, it is going to take a while to do the whole section
[11:51:54] <marostegui>	 But in wikidata the table is twice the space almost so....:-/
[12:40:49] <volans>	 marostegui, arnaudb: FYI https://phabricator.wikimedia.org/T370029#9984960
[12:42:56] <marostegui>	 volans: I think it is done manually, but I believe arnaudb knows better
[12:44:24] <volans>	 I hope at least with debdeploy :D
[12:44:50] <volans>	 I was wondering if leaving those hosts with an older version was intentional for some reason
[12:44:51] <arnaudb>	 I'll eat, and I'll check
[12:44:58] <arnaudb>	 volans: I don't think so
[12:45:23] <arnaudb>	 it could be those up to date hosts have been reimaged recently
[12:48:35] <marostegui>	 volans: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 6:00:00 this means 24+6h right?
[12:49:15] <volans>	 correct
[12:49:21] <marostegui>	 thanks
[12:50:12] <volans>	 >>> from datetime import timedelta
[12:50:12] <volans>	 >>> print(timedelta(hours=30))
[12:50:12] <volans>	 1 day, 6:00:00
[12:50:40] <marostegui>	 yeah, it is a strange format to report
[12:51:13] <volans>	 I didn't want to add another dependency just to print in a fancy way a timedelta :) but we can change that
[12:51:50] <marostegui>	 nah it is okay, I just have never used 30 hours as downtime in a schema change, so i just wanted to confirm it is 24+6 :)
[12:52:14] <marostegui>	 Otherwise it would page and I'd need to refer to better_call_volans.jpg
[12:53:05] <volans>	 ahahaha
[14:34:03] <arnaudb>	 topranks: https://phabricator.wikimedia.org/T365997#9985920 ready!
[14:34:25] <topranks>	 arnaudb: super thank you!
[15:09:54] <dhinus>	 arnaudb marostegui the clouddb1019 alert should be routed to team=wmcs, see my comment in T315866. right now the icinga alerts are tagged with team=wmcs, but the new prometheus ones are tagged with team=data-persistence
[15:09:55] <stashbot>	 T315866: Migrate mysql icinga alerts to alert manager - https://phabricator.wikimedia.org/T315866
[15:11:45] <dhinus>	 not a big issue for us, just something to keep in mind. when do you plan to remove the icinga ones?
[15:15:45] <arnaudb>	 tbd, first step is to reach feature parity
[15:20:51] * volans has an errand to run, will reconnect later tonight
[15:21:02] <arnaudb>	 👋
[15:26:43] <dhinus>	 arnaudb: ok, then no rush for the team label, because we will still see the icinga alert
[15:29:55] <dhinus>	 I think you also silenced the icinga one in alerts.wikimedia.org, and that one should not be silenced
[15:30:49] <arnaudb>	 ack
[15:33:05] <arnaudb>	 silence 53432765-2729-4b06-9198-a04d03c9966c deleted dhinus 
[15:33:17] <dhinus>	 thanks!
[15:34:05] <dhinus>	 what's this other one matching a few clouddbs? deff9389-3642-42e2-9f98-fb542a4bf5d3
[16:27:40] <dhinus>	 I added a new silence that (if I got the params right) should silence the alerts for clouddb* going to team=data-persistence, without silencing the old icinga ones going to team=wmcs