[04:33:36] <marostegui>	 Going to start with s6 switchover
[04:43:31] <marostegui>	 elukey: For https://phabricator.wikimedia.org/T371132 I am off next week, do you want to leave db1179 (the only one which is in production) done today so you aren't blocked on me for the databases part?
[04:55:35] <marostegui>	 s6 is done, I am going to do s8 now
[05:55:15] <marostegui>	 set global binlog_format=ROW;
[05:55:20] <marostegui>	 sorry, wrong window
[06:06:18] <marostegui>	 Can I get a review of https://gerrit.wikimedia.org/r/c/operations/puppet/+/1058295 - this is for Jaime's new backups hosts. He asked to get them ordered and installed while he's out. 
[06:06:54] <marostegui>	 https://phabricator.wikimedia.org/T371416 here he mentions the recipe we need to use
[06:46:08] <elukey>	 marostegui: holaaaa
[06:46:22] <marostegui>	 elukey: holaaaaaaa guapo!
[06:46:49] <elukey>	 definitely yes, the cookbook reboots the host afaics, so once you have time and you want to depool I'll be ready
[06:47:24] <marostegui>	 Sure, I will do it now
[06:50:09] <marostegui>	 elukey: you can proceed anytime - I have also downtimed it (1h)
[06:51:30] <volans>	 backup CR done
[06:52:17] <marostegui>	 thank you volans - I am sending the fix
[06:57:18] <elukey>	 marostegui: ack thanks!
[06:57:22] <elukey>	 so db1179 right?
[06:57:27] <marostegui>	 elukey: yeop
[06:57:29] <marostegui>	 yep
[07:01:31] <elukey>	 started!
[07:05:08] <marostegui>	 elukey: 🤌
[07:07:30] <volans>	 lol
[07:15:19] <marostegui>	 I see the host is back \o/
[07:15:38] <marostegui>	 I will wait for the green light though, to start mariadb etc
[07:20:24] <elukey>	 marostegui: db1179 done!
[07:20:31] <elukey>	 checked and it was rebooted
[07:20:42] <elukey>	 thanks a lot!
[07:21:03] <elukey>	 I'll start also all the others
[07:36:52] <marostegui>	 Thanks elukey !
[09:15:34] <dhinus>	 clouddb1019 is taking a long time to apply the s4 schema change (clouddb1015 did it in 24 hours, clouddb1019 is taking 3+ days)
[09:15:55] <dhinus>	 I think this is connected to T367778, still not clear if it's due to user load or to something wrong with that server
[09:15:56] <stashbot>	 T367778: [wikireplicas] frequent replag spikes in clouddb hosts - https://phabricator.wikimedia.org/T367778
[09:16:10] <dhinus>	 I'll try depooling clouddb1019 for a while and send all the traffic to clouddb1015
[09:18:50] <marostegui>	 dhinus: There are lots of variables on why that could be, including the innodb buffer pool size efficiency based on the queries the host get
[09:19:41] <dhinus>	 yep, pretty hard to draw a clear conclusion. I will depool for now and see how the other server handles the traffic
[09:20:07] <marostegui>	 sounds good
[09:20:29] <dhinus>	 I will temporaraily increase the pt-kill threshold on clouddb1015 to match the "analytics" threshold used in clouddb1019
[09:21:43] <marostegui>	 yeah mean you'll allow queries to run for longer?
[09:22:23] <marostegui>	 *you mean
[09:22:30] <dhinus>	 yep, otherwise analytics queries currently going to 1019 will be killed after 5 mins if they run on 1015
[09:22:46] <marostegui>	 Yep
[09:23:50] <dhinus>	 I will set the threshold to 10800 in /etc/default/wmf-pt-kill on 1015, so it should behave just like 1019
[09:24:26] <dhinus>	 the schema change is already applied on 1015 and replag is at 0
[09:31:30] <Amir1>	 dhinus: sometimes it has to wait for table lock, make sure the "show slave status" is on "alter table" state
[09:32:01] <dhinus>	 it's on "copy to tmp table"
[09:32:17] <Amir1>	 that's good too
[09:32:37] <Amir1>	 I've seen replication got hold up for days in one case (it was clouddb1021 I think) because queries were constantly locking the table
[09:34:39] <Amir1>	 >      11 | system user  |                      | commonswiki   | Slave_SQL |  xxx | copy to tmp table                | ALTER TABLE revision CHANGE rev_id rev_id 
[09:34:54] <Amir1>	 it just needs time
[09:35:28] <dhinus>	 yep, but it's taking 3x the time it took on the other server... hopefully now that I depooled it should be faster
[09:36:34] <Amir1>	 yeah, it'll help
[12:27:15] <dhinus>	 here's a "fun" Grafana issue that hopefully is rare enough that I'm the only one who encountered it: T367778
[12:27:16] <stashbot>	 T367778: [wikireplicas] frequent replag spikes in clouddb hosts - https://phabricator.wikimedia.org/T367778
[12:27:21] <dhinus>	 sorry wrong task
[12:27:30] <dhinus>	 T371485
[12:27:34] <stashbot>	 T371485: Grafana MySQL charts can be inconsistent when zooming out - https://phabricator.wikimedia.org/T371485
[12:27:39] <dhinus>	 cc godog 
[12:28:52] <godog>	 dhinus: ack thx, will take a look
[12:36:35] <marostegui>	 dhinus: what should be the right tags for this https://phabricator.wikimedia.org/T371486 ?
[12:36:50] <marostegui>	 And same for https://phabricator.wikimedia.org/T371488
[12:36:55] <marostegui>	 It is not something we (DBAS) do
[12:37:02] <marostegui>	 It would be either wmcs or data-engineering
[12:41:21] <dhinus>	 I'd say #data-services in the "wikreplicas" column
[12:41:58] <marostegui>	 dhinus: Just that one? No team tag?
[12:42:15] <dhinus>	 you can add "cloud-services-team" but I'll see them anyway
[12:42:24] <marostegui>	 ok :)
[12:42:30] <dhinus>	 we're considering auto-adding the team tag for all tags related to things that we manage
[14:16:48] <jinxer-wm>	 FIRING: [2x] MysqlReplicationLagPtHeartbeat: MySQL instance db2148:9104 has too large replication lag (15m 0s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db2148&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat
[14:17:30] <marostegui>	 volans: https://phabricator.wikimedia.org/T369654#10032068 all those hosts are bookworm
[14:17:38] <marostegui>	 Checking db2148
[14:18:12] <marostegui>	 jeez another index corruption, I will fix it and ping mariadb
[14:19:30] <Amir1>	 >              Last_SQL_Error: Error 'Index for table 'pagelinks' is corrupt; try to repair it' on query. Default database: 'idwiki'. Query: 
[14:19:36] <marostegui>	 I am fixing it yes
[14:19:40] <marostegui>	 I will upgrade to .18 too
[14:19:48] <Amir1>	 Is it depooled?
[14:19:50] <Amir1>	 I can do it
[14:20:04] <marostegui>	 it is
[14:20:06] <marostegui>	 I just did it
[14:20:14] <Amir1>	 thanks
[14:20:19] <Amir1>	 I was too slow :(
[14:20:46] <volans>	 marostegui: re T369654#10032068: I know, but the issue exists only because we're still supporting the possibility to reimage into buster and puppet 5 :D
[14:20:47] <stashbot>	 T369654: Q1:rack/setup/install db22[21-40] - https://phabricator.wikimedia.org/T369654
[14:20:56] <marostegui>	 volans: ah right right
[14:21:02] <marostegui>	 You gave me a heart attack
[14:21:14] <volans>	 :D
[14:21:16] <volans>	 sorry
[14:29:58] <Daimona>	 Hi folks! On beta, I'm getting the following error when querying a table for a new feature: "ERROR 1728 (HY000): Cannot load from mysql.proc. The table is probably corrupted" Would someone be willing to look into this or tell me where to look?
[14:31:02] <Amir1>	 corrupted tables are fun
[14:31:17] <Amir1>	 try running "optimize table <name of table>"
[14:31:19] <marostegui>	 Daimona: I don't even have access to beta but you can try a mysql_upgrade --force
[14:31:47] <Daimona>	 FTR, it's reproducible with `sql wikishared` and then: SELECT ceil_id,ceil_name,ceil_status,ceil_created_at,ceil_user_id,COUNT (ceilu_id) AS `ceil_editor_count` FROM `ce_invitation_lists` LEFT JOIN `ce_invitation_list_users` ON ((ceil_id=ceilu_ceil_id) AND (ceilu_score >= 25)) WHERE ceil_wiki = 'metawiki';
[14:32:34] <Daimona>	 optimizing didn't do anything
[14:36:06] <Daimona>	 As for, mysq_upgrade, I've no idea if I need to specify other options, run it on a specific host, or whatever
[14:36:34] <marostegui>	 Daimona: On the database host, just as root
[14:38:08] <Daimona>	 I don't know what that host is, and what the password is (I'm sure someone explained this to me a few years ago, but...)
[14:38:34] <marostegui>	 :( I don't have access to beta so I cannot help much
[14:39:11] <Amir1>	 I can take a look once I'm done with the checklist runbook
[14:40:13] <Daimona>	 Wait a sec
[14:40:25] <marostegui>	 For what is worth, we don't maintain beta :)
[14:40:40] <Daimona>	 It seems to be caused by an extra space after `COUNT` in the query???!!!
[14:40:54] <marostegui>	 woot
[14:41:06] <Daimona>	 SELECT COUNT (ceilu_id) FROM `ce_invitation_list_users`; <-- triggers the error
[14:41:19] <Daimona>	 SELECT COUNT(ceilu_id) FROM `ce_invitation_list_users`; <-- works just fine
[14:41:31] <marostegui>	 that makes no sense :-/
[14:41:40] <marostegui>	 Daimona: can you run analyze table mysql.proc ?
[14:42:58] <Daimona>	 Permission denied and I don't know how to fix that per above :O
[14:43:07] <marostegui>	 right...
[14:43:29] <marostegui>	 I wonder if this is just part of a general corruption of the systems tables after an upgrade or half of it
[14:56:59] <Daimona>	 It seems like this might be intended behaviour according to the note in https://dev.mysql.com/doc/refman/8.4/en/functions.html. I didn't find an equivalent page for mariadb but I guess it doesn't matter
[14:59:00] <Daimona>	 I imagine that without the space, the parser hallucinates and think I'm querying whatever strange table that doesn't exist or something. Because saying "Heeeeelp the table is corrupted!" is not quite on the same level as "bozo, you have an extra space in your query".
[15:00:55] <Daimona>	 So, I guess I'll add this to my list of ridiculously unhelpful MySQL error messages and update the code. Still, thanks y'all for assisting!