[00:44:01] <Platonides>	 musikanimal: you may want to make the bot run the usernames through the username regex to enure it is legal
[01:09:16] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 8.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[01:12:58] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[05:46:42] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db1117 is CRITICAL: 53 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321
[05:47:26] <icinga-wm>	 PROBLEM - MariaDB sustained replica lag on m1 on db2160 is CRITICAL: 30.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[05:48:32] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db1117 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1117&var-port=13321
[05:49:16] <icinga-wm>	 RECOVERY - MariaDB sustained replica lag on m1 on db2160 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2160&var-port=13321
[11:29:46] <jynus>	 as a followup to the meeting yestarday, this is the error that missled me: "Wikimedia\Rdbms\LoadMonitor::computeServerStates: host {db_server} is unreachable"
[11:30:06] <jynus>	 and this is the ticket that Chris mentioned on the other incident doc: T265386
[11:30:06] <stashbot>	 T265386: Make LoadMonitor server states more up-to-date and respond to outages more quickly - https://phabricator.wikimedia.org/T265386
[11:31:35] <jynus>	 ^this is the one I heard and maybe (I haven't checked) related to mw-level improvements Amir1 mentioned
[11:32:13] <Amir1>	 jynus: every time I heard LoadMonitor, I want to cry
[11:32:40] <jynus>	 just passing things other SREs mentioned, don't shoot the messenger :-D
[11:32:53] <Amir1>	 currently, it's just useless, was made to prevent incidents, makes a ton of connections and db queries, including cross-dc ones and have prevented zero (0) outages
[11:33:16] <Amir1>	 the plan is to rewrite it fully
[11:33:18] <Amir1>	 Tim is on it
[11:33:19] <jynus>	 I am in the opinion that load balancing should be its own dedicated service with more standardized tooling
[11:33:42] <Amir1>	 Tim's work: T314020
[11:33:42] <stashbot>	 T314020: LoadMonitor connection weighting reimagined - https://phabricator.wikimedia.org/T314020
[11:34:22] <Amir1>	 My opinion is that at least for CLI, it should be like that (having a proxy in between for LB)
[11:34:57] <jynus>	 well, also use standarized wmf configuration
[11:35:05] <Amir1>	 MW's rdbms library is trying to juggle two distinct worlds, long-running scripts and short web requests 
[11:35:35] <jynus>	 sure, there may be some customization needs, but reinventing algorithm that things like haproxy has very robusltly implemented seems not the right direction to me
[11:35:50] <jynus>	 haproxy or many other 3rd party tools
[11:36:14] <Amir1>	 yeah but let's not use sqlproxy, I do enough sql day and night :P
[11:36:20] <Amir1>	 proxysql
[11:36:53] <Amir1>	 yeah, my hope is to have maint scripts switch to haproxy maybe in a year or two, will see
[11:37:11] <jynus>	 and yes, something more custom could fit better, but in the end it is humans administrating it, and not only one developer that will have to manage it (standarization helps onboarding)
[11:47:56] <Amir1>	 jynus: if you feel like it, can you write a doc outlining the benefits? 
[11:49:53] <jynus>	 sure, it is more non technical and more practical reasons; basically the same idea of what k8s provide: standarization and microservices, which is more or less the general philosopy SREs (and WMF) are moving to
[12:25:59] <jynus>	 I am going to ack "Reduced availability for job mysql-labs in ops@codfw" because afaik it is just T326584
[12:26:00] <stashbot>	 T326584: Decomission db209[345] - https://phabricator.wikimedia.org/T326584
[12:26:17] <marostegui>	 jynus: mmm but db2093 is active?
[12:26:32] <marostegui>	 I just stopped mysql there
[12:26:35] <jynus>	 I think it is just complaining about db2084
[12:26:39] <jynus>	 *94
[12:26:48] <marostegui>	 but db2094 is sanitarium
[12:27:09] <jynus>	 I think it is in the "labs" prometheus group (as it is not core)
[12:27:17] <marostegui>	 Ah then yeah
[12:27:24] <marostegui>	 Might be related then yeah
[12:27:29] <jynus>	 but because it has very little instances
[12:27:37] <jynus>	 it complains even if there is only 1 host down
[12:27:53] <jynus>	 while for core would have to fail a lot of hosts
[12:30:59] <jynus>	 alertmanager UI is great, but at the same time suspiciously too well thought compared to how barebones icinga is that make me suspicious
[12:31:22] <marostegui>	 I find alertmanager hard to read
[12:31:23] <jynus>	 I've silenced it for 2 days linking to the task
[12:31:33] <jynus>	 oh, I may have some tips
[12:31:44] <jynus>	 I added some filters and now I prefer it over icinga
[12:31:53] <marostegui>	 which filters do you have?
[12:32:16] <jynus>	 @state=active team=sre alertname!=Check systemd state
[12:33:04] <jynus>	 and that gives me a much cleaner dashboard
[12:33:11] <marostegui>	 For me it is still soooo full
[12:33:19] <marostegui>	 I find it impossible to get the data
[12:33:30] <jynus>	 ha ha ha but that is a wmf issue, not a ui!
[12:33:36] <marostegui>	 I guess it is a matter of getting used to it
[12:33:48] <marostegui>	 No, I find it very hard to read
[12:33:54] <marostegui>	 So many different boxes, colors, etc
[12:33:55] <jynus>	 I am not fond of the boxes
[12:34:15] <jynus>	 they are not super readable, for example, for finding the host
[12:56:16] <jynus>	 with you permission, I will put down db2184 in the afternoon to test again the 10.6 backup recovery
[12:56:27] <marostegui>	 sounds good
[12:56:42] <marostegui>	 I also gave you some more homework before you leave on holidays https://gerrit.wikimedia.org/r/c/operations/puppet/+/892948
[12:56:43] <jynus>	 I will rename the datadir just in case
[12:56:52] <jynus>	 let me see
[12:57:24] <jynus>	 looks fine to me, can be merged now?
[12:57:32] <marostegui>	 sure, anytime
[12:57:35] <marostegui>	 Let me merge it
[12:57:41] <marostegui>	 But maybe you need to try a backup for it?
[12:57:55] <jynus>	 sure, I think it takes 1 second to backup that :-D
[12:57:59] <marostegui>	 ok, merging!
[12:58:24] <jynus>	 let me run it now
[12:58:26] <marostegui>	 merged
[13:01:52] <jynus>	 I was wrong, it took 4 seconds: https://phabricator.wikimedia.org/T326596#8652464
[13:02:23] <marostegui>	 Can I also get a sanity check on https://gerrit.wikimedia.org/r/c/operations/puppet/+/892953 
[13:02:32] <marostegui>	 jynus: that's 4 times what you predicted!!!
[13:02:57] <Amir1>	 marostegui: just the binlog of pc1012 has gone from 310GB to 240GB
[13:03:07] <marostegui>	 Amir1: that's awesome
[13:04:31] <jynus>	 marostegui: the ip is fine, but I am not sure what those are used for anymore?
[13:04:42] <marostegui>	 jynus: Yeah, I was trying to figure that out now
[13:05:00] <jynus>	 it could be a leftover predating cumin hosts
[13:05:11] <marostegui>	 jynus: They are used for orchestrator I believe
[13:05:16] <jynus>	 or from the times of tendril
[13:05:27] <jynus>	 but that would be dborch, right?
[13:05:39] <marostegui>	 jynus: yeah, but the DB is in db1115/db2093
[13:05:42] <jynus>	 are they needed from the dbs of orchestrator?
[13:06:04] <jynus>	 what I mean orch is at: 208.80.155.103 # dborch1001.wikimedia.org
[13:06:10] <jynus>	 not so sure about the dbs
[13:06:11] <marostegui>	 yeah, that's the frontend
[13:06:15] <marostegui>	 But I think we run stuff from the db too
[13:06:32] <jynus>	 that I don't know (I am not saying yes or now)
[13:08:02] <jynus>	 (I wasn't involved on anything orch related)
[13:15:38] <jynus>	 think if there is something you both may need me for before the end of the week- I have many things WIP re:backups, but shouldn't affect you (database backups, bacula should work as usual)
[13:16:38] <marostegui>	 will do!
[14:06:07] <marostegui>	 Any objections to this change? https://gerrit.wikimedia.org/r/c/operations/puppet/+/892964/
[14:11:38] <marostegui>	 interesting, that syntax is wrong
[14:13:02] <jynus>	 oh
[14:13:11] <marostegui>	 Yeah I was running the compiler
[14:13:58] <jynus>	 Its only 3, apparently: *-*-*
[14:14:14] <marostegui>	 yeah
[14:14:19] <marostegui>	 I was just sending the patch
[14:17:52] <marostegui>	 https://puppet-compiler.wmflabs.org/output/892964/39867/
[14:17:54] <marostegui>	 This looks a lot better
[14:17:56] <marostegui>	 So merging!
[15:36:02] <Amir1>	 marostegui: FWIW, everytime I make a change to systemd timers, I test it with sudo systemd-analyze calendar "Mon,Tue *-*-01..04 12:00:00"
[15:36:22] <Amir1>	 h/t https://wiki.archlinux.org/title/Systemd/Timers
[15:36:46] <marostegui>	 ah nice
[15:37:45] <marostegui>	 Normalized form: *-*-* 05:00:00
[15:37:45] <marostegui>	     Next elapse: Wed 2023-03-01 05:00:00 UTC
[15:37:46] <marostegui>	 Looks good!
[16:48:17] <jynus>	 I'ts gotten late and don't want to cause alerts while people are busy, so I will search other time for the recovery test
[16:50:35] <jynus>	 (but before the end of the week)