[05:11:38] <marostegui>	 I am switching s6 codfw master
[05:11:43] <marostegui>	 sanitarium master that is
[05:11:49] <marostegui>	 So I am stopping codfw master
[05:19:52] <icinga-wm>	 PROBLEM - Check unit status of swift_ring_manager on ms-fe1009 is CRITICAL: CRITICAL: Status of the systemd unit swift_ring_manager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[05:33:25] <marostegui>	 s6 done, going for s7 now
[06:14:42] <icinga-wm>	 RECOVERY - Check unit status of swift_ring_manager on ms-fe1009 is OK: OK: Status of the systemd unit swift_ring_manager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[07:02:36] <marostegui>	 jynus: when would it be a good day to stop db2078 (misc multi instance) for a couple of hours to clone db2160?
[07:02:51] <jynus>	 any time now
[07:02:56] <marostegui>	 \o/
[07:02:56] <jynus>	 until monday
[07:02:58] <marostegui>	 Doing it now then
[07:05:01] <marostegui>	 Amir1: db1132 fully repooled with P_S disabled, all yours for the test
[07:05:54] <Amir1>	 marostegui: awesome, going to depool it first :P
[07:06:04] <marostegui>	 :)
[07:06:13] <marostegui>	 Yeah, I pooled it to get it warmed up
[07:19:35] <Amir1>	 marostegui: https://people.wikimedia.org/~ladsgroup/mariadb_flamegraphs/superbusy-without-p_s.106.svg 
[07:23:56] <marostegui>	 Going to compare that with the one yesterday
[07:25:02] <marostegui>	 So there's a bit of an improvement without P_S
[07:27:48] <marostegui>	 We can maybe try to tune a bit the thread pool?
[07:27:52] <marostegui>	 Thoughts?
[07:28:00] <Amir1>	 above my paygrade
[07:28:05] <marostegui>	 XDDD
[07:28:34] <marostegui>	 Amir1: On 10.4 you were never able to generate those errors after 700 connections?
[07:28:47] <Amir1>	 marostegui: I didn't try, I can
[07:28:55] <marostegui>	 Amir1: let's try if you can yeah
[07:29:01] <Amir1>	 my biggest worry is that is it fine to depool two hosts?
[07:29:09] <marostegui>	 Amir1: yeah, it is ok for a bit
[07:29:15] <marostegui>	 we can repool both of them later
[07:29:17] <Amir1>	 ok
[07:29:19] <marostegui>	 I want to see how 10.4 behaves
[07:34:01] <Amir1>	 marostegui: good news, it went kaboom after seven hundred 
[07:34:10] <marostegui>	 \o/
[07:34:12] <marostegui>	 Yay!!
[07:34:24] <marostegui>	 \o\ |o| /o/
[07:36:16] <marostegui>	 Amir1: ok, can you repool both hosts?
[07:36:24] <Amir1>	 sure
[07:36:29] <Amir1>	 let me make the fancy svg 
[07:36:51] <marostegui>	 What I am going to do is disable P_S on the other 10.6 hosts (s4 and s7) and leave it for a few days to see if it gets reproduced with one of the "cache scenarios" if you know what I mean
[07:37:06] <Amir1>	 wink wink
[07:37:09] <marostegui>	 XD
[07:37:19] <marostegui>	 As I am sure it won't take long
[07:37:37] <marostegui>	 And s8 as well
[07:39:11] <Amir1>	 https://people.wikimedia.org/~ladsgroup/mariadb_flamegraphs/superbusy-with-p_s.104.svg 
[07:40:20] <marostegui>	 They pretty similar
[07:46:39] <marostegui>	 Amir1: to confirm, you are repooling both hosts in s1?
[07:46:46] <Amir1>	 yup
[07:46:50] <marostegui>	 great
[08:48:21] <marostegui>	 I am trying to think if we end up having to disable p_s on 10.6.....that's not great but I guess we'd need to come back to pt-kill :(
[09:10:06] <jynus>	 if there is something you will need from me soon, please speak up before my vacations... otherwise I will focus on monitoring improvement rather than starting one of the other big projects
[09:11:37] <marostegui>	 jynus: thoughts on the regression?
[09:12:32] <jynus>	 my guess it is either pool of connections tracing or memory tracing, which is what is new on p_s since 10.5
[09:13:18] <jynus>	 disabling parts of p_s or tuning pool of connections to avoid that shouldn't be too hard- when I first implement p_s I had to tune it a lot due to our memory size
[09:13:20] <marostegui>	 I guess I could try to disable thread pool, but I prefer to go for p_s for now (as it is less impacting I guess)
[09:13:41] <jynus>	 disabling p_s is navigating blind, I don't like it
[09:13:53] <marostegui>	 what do you suggest?
[09:14:03] <jynus>	 disabling only the memory parts
[09:14:11] <marostegui>	 let me see
[09:15:07] <jynus>	 or the threads
[09:16:04] <jynus>	 so either performance-schema-instrument='memory/%=OFF'
[09:16:53] <jynus>	 or thread_instrumentation           | NO
[09:17:05] <marostegui>	 I will try the first one then
[09:17:20] <jynus>	 I think seeing P_S as the cause is a great step, but not the last
[09:18:53] <jynus>	 note I didn't review any of the P_S initial tuning in the last 2 upgrades, there may be a lot to adapt there
[09:21:07] <marostegui>	 ok, disabled it on db1132 with: performance-schema-instrument='memory/%=OFF' on my.cnf
[09:21:45] <jynus>	 e.g. obviously I don't know if that will work, but if it does while keeping the rest of P_S it would be great!
[09:21:52] <marostegui>	 yeah definitely
[09:22:02] <jynus>	 as we would keep process and query performance information
[09:22:16] <marostegui>	 I am repooling it to warm it up and then I will ask Amir1 to run the test and see what happens
[09:22:22] <marostegui>	 Ifn ot we can go also for thread_instrumentation
[09:22:51] <jynus>	 memory and thread instrumentation is what would make sense to me for thead contention
[09:23:01] <marostegui>	 Yeah
[09:23:05] <marostegui>	 Hopefully it is one of those
[09:23:40] <jynus>	 also because were are not in a hurry- if we were full 10.6 now, we take the drastic approach
[09:24:08] <jynus>	 as we have time -I think-, we can tune and test 0:-), and wait for upgrade
[09:24:23] <jynus>	 which BTW also made the bullseye + 10.4 upgrade a win
[09:24:38] <jynus>	 that was a brilliant idea of yours, marostegui
[09:25:02] <marostegui>	 haha whatever we can do to work less!
[09:25:15] <jynus>	 yeah, but imagine-  we would be in a pretty bad situation right now
[09:25:19] <marostegui>	 yeah
[09:25:28] <marostegui>	 We'd have combined many variables
[09:25:34] <marostegui>	 We could think it could be even bullseye
[09:28:28] <jynus>	 and sorry to be the "conservative" person on the gang, but I come from the backups (risk avoidance) perspective- only upgrade when we are very sure things work as expected :-P
[09:29:18] <marostegui>	 :)
[09:32:43] <jynus>	 on a related note, it was clear and still is clear with things like  MDEV-23936 that not all parts of mariadb receive the same attention :-(
[09:33:06] <marostegui>	 Yeah, I am thinking about pinging someone
[09:33:44] <jynus>	 ps was never recommended because "performance"
[09:34:09] <Amir1>	 I was pinged. I'm afk so unless you know a way to ssh to production on a phone (with yubikey support). I can't do it atm
[09:34:18] <jynus>	 (suprising, real-time tracing has an impact on performance)
[09:34:32] <marostegui>	 Amir1: not a rush!
[09:35:00] <Amir1>	 jynus: fwiw mw does sampling for perf measuring 
[09:35:17] <jynus>	 so does P_S!
[09:36:00] <jynus>	 my complain is if mariadb ignores a functionality, of course it gets spoilt!
[09:36:23] <jynus>	 (but later they want to boast about it on the marketing materials)
[10:00:35] <marostegui>	 It is crazy how big otrs is
[10:02:54] <jynus>	 T138915
[10:02:55] <stashbot>	 T138915: OTRS database is "too large" - https://phabricator.wikimedia.org/T138915
[10:03:29] <jynus>	 https://i.imgflip.com/6lxa3s.jpg
[10:04:58] <marostegui>	 XDD
[10:05:02] <marostegui>	 yeah I am aware of that ticket
[11:49:22] <jynus>	 I am going to take some air between meetings, take a break/lunch outside
[13:10:25] <marostegui>	 Amir1: you can go for db1132 (it is pooled and warm)
[13:11:11] <Amir1>	 sure
[13:27:49] <Amir1>	 marostegui: https://people.wikimedia.org/~ladsgroup/mariadb_flamegraphs/superbusy-with-p_s.mem.106.svg
[13:28:14] <Amir1>	 died with 600 but took longer. Maybe some sort of "hanging connection"?
[13:28:26] <Amir1>	 otherwise the connection should close and be done
[13:28:41] <marostegui>	 we can try disabling also thread_instrumentation as jaime suggested
[13:29:10] <marostegui>	 I will do that after our meeting
[14:31:35] <marostegui>	 root@db1132.eqiad.wmnet[(none)]> UPDATE performance_schema.setup_consumers
[14:31:35] <marostegui>	     -> SET ENABLED = 'NO' WHERE NAME LIKE 'thread_instrumentation';
[14:31:35] <marostegui>	 Query OK, 1 row affected (0.000 sec)
[14:31:35] <marostegui>	 Rows matched: 1  Changed: 1  Warnings: 0
[14:31:41] <marostegui>	 Amir1: can you hit db1132 again? ^
[14:31:58] <Amir1>	 doesn't it need restart?
[14:34:59] <marostegui>	 Doesn't look like
[14:35:04] <marostegui>	 | thread_instrumentation           | NO      |
[14:36:56] <Amir1>	 awesome
[14:37:43] <jynus>	 yeah, enabling/disabling it fully requires a restart, but most instrumentation tuning is dynamic
[14:42:57] <Amir1>	 died again :(
[14:45:56] <marostegui>	 600 too?
[14:46:27] <Amir1>	 yeah
[14:46:31] <jynus>	 (if you don't mind - doesn't have to be today- tracking all tests done on the ticket (apologies if already done)? That way we can not lose track of them 0:-)
[14:46:42] <marostegui>	 Yeah, I was just writing that
[14:46:54] <jynus>	 ah, ok sorry, didn't mean to pressure 
[14:46:54] <akosiaris>	 600 only? I was hoping it would be over 9000 !
[14:47:19] <akosiaris>	 and with that useless comment, I am gonna show myself the door ;-)
[14:47:20] <Amir1>	 it's not even really 600 concurrent threads, it's lower
[14:48:08] <jynus>	 akosiaris: "Amir1, what does the concurrency say about mariadb's regression": https://www.youtube.com/watch?v=SiMHTK15Pik
[14:48:10] <Amir1>	 all from cumin1001, maybe we make a mess in network because of that
[14:48:53] <Amir1>	 aaaah, I got the reference now
[14:48:57] <jynus>	 lol
[14:50:20] <marostegui>	 https://phabricator.wikimedia.org/T311106#8055788
[14:50:21] <jynus>	 akosiaris: could I get 5 minutes of your time later this week for some rubber duck session about bacula?
[14:50:23] <Amir1>	 https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1132&var-port=9104&from=now-1h&to=now&viewPanel=10
[14:50:49] <akosiaris>	 jynus: can we say next week, say Monday instead? This is a tough week for me.
[14:50:54] <jynus>	 yeah
[14:51:29] <jynus>	 it won't be anything weird I have some doubts and need to talk with someone about them, and who's better :-D
[14:51:53] <jynus>	 also permision to break it, but we won't talk about that