[00:13:29] <icinga-wm>	 RECOVERY - Check unit status of swift_ring_manager on thanos-fe1001 is OK: OK: Status of the systemd unit swift_ring_manager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[03:16:41] <icinga-wm>	 PROBLEM - Check unit status of swift_ring_manager on thanos-fe1001 is CRITICAL: CRITICAL: Status of the systemd unit swift_ring_manager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[06:51:09] <marostegui>	 Going to switchover es1, es2 and es3 masters (RO sections()
[06:52:06] <jynus>	 are es4, es5 hosts 10.6?
[06:52:53] <marostegui>	 jynus: there is one 10.6 host in es4 and none on es5
[06:53:03] <marostegui>	 The es1, es2 and es3 are just needed for kernel reboots
[06:53:29] <jynus>	 becase I am having lately issues with backups
[06:53:52] <marostegui>	 only es1022 is 10.6
[06:53:58] <marostegui>	 what sort of issues?
[06:54:39] <jynus>	 for example, yesterday all 4 backups on both eqiad and codfw failed at the same time, a few minutes after 0 hours (When it starts)
[06:55:05] <marostegui>	 jynus: That's weird, that host has been there for weeks though
[06:55:21] <jynus>	 then it is not related to 10.6
[06:55:21] <marostegui>	 So es4 has 1022 and 2022 with 10.6
[06:55:27] <marostegui>	 the rest are 10.4
[06:55:34] <marostegui>	 es5 has 0 10.6
[06:55:43] <jynus>	 but it is weird all failed, specially both dcs at around the same time
[06:55:55] <marostegui>	 network stuff perhaps?
[06:55:57] <jynus>	 the hosts also didn't show anything special
[06:56:25] <jynus>	 network seems weird on both dcs at the same time
[06:56:37] <marostegui>	 but both hosts too
[06:56:51] <marostegui>	 was it at the time of codfw power outage?
[06:57:03] <jynus>	 no, it was later
[07:22:31] <jynus>	 I have sent you a reminder invitation
[07:22:50] <marostegui>	 got it
[07:22:58] <jynus>	 this is in case a truck runs over me, so we remember to reenable backups for es
[07:23:39] <jynus>	 they have been disabled because they are running now, and if we let them with the original schedule they will backup the same run from last week, so I disable and enable tomorrow
[07:26:27] <marostegui>	 XDDDDDD
[07:26:34] <marostegui>	 I hope I don't need to use that invitation
[08:07:03] <jynus>	 backups failed again, this time, for now, only on codfw
[08:07:22] <marostegui>	 mmm 10.6?
[08:08:01] <jynus>	 one at least
[08:11:10] <jynus>	 https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-job=All&var-server=es2022&var-port=9104&from=1655880489262&to=1655880988695
[08:12:56] <marostegui>	 nothing on the logs at a first glance
[08:12:59] <marostegui>	 crashes, reboots etc
[08:15:36] <jynus>	 maybe, for some reason the db overload if recently restarted or something
[08:15:48] <jynus>	 hit ratios was low, but that is expected for a dump
[08:16:00] <marostegui>	 it was rebooted a week ago for kernel upgrade
[08:18:07] <jynus>	 or maybe hitting some kind of corruption or hw failure
[08:18:36] <jynus>	 nothing on dmesg
[08:19:31] <jynus>	 so no worries on backup side (other than being anoying) but wondering if this is another data point regarding 10.6
[08:20:24] <jynus>	 after all- the error is the same (connection lost/unavailable) and this is a very slow stress test
[08:20:57] <marostegui>	 But that host has been there for weeks
[08:21:20] <marostegui>	 Since May 26th from what I can see
[08:21:37] <jynus>	 I am speaking facts, not saying they make a lot of sense :-/
[08:22:00] <marostegui>	 yeah :(
[08:26:30] <jynus>	 on the good side, we are getting free memory from space vacuum: https://phabricator.wikimedia.org/P29958
[08:26:54] <jynus>	 -2GB for background processes!
[08:28:35] <jynus>	 and that is not our custom sys, it is p_s
[08:29:05] <jynus>	 I suggest one of the first things to try is to disable p_s, I doubt mariadb tests with p_s=on :-(
[08:29:43] <marostegui>	 you mean for https://phabricator.wikimedia.org/T311106
[08:29:44] <marostegui>	 ?
[08:29:51] <jynus>	 yeah
[08:29:56] <marostegui>	 Good point
[08:30:00] <marostegui>	 I have one doubt though
[08:30:20] <marostegui>	 Not sure how to proceed with all the tests, ie: we disable p_s, and should I repool the hosts and wait to see their behaviour?
[08:30:24] <jynus>	 negative memory counts should be a trivial bug, but who knows what else could be broken there
[08:30:30] <marostegui>	 Otherwise we can do many things and end up not knowing which one was it
[08:30:40] <jynus>	 marostegui: I would first keep things the same to reproduce the issue
[08:31:06] <jynus>	 then I am trying to see if this bug is reported on jira
[08:31:07] <marostegui>	 Let me add the suggestion to the task for now
[08:33:05] <marostegui>	 We should probably have a small meeting with Amir.1 once he's back next Monday to see how we can try to reproduce this simulating MW 
[08:33:31] <jynus>	 ticket implementation: https://jira.mariadb.org/browse/MDEV-16431
[08:33:45] <jynus>	 and: https://jira.mariadb.org/browse/MDEV-6114
[08:33:46] <marostegui>	 haha simon!
[08:33:48] <jynus>	 new in 10.5
[08:34:01] <jynus>	 but lots of comment regarding memory leaks/increase usage
[08:34:09] <jynus>	 I think we contained that by tuning the defaults
[08:34:36] <marostegui>	 I have noticed weird memory issues though
[08:34:39] <marostegui>	 But of course who knows
[08:34:57] <jynus>	 https://jira.mariadb.org/browse/MDEV-23936
[08:35:39] <jynus>	 "it has a reproducible (on Windows, or maybe generally with threadpool)"
[08:36:02] <jynus>	 p_s and the pool where the 2 things I suspected first
[08:36:17] <marostegui>	 I never suspected about p_s
[08:36:37] <jynus>	 in any case, vote for that issue if you can
[08:36:57] <marostegui>	 Done!
[08:36:59] <jynus>	 memory instrumentation is useful, but not when it returns negative numbers
[08:37:03] <marostegui>	 I can try to ping valerii as well
[08:37:09] <marostegui>	 And see what he saw
[08:37:49] <jynus>	 I will add a comment
[08:38:00] <marostegui>	 +1
[08:38:11] <marostegui>	 For now I have depooled all 10.6 from s1, s4, s7 and s8
[08:38:13] <jynus>	 is 10.6.7 the latest version of that branch?
[08:38:20] <marostegui>	 We are running 10..68
[08:38:23] <marostegui>	 10.6.8
[08:38:26] <jynus>	 ah
[08:38:28] <jynus>	 well, that
[08:38:47] <marostegui>	 that is the latest
[08:43:42] <jynus>	 sorry, I should have been more explicit
[08:44:05] <jynus>	 I meant to test with p_s disabled as one possible factor
[08:44:19] <jynus>	 not necesarilly disable things now
[08:45:02] <jynus>	 that and the pool of threads- even if losing both would be a huge loss for us
[08:47:37] <jynus>	 e.g. last time they changed the pool of threads, 10.0? we had to tune a buffer that was too low
[08:47:38] <marostegui>	 yeah yeah 
[08:47:45] <marostegui>	 I am not changing anything now :)
[09:13:22] <icinga-wm>	 RECOVERY - Check unit status of swift_ring_manager on thanos-fe1001 is OK: OK: Status of the systemd unit swift_ring_manager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[09:26:00] <question_mark>	 i will share a google doc later today to collect potential projects for Q1 (OKR planning etc) - just as simple bullets for now, no OKR language yet.
[09:26:15] <question_mark>	 to start off that discussion for next week
[09:27:15] <marostegui>	 that reminds me I need to fill out the other doc, question_mark hehe
[09:27:17] <marostegui>	 I will note that
[09:27:25] <question_mark>	 :)
[09:27:56] <question_mark>	 yeah that's helpful for looking a bit further out and knowing what will come after
[09:29:05] <marostegui>	 question_mark: was that a spreadsheet? I cannot find it
[09:29:20] <question_mark>	 yes
[09:29:28] <question_mark>	 let me link you to the doc actually, it has a link to the spreadsheet as well
[09:29:38] <marostegui>	 ah cool
[09:29:41] <marostegui>	 thanks
[09:31:07] <question_mark>	 sent
[09:31:16] <question_mark>	 can you tell Filippo is sitting in front of me? ;)
[09:33:39] <question_mark>	 riccardo is here as well, shall I give him some db work? should prevent him from getting more rusty ;)
[09:34:08] <volans>	 lol
[09:34:36] <marostegui>	 question_mark: tell him we are hiring!
[09:34:45] * Emperor is looking forward to the day when swift nodes reboot first time reliably
[09:35:13] <question_mark>	 Emperor: how could you possibly be certain with a single day ;)
[09:35:21] <question_mark>	 within
[09:35:50] <Emperor>	 I have some robust "disaster recovery" testing plans ;-p
[10:20:58] <icinga-wm>	 PROBLEM - Check unit status of swift_ring_manager on thanos-fe1001 is CRITICAL: CRITICAL: Status of the systemd unit swift_ring_manager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[13:02:41] <jynus>	 db1124 restarted recentl, maybe? Icinga: CRIT: read_only: "True", expected "False" (disabled alerts due to being a test host)
[13:02:54] <marostegui>	 ah yes
[13:02:57] <marostegui>	 I did it this morning
[13:03:01] <marostegui>	 fixing it now
[13:03:03] <marostegui>	 thanks for the heads up
[13:03:13] <marostegui>	 done
[13:03:14] <jynus>	 no big deal
[13:03:41] <jynus>	 I have to check db* alerts as sometimes (quite often) they are backup sources and I am breaking them
[13:19:37] <icinga-wm>	 RECOVERY - Check unit status of swift_ring_manager on thanos-fe1001 is OK: OK: Status of the systemd unit swift_ring_manager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[14:36:49] <icinga-wm>	 PROBLEM - Check unit status of swift_ring_manager on thanos-fe1001 is CRITICAL: CRITICAL: Status of the systemd unit swift_ring_manager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[15:06:29] <RhinosF1>	 Hey, https://gerrit.wikimedia.org/r/c/operations/puppet/+/800739 got merged but there are issues with the column still being in the view. Is this just analytics needing to run maintain-views ?
[15:08:15] <marostegui>	 RhinosF1: it needs to run maintain-views
[15:08:25] <marostegui>	 do you have a wiki for me to check?
[15:08:30] <RhinosF1>	 marostegui: altwiki
[15:08:47] <marostegui>	 that's s3?
[15:08:53] <RhinosF1>	 https://phabricator.wikimedia.org/T311148
[15:09:03] <marostegui>	 checking
[15:09:23] <marostegui>	 s5
[15:09:24] <marostegui>	 checking
[15:09:41] <marostegui>	 mmm it works on clouddb1021:3315
[15:09:43] <marostegui>	 let me check the others
[15:09:59] <RhinosF1>	 enwiki's view looks updated
[15:10:11] <RhinosF1>	 so not sure if it got only half done
[15:10:13] <marostegui>	 yeah, 1020 is missing
[15:10:22] <marostegui>	 let me fix that
[15:10:29] <icinga-wm>	 RECOVERY - Check unit status of swift_ring_manager on thanos-fe1001 is OK: OK: Status of the systemd unit swift_ring_manager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[15:12:50] <marostegui>	 RhinosF1: fixed and closed the task: https://phabricator.wikimedia.org/T311148#8020199
[15:12:53] <marostegui>	 thanks for the heads up
[15:14:04] <RhinosF1>	 marostegui: no problem, thanks for looking so quick!
[15:24:11] <icinga-wm>	 PROBLEM - Check unit status of swift_ring_manager on thanos-fe1001 is CRITICAL: CRITICAL: Status of the systemd unit swift_ring_manager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[15:59:20] <Emperor>	 le sigh, it's always about this time of day that one node needs seemingly-endless reboots :(
[16:00:08] <Emperor>	 I'm not going to get to any more swift hosts for T310483 beyond what I've updated on there just now. Hopefully ms-be2063 will get its drives up in the right order before my taxi shows up...
[16:31:10] <Emperor>	 OK, this is taking the mickey. After a dozen reboots of sdb and sda coming up swapped, I gave in and re-labelled them the way in which they kept appearing. Since doing that, every reboot since they've come back the other way round.
[16:36:05] <Emperor>	 Ah, we may finally be there...
[16:37:11] <RhinosF1>	 Emperor: I don't think the Swift nodes want to you to finish on time
[17:14:16] <icinga-wm>	 RECOVERY - Check unit status of swift_ring_manager on thanos-fe1001 is OK: OK: Status of the systemd unit swift_ring_manager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[18:13:22] <icinga-wm>	 PROBLEM - Check unit status of swift_ring_manager on thanos-fe1001 is CRITICAL: CRITICAL: Status of the systemd unit swift_ring_manager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[21:10:03] <icinga-wm>	 RECOVERY - Check unit status of swift_ring_manager on thanos-fe1001 is OK: OK: Status of the systemd unit swift_ring_manager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[21:23:47] <icinga-wm>	 PROBLEM - Check unit status of swift_ring_manager on thanos-fe1001 is CRITICAL: CRITICAL: Status of the systemd unit swift_ring_manager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[22:20:31] <icinga-wm>	 RECOVERY - Check unit status of swift_ring_manager on thanos-fe1001 is OK: OK: Status of the systemd unit swift_ring_manager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[23:19:41] <icinga-wm>	 PROBLEM - Check unit status of swift_ring_manager on thanos-fe1001 is CRITICAL: CRITICAL: Status of the systemd unit swift_ring_manager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers