[07:39:05] <godog>	 greetings
[08:10:57] <dhinus>	 morning
[08:21:23] <dcaro>	 morning!
[10:00:47] * dcaro relocating and lunch
[14:55:40] <godog>	 neat yeah I'd expect puppet to eventually converge
[14:55:54] <dcaro>	 waiting for a full passing nova-fullstack run
[14:58:05] <dcaro>	 hmpf... we got slow operations
[14:58:08] <dcaro>	 (in ceph)
[14:58:11] <andrewbogott>	 ceph seems to have 0 complaints, is that because the outage was only
[14:58:17] <andrewbogott>	 argggg I cursed us by starting to type that
[14:58:25] <dcaro>	 "slow operations indication"
[14:58:43] <andrewbogott>	 to be fair there were already slow ops weren't there?
[14:59:03] <dcaro>	 oh, true, do we have a graph somewhere to see if they are new?
[14:59:37] <dcaro>	 but the cluster was not in warning no/
[14:59:37] <dcaro>	 ?
[14:59:44] <dcaro>	 so we passed the new threshold?
[15:00:28] <andrewbogott>	 you're probably right, it was clear and now is re-alerting
[15:00:33] <andrewbogott>	 but it was in that same state a week ago...
[15:00:38] <andrewbogott>	 if you want reading material, https://phabricator.wikimedia.org/T429387
[15:00:41] <dcaro>	 I think this is not valid anymore xd
[15:00:44] <dcaro>	 https://usercontent.irccloud-cdn.com/file/SW9zVrmD/image.png
[15:01:02] <dcaro>	 openstack apis are recovering though
[15:01:11] <andrewbogott>	 Basically, that alert shows how many slow ops were detected /in the last 24 hours/ so even if everything is working fine it will show as unhealthy for a day
[15:01:13] <dcaro>	 and nova-fullstack did a full run
[15:01:28] <dcaro>	 hmmm, that sounds kinda annoying
[15:01:34] <dcaro>	 better than nothing though
[15:01:36] <andrewbogott>	 it is, we need to change the thresholds
[15:01:42] <andrewbogott>	 something to pick up from riccardo I think
[15:05:46] <dcaro>	 yep, I think that the stats for slow ops might not be there anymore :/
[15:05:55] <dcaro>	 (maybe the meaning changed or something)
[15:07:21] <dcaro>	 I think `BLUESTORE_SLOW_OP_ALERT` is not the same that metric is counting
[15:09:03] <dcaro>	 that means this might not be valid anymore https://gerrit.wikimedia.org/g/operations/alerts/+/6a42a3bb9b68c81198e5e58e84a597d593aff61b/team-wmcs/ceph_cluster.yaml
[15:15:36] <andrewbogott>	 you think it only reports the 'recently seen' slowops and not current/immediate slow ops?
[15:16:33] <dcaro>	 hmmm.... this is probably not as it should go
[15:16:36] <dcaro>	 https://www.irccloud.com/pastebin/M5UyQPqD/
[15:16:48] <dcaro>	 I suspect that gathering metrics from ceph is broken, or half-broken at least
[15:17:32] <andrewbogott>	 looks like it
[15:17:33] <dcaro>	 I mean, some things are getting there for sure, it shows the status as warning, and some other metrics
[15:17:46] <dcaro>	 like misplaced objects
[15:17:47] <andrewbogott>	 can you make a ticket about that?
[15:17:51] <dcaro>	 +1
[15:18:02] <dcaro>	 oh, might be fetching it from "not the primary" monitor
[15:20:02] <dcaro>	 andrewbogott: https://phabricator.wikimedia.org/T430846 feel free to add/rename/rewrite stuff
[15:20:09] <dcaro>	 I'll dump what I've looked into so far
[15:20:38] <andrewbogott>	 thank you!
[18:38:25] * dcaro off
[18:39:00] <dcaro>	 Raymond_Ndibe: I left some MRs untested, but feel free to review code-wise if you want/have time (mentioned it in the MR themselves, starting with jobs-api!327)
[18:45:29] <dcaro>	 (I have to recreate my lima-kilo, that got butchered, to test)
[18:45:31] <dcaro>	 cya!