[01:15:39] * bd808 off
[02:17:46] <andrewbogott>	 tools-db alerted but seems fine, I'm not sure what's up
[02:23:58] <andrewbogott>	 Well folks that's three outages within 8 hours, only two more and we win a stuffed sonic plushie
[03:28:26] <andrewbogott>	 New version of the grid-killing scripts at https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/2  <- new pr because I can't get used to the gitlab patch model
[04:30:38] * dhinus paged: the cloudvirt node cloudvirt1063 is unreachable
[04:31:07] <andrewbogott>	 dhinus: I took it offline and tried to silence it but apparently failed.
[04:31:56] <dhinus>	 no prob
[04:33:04] <dhinus>	 I'll go back to sleep :)
[04:33:15] <andrewbogott>	 sorry for the rude awakening
[09:42:01] * dhinus paged: ToolsToolsDBWritableState
[09:43:49] <dhinus>	 restarted toolsdb and set to read-write
[09:45:25] <dcaro>	 did we get any logsV
[09:45:26] <dcaro>	 ?
[09:45:59] <dhinus>	 nope, because the tools-db-1 VM was restarted 7 hours ago :( and I didn't get to add the "tee" yesterday
[09:46:07] <dhinus>	 I will do it today
[09:47:34] <dhinus>	 the cloudvirt1063 alert is still firing
[09:49:57] <dhinus>	 and phaultfinder created not just one but two phab tasks about it :)
[09:51:43] <dhinus>	 can I set it to "failed" in netbox? I don't remember if that's the correct procedure
[09:51:48] <dhinus>	 a.ndrew has already created a task for DCops
[09:52:11] <dcaro>	 any difference between them? (last time the issue was that the project was wrong, so it would get automatically changed, and then when trying to check if it already existed it did not find it again)
[09:53:20] <dcaro>	 re setting as failed, I think so yes https://wikitech.wikimedia.org/wiki/Server_Lifecycle#Active_-%3E_Failed
[09:54:30] <dhinus>	 set to failed
[09:54:49] <dhinus>	 tasks are T353406 and T353409
[09:54:49] <stashbot>	 T353406: NodeDown cloudvirt1063 - https://phabricator.wikimedia.org/T353406
[09:54:50] <stashbot>	 T353409: NodeDown  Node cloudvirt1063 has been down for long. - https://phabricator.wikimedia.org/T353409
[09:55:01] <dhinus>	 the second one is "unreachable for more than two hours." and is critical instead of page
[09:58:36] <dcaro>	 we have a special alert it seems for NodeDown for cloudvirts only, but we are not filtering them out from the `down for too long` one yep
[10:18:33] <moritzm>	 FYI, I'm running the sre.puppet.sync-netbox-hiera cookbook and it's showing me that cloudvirt1063 is faile in the diff, given backscroll I'll merge that along
[10:24:08] <dhinus>	 moritzm: thanks, should I run that cookbook when I set a node to "failed"?
[10:32:33] <dcaro>	 fixed all the puppet-alerts stuff... going to the library, be back in a bit
[11:46:45] <moritzm>	 dhinus: I think so, yes. I'm not 100% sure myself what it's used for
[12:45:07] <topranks>	 yeah it's best to run it, we sync the netbox data for hosts that are status "active", if they change to or from that the sync will remove the associated data from hiera (host name, rack location etc.)
[12:45:36] <topranks>	 I'm not sure how widely that synced-data is used by puppet roles but best to not leave an outstanding diff that might confuse the next person 
[13:33:51] <komla>	 If a tool is stopped and there are no changes to the code or config files, are there any other files in the tool directory (ie /data/project/*) that could have its modification time changed/updated?
[13:34:12] <komla>	 I'm trying to sort the tools according to their most recently updated timestamp.
[13:34:12] <komla>	 1:21 PM 
[13:34:12] <komla>	 When the time time comes to turn them off, I will then start from the 'oldest'
[13:41:35] <dcaro>	 we manually fixed also a bunch of kubernetes configurations at some point, and similar stuff
[13:41:43] <dcaro>	 that might changes some timestamps around
[13:42:13] <dcaro>	 I'd say it's "approximately correct", that might be good enough for what you want to do :)
[13:43:01] <dcaro>	 though make sure you don't get weird values (like all the tools were updated today, or there's some tools that were not updated since 1970)
[13:50:22] <dhinus>	 is there a way to set cloudvirt1063 to "maintenance" to disable all the alerts? or should I just silencce all the alerts in alertmanager+icinga?
[13:51:07] <dhinus>	 I guess the canary alert can be fixed by deleting the canary in horizon
[13:52:38] <dcaro>	 I think that you can move the host out of the ceph aggregate or similar, that might work for a few of the alerts, but I think not all
[13:53:51] <dhinus>	 I'll try that
[13:54:21] <dhinus>	 I'm also seeing 4 separate "NodeDown" alerts in alertmanager
[14:21:43] <andrewbogott>	 Was my main flaw setting the downtime on <hostname> rather than <hostname>*?  If did a nice auto-complete for the former so I foolishly thought that something had changed and that that would work. (And I also didn't anticipate the sneaky 2-hour delay page)
[14:23:12] <komla>	 dcaro: thanks!
[14:24:00] <andrewbogott>	 Also dhinus I did move it out of the ceph aggregate didn't I?
[14:24:57] <dhinus>	 andrewbogott: haven't checked yet, I'm debugging some alerting stuff with dcaro 
[14:25:30] <andrewbogott>	 ok
[14:26:04] <dhinus>	 did you downtime on alertmanager or icinga or both?
[14:26:52] <dhinus>	 I think the page came from alertmanager this time, and I can see the "page" alert is acked by andrewbogott 
[14:26:56] <andrewbogott>	 alertmanager
[14:27:20] <dhinus>	 maybe you acked it after the alert triggered and sent the page?
[14:27:38] <dhinus>	 the ACK! is at 4:31 GMT
[14:28:31] <dhinus>	 the silence is on cloudvirt1063 so I suspect that like you said it was missing the * at the end
[14:29:02] <andrewbogott>	 that sounds right
[16:14:27] <bd808>	 andrewbogott: "new pr because I can't get used to the gitlab patch model" -- I tend to force push to the dev branch. That makes it mostly like the `git review --no-rebase` command I mostly use with Gerrit.
[16:15:26] <bd808>	 GitLab can also show you diffs between force pushes done like that which makes the review experience a bit more like my typical Gerrit review workflow too.
[16:15:41] <andrewbogott>	 Oh yeah, I actually tried to force push but it wouldn't let me
[16:15:49] <andrewbogott>	 maybe because I was on a main branch rather than a topic branch?
[16:16:04] <andrewbogott>	 But anyway, it's good to know that it's technically possible to do things the way I like to do them!
[16:16:15] <andrewbogott>	 komla: 
[16:16:16] <bd808>	 ah, could be yeah. the primary repo's branch protection might copy to your clone
[16:16:24] <andrewbogott>	 https://www.irccloud.com/pastebin/ZkyhFso6/
[16:16:40] <andrewbogott>	 ^ poorly-formatted instructions for de- and re-gridding a tool
[16:16:58] <andrewbogott>	 I will also send an email to the list with all that once you're convinced it works
[16:17:46] <bd808>	 I have been trying to stick with starting a `work/bd808/<topic>` branch for each set of changes whether I am working in the primary repo or a fork. It's just easier for my brain to stick with a pattern.
[16:18:44] <andrewbogott>	 Yep, that's better than just immediately mangling the main branch which is what I did. I guess that only really makes sense if I were going to make a new fork for every patchset.
[16:23:41] <dhinus>	 dcaro: quick review? https://gerrit.wikimedia.org/r/c/operations/alerts/+/983156
[16:23:43] <dcaro>	 I usually use a <topic> branch, no matter if I'm on my clone or the original repo. And sometimes add more than one commit if it makes sense (ex. small refactor + feature that uses it or similar)
[16:30:01] <dhinus>	 ToolsDB crashed again. I'm restarting it.
[16:30:05] <dhinus>	 this time we should have logs
[16:32:47] <dcaro>	 \o/
[16:36:10] <bd808>	 OIT says my new laptop should arrive just in time for me to take the end of year break. Looks like I might spend a good chunk of quiet week setting up a new work laptop for the first time in ~5 years. Moving to the M* series macbooks feels like a good time to actually setup deliberately rather than just restoring my prior backup on new hardware.
[16:44:30] <taavi>	 is anyone looking at the open toolforge quota request?
[16:47:47] <dcaro>	 I don't think so
[16:48:10] <andrewbogott>	 I'm not (and the weekly etherpad is empty?)
[16:51:01] * andrewbogott makes the etherpad
[17:46:37] <taavi>	 andrewbogott: the disable-tool cron has started failing on tools-sgecron-2 at least https://phabricator.wikimedia.org/P54454
[17:53:01] <andrewbogott>	 argh
[17:53:05] <andrewbogott>	 thanks taavi, I'll look
[17:53:13] <taavi>	 tox
[17:53:17] <taavi>	 oops wrong window
[18:30:28] <taavi>	 https://gitlab.wikimedia.org/repos/cloud/toolforge/disable-tool/-/merge_requests/4
[23:00:00] <andrewbogott>	 toolsdb will never let us rest
[23:00:57] <andrewbogott>	 how is it that I don't get paged until 20 minutes after it goes down?