[06:27:38] !issync [06:27:38] Syncing #wikimedia-sre [06:27:40] Set /cs flags #wikimedia-sre legoktm +Afiortv [06:27:42] Set /cs flags #wikimedia-sre manuel +Aiotv [06:56:21] mr Brooker's blog always a good read https://brooker.co.za/blog/2021/05/24/metastable.html [07:48:21] godog: good read, thx! [07:52:14] indeed, other posts also recommended [08:26:21] nice! thanks for sharing [09:29:05] Krinkle: per https://phabricator.wikimedia.org/T282761#7110427, you can now start the purge process against pc1007 [12:31:47] Can someone take a look at urldownloader1002? it is filling up and I don't know much about these sort of hosts [12:32:10] moritzm: I saw you logged in recently, maybe you could help? ^ [12:33:40] ah, the age-old you-touched-it-last-it's-yours-for-life rule in action. ;) [12:34:05] kormat: I actually saw a procurement task from him! so he must be guilty! [12:34:31] I'm not near my laptop currently, but it might be one of the older, under dimensioned VMs with 10G diskspace, maybe try cleaning up unused kernel packages [12:34:37] sentenced to 20 years hard labour *bangs gavel* [12:34:52] moritzm: yeah, it has just 8G [12:34:59] other than that, urldownloaders simply run squid, which should take little resources by itself [12:35:36] cleaned up some kernels, we are back at 79% [12:35:46] ack, those will be recreated with bigger disks when moving to bullseye [12:35:50] thanks moritzm! [12:37:02] removing _all_ kernels will also fix problems with disk space [12:37:52] either that, or replacing the Linux kernel with Hurd [12:38:41] or just if we use windows on our servers, we won't have any issues with linux kernal taking any disk space [12:39:04] crowd-sourced innovation. keep 'em coming folks [12:43:35] sell Linux to Andrew Lee and hope that after a couple of years he sells it to a Israeli company [12:46:09] let's not make a solution where we want to move off linux basically overnight [12:48:24] bsd is waiting... :-P [12:49:36] Berkeley Subsystem for DOS? Works great on Windows I hear. [13:14:00] kormat: ack, thx [15:27:48] i cannot remember how to set up custom icinga checks in puppet [15:28:05] i want to run a command and check its retval [15:28:24] do I have to do a bunch of things on the icinga server to define the check? [15:28:30] there's something simpler...right? [15:29:19] the icinga motto: "there _must_ be a better way" [15:30:38] nrpe::monitor_service ? [15:30:44] with shell wrapper for command declared on host? [15:31:40] ottomata: yeah, sounds right [15:31:47] e.g. that's what we use here: modules/mariadb/manifests/monitor_readonly.pp [15:32:31] ok and db-check-health with --icinga just spits out the icinga expected output? [15:32:37] OK: ... exit 0 etc? [15:33:36] yeah. example output: `Version 10.4.18-MariaDB-log, Uptime 7371352s, read_only: False, event_scheduler: True, 3537.01 QPS, connection latency: 0.003155s, query latency: 0.000415s` [15:36:50] ok thanks [15:37:12] ottomata, the check must return 0, 1, 2 or 3 [15:37:18] aye [15:37:28] which is ok, w, c, u [15:37:38] and the firt line of stdout goes to icinga [15:37:52] and I think the second and others goes to icinga too, but more hidden [15:39:27] ok great [15:39:46] but check as there may be both icinga and puppet wrappers if it is a common task [15:40:16] i just need to run a command and alarm if retval not 0 [15:40:20] haven't seen that yet from my perusing [15:41:27] yeah, either add what kormat said or if you don't want to touch the underling functionality a small wrapper [17:24:15] kormat: ok, so I'll be hammering pc1007 momentarily [17:30:59] Spot check: pc1007/parsercache/pc208 contains 1,395,440 rows of which 376,626 where exptime less than .. [17:31:02] that looks right [17:32:08] I'll purge a day extra considering it takes a while to do all this purging and compacting [20:29:16] Successfully added .*@wikimedia/Martin-Urbanec [20:29:16] @trustadd .*@wikimedia/Martin-Urbanec admin [21:28:20] cdanis: got another klaxon patch for you - sorry for the inconvenience :( [21:34:22] ma: no problem at all, thanks for the fixes [21:46:08] cdanis: you're welcome. The lead dev of Kiwi IRC said he's leaving IRC completly due to the freenode/libera drama though. Not sure what'd happen with kiwiirc though.