[00:03:13] (DiskSpace) resolved: Disk space puppetmaster1001:9100:/ 5.16% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=puppetmaster1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [08:42:09] here comes Netbox 3.6 - https://github.com/netbox-community/netbox/releases/tag/v3.6-beta1 [08:51:04] a couple of annoying breaking changes but some sweet new features too [08:51:10] yeah I saw them yesterday [08:58:14] I guess it's significant enough to be one of someone's next Q goals now [10:54:11] 10SRE-tools, 10Infrastructure-Foundations, 10Spicerack: Spicerack: don't write logs to disk - https://phabricator.wikimedia.org/T342079 (10Volans) Those are various different requests: 1) `logger.debug("cfssl output: %s.", str(cfssl_raw.stdout))` is part of the cookbook, if you don't want it logged to disk... [13:16:27] XioNoX: hi! wb [13:16:53] I wanted to discuss with you if we should consider upgrading anycast-hc to 0.9.1. we are currently running 0.8.2 in prod [13:17:15] three reasons I guess: 1) we are upgrading Traffic hosts to bookworm so have to rebuild the package and might as well upgrade the upstream version with that [13:17:29] 2) we already behind the current release so might be helpful to catch up at some point [13:19:01] 3) https://github.com/unixsurfer/anycast_healthchecker/commit/c7af5e67eea51344429621de6af7b36e6cbfad9d might be helpful, where we can override bird_reconfigure_cmd (so basically this will call a script instead of the `bird configure' and we can do additional things instead of just reconfiguring bird) [13:19:12] let me know your thoughts on this and if you think it's a good idea, I will file a task! [13:28:02] sukhe: +1 [13:28:10] all good reasons [13:28:55] XioNoX: ok! thanks [13:29:08] will be working on that shortly and testing the durum hosts with it [13:29:15] then eventual rollout to the dnsrecs one [13:30:33] latest release is 2020, I think the tool is feature complete :) [13:30:42] :) [13:31:46] sukhe: https://github.com/unixsurfer/anycast_healthchecker/pull/26#issuecomment-1663991320 [13:34:00] sukhe: do you think https://github.com/unixsurfer/anycast_healthchecker/pull/28 would be useful to us? [13:34:03] XioNoX: you want us to pull this in? [13:34:28] the first one I meant, dummy prefix [13:34:31] sukhe: nah, I'd prefer it to be merged/tested first [13:34:53] yeah, without unixsurfer's review, doesn't inspire confidence [13:35:52] yeah and we don't have a strict need for it, it would just be cleaner [13:35:55] so better to wait [13:36:10] for the 2nd one I don't know if it's worth pinging them on it [13:36:29] for now when we do maintenance we shutdown bird [13:36:54] with this we could instead shutdown anycast-hc [13:37:05] and the session would stay up [13:37:09] but not a big change [13:37:43] but do we actually want the session to stay up? [13:38:24] less alerting noise I guess [13:38:45] and maybe slightly smoother [13:39:21] hmm I guess I am seeing it from the angle of that I actually don't want the session to be up and to be alerted when we shut down bird [13:41:05] yeah session up doesn't mean prefixes being advertised [13:41:34] anyway, I left a comment on https://github.com/unixsurfer/anycast_healthchecker/pull/28#issuecomment-1664005190 and we can discuss it the day it's merged :) [13:41:50] ok [13:42:28] I am going to update https://gerrit.wikimedia.org/r/plugins/gitiles/operations/debs/python-anycast-healthchecker/ [13:42:39] but thanks, that's all I needed from you, to check if this is fine to upgrade upstream [13:42:42] the rest is on me [13:43:03] I thought you might have some other context that I didn't; I discussed this briefly with topranks when you were out [13:45:49] sukhe: thanks a lot! [13:47:06] duck duck go doesn't know me well enough, if I search for "bird" it shows me picture of the animal... [13:47:20] an actual feature and not a bug! :) [13:47:39] "25.4.2023 BIRD 1 end of life will happen at the end of 2023. Please finish you upgrade to BIRD 2 ASAP!" [13:47:41] If I search for "free range routing" it shows me a chicken :P [13:48:27] topranks: maybe you searched for free range chickens before :) [13:48:34] hahaha [13:49:01] if only FRR used bird under the hood... :) [13:49:19] :D [13:49:56] sukhe: I was thinking of the issue where bird is not advertising prefixes to the routers while it should, and required a bird restart. I'm wondering if it could be related to the bookworm upgrade [13:50:32] XioNoX: no hosts are on bookworm right now though [13:50:43] or do you mean you think that might help if we upgrade to bookworm? [13:51:28] well, now, maybe :) [13:51:54] XioNoX: the most weird thing I guess is that we only see them on durum and doh hosts [13:51:57] not on dnsrec [13:52:11] the stack is the same, the only difference is that durum and doh do IPv6 anycast as well [13:52:37] but the prefixes that are not advertised are mostly IPv4 [13:53:36] we're on 2.0.7, latest in bookworm or bullseye-backport is 2.0.12, latest upstream is 2.13 (similar to 2.0.13, just versioning change) [13:53:53] which is quite nice [13:54:58] I'll skim through https://gitlab.nic.cz/labs/bird/-/blob/v2.13.1/NEWS see if anything can be related [13:55:36] we can build any version we want by backporting it, if there is a compelling reason [13:55:46] otherwise we will just stick to the version in debian, for ease of maintenance [13:55:53] but yeah, if we see something we really like, we will backport and build it [13:59:51] sukhe: eh all release have either "Important bugfixes" or "Many bugfixes" [14:04:16] so yeah, depending on how difficult it is, it might be worth upgrading to the latest bullseye-backport version [14:05:24] bookworm is 2.0.12, same as bullseye-backport [14:05:28] https://packages.debian.org/bookworm/bird2 [14:06:58] yep [14:26:59] topranks: https://www.ripe.net/ripe/mail/archives/routing-wg/2023-June/004748.html the BGP world is busy! [21:07:22] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, 10collaboration-services, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10thcipriani) Noticed today that display names changed to using `cn` instead of `uid` (discussed back in {T288392}): {F37163889 siz... [21:30:03] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, 10collaboration-services, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10bd808) >>! In T320390#9067911, @thcipriani wrote: > Noticed today that display names changed to using `cn` instead of `uid` (discu... [22:32:10] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, 10collaboration-services, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10brennen) To summarize discussion from Slack and libera.chat `#wikimedia-gitlab`: - Apart from T343485, we don't believe this has... [23:53:13] (DiskSpace) firing: Disk space puppetmaster1001:9100:/ 5.947% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=puppetmaster1001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace