[00:55:13] <bd808>	 T358538 and related https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/210 could use +1, review, and deploy when someone gets a chance. This will unblock an attempt to make the wikibugs irc process more stable by putting a bouncer between the bot and libra.chat.
[00:55:14] <stashbot>	 T358538: Request increased quota for wikibugs Toolforge tool - https://phabricator.wikimedia.org/T358538
[01:00:18] * bd808 off
[07:05:28] <XioNoX>	 are those IPs in use https://netbox.wikimedia.org/ipam/ip-addresses/10716/ and https://netbox.wikimedia.org/ipam/ip-addresses/10718/ ? Comment says "recursor for codfw1dev VMs" but they don't reply to pings, so maybe we just forgot about them after a migration ?
[08:00:29] <taavi>	 morning
[08:02:02] <taavi>	 XioNoX: yep, I think those are outdated since we moved to an anycasted cloud-private address, https://gerrit.wikimedia.org/r/c/operations/dns/+/930808
[08:04:04] <taavi>	 XioNoX: https://netbox.wikimedia.org/ipam/ip-addresses/3519/ is also no longer responding to any traffic, although with that one you still have the risk of outdated clients trying to use it
[08:37:49] <taavi>	 dcaro: Raymond_Ndibe: https://gerrit.wikimedia.org/r/c/operations/puppet/+/998659 doesn't seem to have worked as expected, T358552 seems to still have failed due to ` pwrite() "/var/lib/nginx/body/0000327926" failed (28: No space left on device)`
[08:37:49] <stashbot>	 T358552: Rust image build on toolforge fails - https://phabricator.wikimedia.org/T358552
[08:38:11] <taavi>	 oh and that even seems to be a dupe of T354116 from December
[08:38:11] <stashbot>	 T354116: Harbor uploads sometimes fail due to tmpfs space on project-proxy - https://phabricator.wikimedia.org/T354116
[08:51:43] <taavi>	 another tools nfs disk space alert, another massive log file by the same name in the same tool as the last time T358555
[08:51:44] <stashbot>	 T358555: cewbot k8s-20230418.fix-redirected-wikilinks-of-templates.out is unreasonably large - https://phabricator.wikimedia.org/T358555
[13:03:58] <taavi>	 I am merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/1006492, in case you see any cloudlb or bgp alerts that's me
[13:18:57] <XioNoX>	 taavi: noted about the IPs, thx ! I'll remove them from Netbox
[18:44:41] <andrewbogott>	 Designate shuffling is done and everything seems to still work. Please let me know if you see bad behavior.
[19:23:31] * bd808 lunch