[07:16:01] <dcaro>	 sweet
[07:44:16] <arturo>	 theory: from a point of view, maintaining the container images like that is not that different from the old tools-images repo, no?
[07:45:09] <dcaro>	 some aspects are similar yes
[07:45:22] <arturo>	 semantically, is very similar, and prone to the same limitations, like version bumps and deprecations
[07:45:25] <dcaro>	 some are not (ex. not being supported by WMCS, yet at least)
[07:45:47] <arturo>	 so I wonder if an alternative is to offer just the git repo, and for each tool account to build their own
[07:48:01] <dcaro>	 I'm not sure how time-compatible building would be though, as in how long can you rebuild an old version of redis without trouble (ex. deps not existing anymore, ...)
[07:48:52] <dcaro>	 For these kind of services, I would lean on pre-built images, as the redis interface (or mariadb/memcached) does not change that often and is usually backwards compatible
[07:49:04] <arturo>	 ok
[07:49:15] <dcaro>	 with language runtimes you have many more issues as they change more often (specially if they come with libs)
[07:50:04] <dcaro>	 I'd be happy to be convinced otherwise though if you want to try it out
[08:12:41] <arturo>	 nah, that's ok. But I think it may be the perfect timing to reflect on it, before wide adoption
[08:12:54] <arturo>	 on other news, maintain-kubeusers did this yesterday:
[08:13:01] <arturo>	 https://www.irccloud.com/pastebin/CCvZ5gcG/
[08:13:18] <arturo>	 a couple of accounts `elcdb` and `lastactiveadms` were apparently deactivated
[08:13:32] <arturo>	 I guess/hope that this was OK, the result of a human action :-)
[08:25:02] <dcaro>	 https://disabled-tools.toolforge.org/ <- they show up there, though it should nod delete them yet I think
[08:25:30] <dcaro>	 it should delete them only after 40 days of being disabled
[08:26:07] <arturo>	 well, the homedir wasn't really deleted, only the `k8s.disabled` flag file was created
[08:26:16] <arturo>	 but `delete` is the name of the operation in the reconciliation loop
[08:26:40] <arturo>	 the rest of the resources, yes, they are deleted. Disabled accounts don't have access to k8s, so that should be expected, no?
[08:27:36] <dcaro>	 sounds ok to me yes, very confusing names though
[09:25:38] <taavi>	 > aggregate_instance_extra_specs:network-agent='ceph'
[09:25:45] <taavi>	 I wonder what I was thinking about yesterday when doing this
[09:26:49] <arturo>	 :-)
[09:29:10] <dcaro>	 `ceph net-agent list`
[09:29:13] <dcaro>	 xd
[09:30:48] <taavi>	 hmm is rabbitmq happy in codfw1dev?
[09:32:50] <dcaro>	 arturo: there's a bunch of duplicated resourcequotas on toolsbeta, duplicated as in each tool namespace, there's  `tool-<toolname>` and `<toolname>` resourcequotas 
[09:33:16] <dcaro>	 is that you testing something?
[09:33:23] <arturo>	 yeah, we should cleanup a bunch of stuff after the recent maintain-kubeuser works
[09:33:45] <dcaro>	 in tools there's also some, but not all
[09:34:00] <arturo>	 we briefly had a version of maintain-kubeusers deployed that would create resourcequotas with the wrong name
[09:35:01] <arturo>	 also, there are old configmaps
[09:35:03] <dcaro>	 hmm, okok, so I guess that the more restrictive version will apply, so we should cleanup before any new change will take effect right?
[09:35:16] <arturo>	 old: maintain-kubeusers, new: maintain-kubeusers-<toolname>
[09:35:33] <dcaro>	 not the config, the resourcequota itself
[09:35:53] <dcaro>	 https://www.irccloud.com/pastebin/kHBPQffS/
[09:36:36] <dcaro>	 that makes me wonder, what does toolforge jobs look for?
[09:37:10] <arturo>	 the same as the namespace name, it seems
[09:37:32] <arturo>	 well, kubernetes does, I think
[09:37:49] <dcaro>	 it just gets the first in the namespace xd
[09:37:52] <dcaro>	 resource_quota = tool_account.k8s_cli.get_objects("resourcequotas")[0]
[09:38:38] <arturo>	 oh, ok. But for actual resource scheduling calculation, I think k8s uses the namespace named
[09:39:18] <dcaro>	 ah, okok
[09:39:36] <dcaro>	 hmm, how did that work then?
[09:40:02] <dcaro>	 ahhh, okok, the ones without 'tool-' are the ones that were created by mistake
[09:40:03] <dcaro>	 xd
[09:40:12] <arturo>	 yeah
[09:46:52] <dcaro>	 hmm, it seems to apply both resourcequotas, not only the one with the same name as the namespace
[09:47:07] <arturo>	 ok
[09:47:14] <arturo>	 so the cleanup is definitely needed
[09:48:18] <dcaro>	 yep, it applies both
[09:48:33] <dcaro>	 yep, cleanup is needed, I'll do
[09:49:46] <arturo>	 thanks
[10:13:23] <dcaro>	 quick review (the cleanup script is there) https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/323
[10:14:49] <dcaro>	 I'll send another patch cleaning up the specific overrides after
[10:36:11] <arturo>	 +1'd
[10:45:14] <taavi>	 arturo: dhinus: after a round of rabbitmq restarts it seems like the new flavors are working fine in codfw1dev. a g3 flavor gets you a linuxbridge agent host, a g4 one gets you an OVS host, and both types will have working network connectivity in the same VLAN-based network
[10:45:26] <taavi>	 next up i will set up the opentofu env in eqiad1 and expand the docs for it
[10:45:40] <arturo>	 great
[10:45:57] <taavi>	 and then we can add the new flavors in eqiad1, and start moving the first hypervisors there to OVS
[10:46:07] <arturo>	 sounds good
[10:46:18] <dhinus>	 awesome
[10:50:23] <dcaro>	 taavi: were the flavour changes related to the rabbitmq restarts/
[10:50:24] <dcaro>	 ?
[10:51:06] <taavi>	 dcaro: i don't know why rabbit was unhappy, but i can't imagine why adding a flavor could have caused that
[10:51:41] <dcaro>	 ack, I was curious to know how it would do so if it did xd
[11:26:43] <dcaro>	 quick review https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/324
[11:27:09] <dcaro>	 cleaning up custom quotas that are lower than the new defaults (deployments 16, services 16)
[11:34:11] <arturo>	 dcaro: +1'd
[11:34:40] <dcaro>	 thanks
[11:45:30] * dcaro lunch
[13:15:07] <blancadesal>	 `apt-get update` fails on tools-sgebastion-10 `E: The repository 'http://mirrors.wikimedia.org/debian buster-backports Release' no longer has a Release file.`
[13:15:16] <blancadesal>	 what's the fix here?
[13:15:53] <blancadesal>	 (context: deploying the latest envvars-cli)
[13:21:30] <dcaro>	 Hmm, that host is an old host that we want to get rid of asap (replaced by the newer bookworm bastions)
[13:21:38] <dcaro>	 and that's one of the reasons xd
[13:22:17] <dcaro>	 maybe we can change the repo to the archived url? (iirc you used that somewhere)
[13:25:42] <blancadesal>	 yes, I can look into that
[13:26:30] <blancadesal>	 is this a puppet thing or can the file be edited on the host directly?
[13:27:10] <taavi>	 or just drop the repo entirely, it's not like we're going to have any new package updates from there
[13:27:54] <taavi>	 cteam: heads up: I'm modifying all existing flavors in eqiad1 to pin them to the linuxbridge agent. this should be a no-op but if you see any scheduling weirdness this might be why
[13:28:27] <dcaro>	 taavi: ack, thanks for the heads up
[13:28:47] <dcaro>	 blancadesal: you can try doing it manually and then running puppet xd (I would do that)
[13:29:41] <blancadesal>	 dcaro: you mean to check if puppet overrides it?
[13:31:14] <dcaro>	 yep
[13:31:31] <dcaro>	 if it does not, then that's ok, if it does, then we need to remove from puppet :)
[13:31:42] <blancadesal>	 ack 
[13:45:25] <andrewbogott>	 taavi: is the idea that then you'll force migration of all VMs and they'll pick up the new pin when they move?
[13:47:19] <dcaro>	 the api gateway changes to move auth there are ready for review https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/23, I'm happy to go over it if anyone wants to review it (no rush, whenever)
[13:49:04] <blancadesal>	 dcaro: puppet did not overwrite the edit, apt-get works again
[13:49:13] <dcaro>	 \o/
[13:49:53] <taavi>	 andrewbogott: the idea is that each VM will be shut down, moved from the g3.$FOO flavor to the equivalent g4.$FOO flavor (and as a part of that re-scheduled to a OVS capable host), the database bits will be flipped to enable the OVS network driver on it, and started up again
[13:51:06] <andrewbogott>	 OK, so when you said you were modifying flavors, you mean you were creating new g4 flavors
[13:52:56] <taavi>	 no, the change I just did was to prevent any VMs using the existing flavors from getting scheduled on OVS hosts, so that I can re-image a few cloudvirts to the new agent. and once I have a few cloudvirts on OVS I can create the g4 flavors
[13:54:00] <andrewbogott>	 ok ok, now I'm following :)
[14:12:56] <taavi>	 last time I get to ask this for a while: is anyone interested in running the toolforge meeting that's about in an hour? https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Monthly_meeting
[14:15:33] <dcaro>	 :),  I can run it if you don't want to run this one, but I'm happy if you run it one last time before you return after the long break (ahem...)
[14:17:13] <dhinus>	 I'd also leave the honour to taavi for one last time (but maybe not last last? :P)
[14:17:19] <taavi>	 sure, I can do it :-)
[15:04:48] <dcaro>	 topranks: I'm seeing some issues with healthchecks between the ceph nodes, is anything hapenning you know of ?
[15:06:56] <topranks>	 no, I'm doing some work elsewhere in the DC but shouldn't affect cloudsw 
[15:06:56] <topranks>	 any particular nodes firing alerts?
[15:06:56] <dcaro>	 looking
[15:06:58] <topranks>	 took a quick glance at the librenms graphs no sudden jump in usage or anything suggesting links saturated etc 
[15:07:24] <taavi>	 soo, now that we have an incident response process.. I'm declaring this an incident. starting a doc
[15:07:32] <arturo>	 ok
[15:07:36] <bd808>	 I had an open ssh session on tools-bastion-12, but trying to `become stashbot` has hung it. Feels like NFS sadness.
[15:19:05] <dcaro>	 this shows many pings lost in the last 6+ hours
[15:19:05] <dcaro>	 https://grafana-rw.wikimedia.org/d/613dNf3Gz/wmcs-ceph-eqiad-performance?orgId=1&from=now-12h&to=now&editPanel=122
[15:19:05] <taavi>	 https://docs.google.com/document/d/1epuk7WD48vQKhHsjNEjKDOLN9UU1CloeWYKxQ3rRKyo/edit
[15:19:05] <dcaro>	 https://www.irccloud.com/pastebin/8MqMIbW0/
[15:19:05] <dcaro>	 thanks taavi
[15:19:05] <dcaro>	 topranks: ^ those are failing
[15:19:05] <taavi>	 dcaro: is there anything anyone can do to help you?
[15:19:05] <dcaro>	 just trying to map which links that are down, running that command to some other cloudcephosd nodes might help figure out which switch/router is not letting the packets flow
[15:19:05] <arturo>	 we are in this meeting room https://meet.google.com/rbm-qwmf-qgf?authuser=1
[15:19:05] <topranks>	 dcaro: hmm, jumbos are the issue then?
[15:19:05] <dcaro>	 look like it, regular pings work between both
[15:19:05] <dcaro>	 anything changed? the ping started working
[15:19:05] <dcaro>	 https://www.irccloud.com/pastebin/qg0lDPD5/
[15:19:05] <taavi>	 andrewbogott: can I task you for sending a notification to cloud-announce@?
[15:19:05] <andrewbogott>	 yes, I'm trying to find the incident response docs :)
[15:19:05] <andrewbogott>	 taavi: you mean about ceph right?
[15:19:05] <arturo>	  RECOVERY - BGP status on lsw1-f3-eqiad.mgmt is OK: BGP OK - up: 22, down: 0, shutdown: 0 https://wikitech.wikimedia.org RECOVERY - BGP status on lsw1-f3-eqiad.mgmt is OK: BGP OK - up: 22, down: 0, shutdown: 0 https://wikitech.wikimedia.org
[15:19:05] <taavi>	 andrewbogott: wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/Incident_Response_Process
[15:19:05] <andrewbogott>	 ty
[15:19:05] <taavi>	 andrewbogott: yeah, since it seems like the approximate impact is "cloud vps is down"
[15:19:05] <topranks>	 yeah even stranger 
[15:19:05] <topranks>	 https://www.irccloud.com/pastebin/b1Lk3fH5/
[15:19:05] <dcaro>	 oh, that sounds to me like some paths broken somewhere
[15:19:05] <arturo>	 maybe some ongoing DCops work on site?
[15:19:05] <topranks>	 indeed
[15:24:17] <arturo>	 https://phabricator.wikimedia.org/P64632
[15:24:17] <taavi>	 topranks: fwiw, we're in https://meet.google.com/rbm-qwmf-qgf
[15:24:17] <taavi>	 and https://docs.google.com/document/d/1epuk7WD48vQKhHsjNEjKDOLN9UU1CloeWYKxQ3rRKyo/edit is the incident doc
[15:24:17] <dcaro>	 the graphs for inter-cloudswitch traffic here https://grafana-rw.wikimedia.org/d/613dNf3Gz/wmcs-ceph-eqiad-performance?orgId=1 seem suspicious, the traffic goes down
[15:24:17] <dcaro>	 slow ops are going down again, anyone did any chahnges?
[15:26:37] <dcaro>	 https://www.irccloud.com/pastebin/MCQyvEGY/
[15:26:54] <dcaro>	 from root@cloudcephmon1001:~# ceph health detail
[15:27:27] <dcaro>	 from that it seems that F4 might have been the one dropping the jumbos
[15:29:03] <dhinus>	 the NFS alert is back to green
[15:30:03] <topranks>	 I shut the links from E4 and F4 to D5 
[15:30:18] <topranks>	 and moved the instance vlan over to the links from C5 too 
[15:30:34] <topranks>	 that seems better at first glance but not sure what's going on still 
[15:30:38] <topranks>	 https://www.irccloud.com/pastebin/O9IuDEvG/
[15:30:52] <taavi>	 when did you do that? just now?
[15:30:57] <dcaro>	 https://usercontent.irccloud-cdn.com/file/kYyhPdTh/image.png
[15:34:04] <andrewbogott>	 topranks: we now think that the issue began around 01:45UTC, 16 or so hours ago.  Did you do anything interesting then?
[15:34:13] <topranks>	 seems the issue was errors on a link, possibly bad optic 
[15:34:15] <topranks>	 https://usercontent.irccloud-cdn.com/file/hflAB32G/image.png
[15:34:21] <dcaro>	 I can confirm that now everything seems to be flowing through c8
[15:34:23] <dcaro>	 https://usercontent.irccloud-cdn.com/file/mD7GQ8rT/image.png
[15:34:25] <topranks>	 that link is currently manually disabled 
[15:34:28] <topranks>	 yeah 
[15:34:55] <dcaro>	 bad optics would match also the fact that pings have been being lost since yesterday
[15:35:35] <topranks>	 yeah well also it's getting worse which sort of matches your ping loss graph 
[15:36:41] <dcaro>	 yep, everything seems working now
[15:39:29] <dcaro>	 topranks: any special template I should use for the task to replace the optics?
[15:40:05] <taavi>	 repeating here from the meet: I am declaring the active incident closed. (the optics still need to be replaced, yes)
[15:41:33] <taavi>	 i'll write the public doc tomorrow I think
[15:42:02] <dcaro>	 taavi: thanks
[15:42:21] <andrewbogott>	 thank you for the quick fix, topranks
[15:43:49] <topranks>	 taavi: ok thanks, sorry this was v. bad timing for me as I was in the middle of an upgrade of another device 
[15:43:56] <topranks>	 so apologies for the poor comms 
[15:44:20] <topranks>	 and yeah one of those nasty ones where it's not a binary UP/DOWN problem but slowly deteriorating 
[15:44:38] <topranks>	 hence no automatic failover etc, the devices seen the link as UP despite the error count increasing 
[15:45:14] <andrewbogott>	 hm, think we need to restart nfs clients?
[15:45:59] <dhinus>	 andrewbogott: do we have any alerts for nfs /client/ failures?
[15:46:06] <dcaro>	 https://usercontent.irccloud-cdn.com/file/JOOM8Bvr/image.png
[15:46:15] <dhinus>	 right
[15:46:16] <dcaro>	 the alert is something like 'too many processes in D state'
[15:46:19] <taavi>	 should I start the reboot cookbook?
[15:46:30] <dcaro>	 I don't see it though
[15:46:36] <dhinus>	 yeah it's not firing
[15:46:53] <andrewbogott>	 taavi: I think so, yes.  Assuming that the nfs server itself is healthy...
[15:47:06] <andrewbogott>	 which it must be since /some/ clients are working fine
[15:47:08] <andrewbogott>	 (e.g. the bastion)
[15:47:20] <andrewbogott>	 so yeah, taavi, please reboot workers.
[15:48:12] <taavi>	 ok, doing
[15:48:20] <dcaro>	 taavi: should be ok yes, that will force pods to get restarted
[15:49:12] <dcaro>	 topranks: what links did you disable exactly? To put in the hardware task
[15:49:43] <topranks>	 I can put the detail there, just looking over a few other things here 
[15:50:12] <topranks>	 it was cloudsw1-d5-eqiad et-0/0/53 to cloudsw1-f4-eqiad et-0/0/54 
[15:50:37] <topranks>	 I had disabled others but brought them back up since, once we confirmed the one that was causing problems 
[15:51:15] <dcaro>	 ack
[15:52:12] <taavi>	 cookbook is running now
[15:52:22] <taavi>	 apparently the wm-bot that SAL relies on is not back yet
[15:52:46] <dcaro>	 xd
[15:53:15] <dcaro>	 we should fix that (nod depend on it at least)
[15:54:08] <andrewbogott>	 topranks: just curious, is there conventional wisdom about what happens /physically/ to explain 'bad optics'?  Like, e.g. does the glass degrade because it's sitting against an LED and gets too hot?
[15:55:27] <topranks>	 yeah it's mostly heat related alright 
[15:55:46] <topranks>	 I'm not fully up on all the details, they're made with certain tolerances and a certain expected lifetime 
[15:55:57] <dhinus>	 I'm checking the remaining alert "No Puppet resources found on instance tools-redis-6 on project tools", Redis is working anyway
[15:56:17] <topranks>	 but in reality that can vary and some can start performing badly like this long before you'd expect 
[15:56:28] <topranks>	 sometimes the receiver can get burnt out if the incoming signal is too hot 
[15:56:50] <andrewbogott>	 makes sense
[15:57:05] <topranks>	 In general SFPs are quite a feat of engineering, getting the electronics and optical parts into a small space and dealing with the heat produced is very tricky 
[15:57:05] <dcaro>	 restarting the mon service on cloudcephmon1002 released the last 2 slow ops (forcing them to rerun on cloudcephmon1003)
[15:59:01] <dcaro>	 topranks: I started T367199, please feel free to fill up, I'm not sure what's needed for a cable change xd
[15:59:02] <stashbot>	 T367199: hw troubleshooting: bad fiber cable between cloudsw1-d5-eqiad port et-0/0/53 to cloudsw1-f4-eqiad port et-0/0/54 - https://phabricator.wikimedia.org/T367199
[16:16:35] <topranks>	 dcaro: thanks 
[16:17:06] <dcaro>	 topranks: got a question, an error counted on one side of the link, will not show up on the other side of the link error count right?
[16:17:29] <topranks>	 fwiw I am about to re-enable that interface, I've disabled all the BGP connections that run over it, which will mean it is not used, but it will enable us to test across the link with ping to verify things when we are testing the optics 
[16:17:34] <topranks>	 dcaro: that can vary 
[16:17:50] <topranks>	 more often than not you only see errors on one side 
[16:18:10] <topranks>	 and those errors can be because of failing module where we see the errors 
[16:18:33] <topranks>	 or can be failing module at the far end, i.e. the transmit from the other side is going bad 
[16:18:35] <dcaro>	 ack, that might explain why our charts did not show errors
[16:18:48] <dcaro>	 (I only added one side of each link 🤦‍♂️)
[16:18:53] <topranks>	 which charts?
[16:18:56] <topranks>	 ah ok 
[16:19:44] <dcaro>	 topranks: okok, I'm ready for the re-enabling
[16:19:57] <dhinus>	 I boldly added an "Action items & follow-up tasks" to the end of the incident doc template :) (and to today's doc)
[16:20:06] <topranks>	 should be fine - I'll do my own checks but just in the interest of full disclosure :) 
[16:20:27] <dcaro>	 dhinus: thanks, I might add stuff there, though I might refine it tomorrow
[16:20:52] <dhinus>	 no rush :)
[16:22:31] <topranks>	 dcaro: ok link back up but not being selected 
[16:23:38] <dcaro>	 looking good so far
[16:26:54] <topranks>	 yep all good 
[16:32:25] <dcaro>	 tools-services-05 is stuck on nfs, I'll reboot manually
[16:33:30] <dcaro>	 that was fast :), back online
[16:34:18] <dcaro>	 btw. the alerts about stuck processes started coming in ~16 min ago
[16:36:08] * dhinus off
[16:37:12] <andrewbogott>	 dcaro: you mean like 'Kubernetes worker tools-k8s-worker-nfs-37 has many processes stuck on IO (probably NFS)'?
[16:37:27] <dcaro>	 yep
[16:38:05] <andrewbogott>	 hopefully those are on workers that taavi hasn't rebooted yet
[16:38:59] <dcaro>	 yep, though I'm finding some non-workers also getting stuck, like tools-services (manually restarted)
[16:39:31] <dcaro>	 or tools-sgebastion-10
[16:39:54] <andrewbogott>	 oh, I just logged into that one
[16:40:00] <dcaro>	 so we might want to extend the alerts to those
[16:40:12] <dcaro>	 I'm waiting for a shell there :)
[16:40:16] <dcaro>	 console works though
[16:40:17] * andrewbogott a bit surprised that sgebastions still exist
[16:40:43] <taavi>	 the reboot cookbook is currently on worker-nfs-37, everything above that has been rebooted and everything below has not
[16:40:48] <dcaro>	 it's the last one, waiting for some fixes on how to use the toolforge api from within toolforge
[16:41:02] <andrewbogott>	 I'm going to do a hard reboot on sgebastion-10 unless you're already doing that
[16:41:09] <dcaro>	 go ahead please
[16:41:30] <andrewbogott>	 ok, done
[16:41:34] <dcaro>	 some users are still logged in
[16:41:50] <bd808>	 T360488 is the main reason for login-buster.toolforge.org still existing I think.
[16:41:51] <stashbot>	 T360488: Missing Perl packages on dev.toolforge.org for anomiebot workflows - https://phabricator.wikimedia.org/T360488
[16:42:01] <dcaro>	 yep
[16:42:55] <andrewbogott>	 taavi: meaning that e.g.  tools-k8s-worker-nfs-19 has rebooted? Because it still shows 'Kubernetes worker tools-k8s-worker-nfs-19 has many processes stuck on IO (probably NFS)' as of 4 minutes ago :(
[16:43:09] <taavi>	 has not
[16:43:13] <bd808>	 I was a bit sad not to find `make` on the new bastions over the weekend when I wanted to make a little deployment helper. I got over it though.
[16:43:20] <andrewbogott>	 oh, it's counting down
[16:43:23] <taavi>	 yes
[16:43:25] <andrewbogott>	 reading fail
[16:43:40] <taavi>	 everything with a number that is above <del>37</del><ins>36</ins> has been rebooted
[16:44:02] <taavi>	 bd808: `make` seems like a reasonable thing to isntall there
[16:44:15] <andrewbogott>	 some of those still show alerts but they're old so likely just need to refresh the check
[16:44:39] <dcaro>	 you can follow https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview?orgId=1&var-cluster=tools&from=now-30m&to=now for the latest data
[16:45:12] <taavi>	 also, very happy right now that I've spent the time to make the reboot cookbook trivially handle nodes that are fully hung up / unresponsible / etc by hard rebooting them (instead of crashing when running 'reboot' gets stuck)
[16:46:05] <andrewbogott>	 Yeah, I wish we didn't need that cookbook but we clearly do
[16:47:34] <dcaro>	  tools-k8s-worker-nfs-51 is reporting choppy stats so the alert does not clear, looking
[16:48:10] <dcaro>	 the host looks ok
[16:48:36] <bd808>	 taavi: :nod: I figured that others would find gnumake uncontroversial for bastion install.
[16:48:41] <bd808>	 I think at this point I would rather have some kind of fix for T360818 that would let folks make custom bastions for themselves in the form of a container.
[16:48:43] <stashbot>	 T360818: Consider adding `kubectl`, `webservice`, and `toolforge` binaries to shell container images - https://phabricator.wikimedia.org/T360818
[16:49:33] <bd808>	 T363027 might be most of what is needed for that
[16:49:33] <stashbot>	 T363027: [builds-builder] Support adding repositories for Apt buildpack - https://phabricator.wikimedia.org/T363027
[16:49:40] <dcaro>	 bd808: +1 from me
[16:50:28] <dcaro>	 for adding custom repos, we might want to control that a bit and only allow certain repos instead (ex, toolforge, etc.)
[16:52:38] <bd808>	 dcaro: yeah, if we are worried about arbitrary package install that could be reasonable. I guess I'm not entirely clear if that is a thing we worry about really in an environment where we let people go wild with language specific package managers (pip, composer, npm)
[16:53:24] <bd808>	 By that I don't mean that we don't care about secure defaults, but we also let people point a lot of foot guns at their own feet
[16:53:45] <dcaro>	 yep, if there's a will there's a way, all we can do is make it hard to do the wrong thing
[16:56:05] <bd808>	 I think if I was in the meeting we had during the Berlin offsite again to talk about buildpacks and other container creation I would ask more questions about why specifically we like buildpacks more than direct Dockerfile usage.
[16:56:43] <bd808>	 I know why it was seen as good at a long ago point, but I don't know if we all share those same concerns today
[16:57:45] <dcaro>	 for people that don't know the difference between a container and a regular process, not having to deal with any of that is easier (ex, if you only have code using standard code packaging, buildpacks are a breeze compared to dockerfiles)
[16:58:39] <dcaro>	 same as using k8s directly vs using toolforge, if you know k8s you can do lots of things that if you try to do with toolforge you will have a hard time
[16:58:58] <dcaro>	 but if you don't know k8s, toolforge will allow you to do many things way easier than if you had to learn k8s
[16:59:24] <dcaro>	 it's a matter of who are they targeted at
[16:59:54] <dcaro>	 (besides any technical struggle, that would most probably be solvable)
[17:00:59] <dcaro>	 with "code packaging" I mean lang-specific package managers (pip/npm/...)
[17:04:06] <topranks>	 dcaro: Valerie has swapped the optics and ping tests over the link are testing clean 
[17:04:15] <dcaro>	 that was quick!
[17:04:29] <topranks>	 unless there is reason not to I will re-enable BGP, and then observe / run ping tests between hosts that were failing earlier?
[17:04:51] <topranks>	 unfortunately sometimes it's only in the presence of significant amount of traffic that an issue shows it's head 
[17:05:05] <topranks>	 but if there is any sign of problem will immediately rollback 
[17:05:18] <topranks>	 dcaro: ha yes she was on site we were lucky :)
[17:05:54] <dcaro>	 let's try, I'm checking on my side
[17:06:17] <topranks>	 ok cool, proceeding 
[17:07:37] <dcaro>	 I'm a bit suspicious though of leaving it enabled, as it might start failing after some usage right? (if it's temp related) so might fail in the middle of the night for me (and with andrewbogott being the only one awake)
[17:07:44] <dcaro>	 I see lost pings
[17:07:45] <topranks>	 nope we had problems 
[17:07:48] <topranks>	 I shut again 
[17:08:01] <dcaro>	 just three though
[17:08:05] <dcaro>	 https://www.irccloud.com/pastebin/pSwmWwZB/
[17:08:10] <dcaro>	 okok
[17:10:26] <dcaro>	 there were a few more lost pings yep :/
[17:10:45] <dcaro>	 https://www.irccloud.com/pastebin/Wf1XZ9Jd/
[17:11:48] <dcaro>	 topranks: is there anything else we can try? does Valerie have another hardware to test? (was only one side changed or both?)
[17:12:15] <topranks>	 we changed it in D5, we're trying the other end of the link now 
[17:12:19] <dcaro>	 maybe it's the cable and not the spf
[17:12:28] <dcaro>	 ack, okok
[17:12:32] <topranks>	 always had to say if it's the receiver on one end or the laser on the other 
[17:12:42] <topranks>	 could be the cable, but usually the cable just breaks all of a sudden 
[17:12:52] <topranks>	 doesn't slowly start getting worse like this 
[17:12:55] <dcaro>	 yep, fiberglass is fragile
[17:13:03] <dcaro>	 it cracks
[17:13:15] <topranks>	 yeah exactly, and link just goes down 
[17:13:38] <topranks>	 typically when you see a small number of errors and then gradually increasing over time it's the electronics 
[17:14:06] <dcaro>	 hmm... now that you mention, in $previous I don't remember a cable breaking ever, but spfs would fail quite often
[17:14:36] <dcaro>	 (cables might be setup broken already, but working and then breaking not sure it happened in ~2years I was around)
[17:15:13] <dcaro>	 besides exravators digging in the wrong place xd
[17:17:38] <topranks>	 haha yeah - it's rare in the datacentre 
[17:17:44] <topranks>	 out in the WAN??  happens all the time 
[17:17:48] <topranks>	 submarines, diggers 
[17:17:56] <topranks>	 and of course the worst offenders - squirrels!
[17:18:06] <topranks>	 those guys just love chewing right through the fiber :P
[17:18:23] <dcaro>	 crunchy :)
[17:18:44] <topranks>	 Valerie swapped the other side - again ping looking clean but I remain dubious 
[17:18:48] <topranks>	 can we try again?
[17:18:58] <dcaro>	 sure, ready
[17:20:10] <topranks>	 ok done
[17:20:23] <dcaro>	 so far it looks better
[17:20:42] <topranks>	 yeah same 
[17:21:48] <dcaro>	 I'll have to go in a bit
[17:22:04] <topranks>	 yeah no probs 
[17:22:08] <topranks>	 that does look better 
[17:22:21] <dcaro>	 is it back at full traffic?
[17:22:30] <topranks>	 what I'll do is leave for a few mins to monitor, but disable BGP again before I clock off 
[17:22:53] <topranks>	 dcaro: no I only brought back up prod-realm IPv4, so there is an IPv6 BGP session and cloud vrf BGP session too which are still disabled 
[17:22:58] <dcaro>	 ack
[17:23:00] <topranks>	 should be a significant level though 
[17:23:34] <topranks>	 zero errors reported on the link though 
[17:23:42] <topranks>	 earlier they started counting up immediately 
[17:23:52] <dcaro>	 yep, promising
[17:24:12] <topranks>	 I'll be around for another hour or so, I'll keep monitoring it and any sign of problem disable 
[17:24:28] <topranks>	 but hopefully that's it fixed 
[17:25:12] <dcaro>	 ack, I'll clock off then, andrewbogott should be around, if anything happens don't hesitate to page me (though taking it off seems it would do the trick worst case scenario)
[17:25:27] <dcaro>	 thanks a lot for the quick fixes!
[17:25:30] <andrewbogott>	 have a good evening!
[17:25:47] <topranks>	 ok np ttyl!
[17:25:53] <topranks>	 https://www.irccloud.com/pastebin/F8c6oo93/
[17:26:07] <dcaro>	 (thank Valerie in my regard too)
[17:26:17] <topranks>	 will do :)