[06:44:14] <dcaro>	 morning
[07:49:13] * dcaro off for a bit
[08:11:03] <arturo>	 morning 
[08:27:42] <blancadesal>	 morning
[08:57:12] <blancadesal>	 arturo: when we talked about an http proxy in the context of the catalyst-k3s design, did you mean this https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reach_Cloud_VPS_servers_from_the_internet, or a custom setup?
[08:58:31] <arturo>	 yes, that could work. It may even have an API
[08:58:55] <arturo>	 a custom proxy could work, too
[08:59:08] <arturo>	 I mean, that page refers to what we call "nova-proxy"
[08:59:20] <arturo>	 which is a Cloud VPS-wide, generally available, HTTP proxy
[08:59:57] <arturo>	 but is deployed via standard puppet. It can be easily re-deployed again for only one project usage (like catalyst)
[09:00:21] <arturo>	 but I can't think of a reason why catalyst can't use the general nova-proxy in the initial iteration
[09:01:03] <dcaro>	 it has an api yes
[09:01:07] <arturo>	 this general nova proxy only works for the wmcloud.org domain though
[09:01:21] <arturo>	 if they are planning on using a custom domain, the should deploy their own proxy
[09:01:46] <arturo>	 i.e, this would work for something like `catalyst.wmcloud.org`
[09:02:04] <blancadesal>	 I think having a custom domain is out of scope for the prototype
[09:02:11] <arturo>	 ok
[09:04:16] <blancadesal>	 ok, this means each time a catalyst-env instance is created, a proxy needs to be created for it, and then it has to be destroyed when the instance is destroyed? And this can be done via an api? 
[09:04:31] <arturo>	 yeah
[09:04:32] <dcaro>	 we don't support subdomains in the common proxy thougm (ex. vm1.catalyst.wmcloud.org)
[09:04:49] <arturo>	 they could do something like `environment1-catalyst.wmcloud.org`
[09:04:53] <dcaro>	 yep
[09:04:59] <dcaro>	 they'd have to xd
[09:04:59] <arturo>	 or even
[09:05:15] <arturo>	 `user1-environment1-catalyst.wmcloud.org`, whatever they need
[09:05:27] <taavi>	 something like *.catalyst.wmcloud.org would be relatively easy to add support for
[09:06:08] <dcaro>	 I think there's a limit for the host name too.
[09:06:09] <arturo>	 I wonder what would be the limit of the internal nginx+lua implementation, if any
[09:06:35] <taavi>	 limit in what sense? scalability?
[09:06:46] <arturo>	 yeah, I mean, in the million range?
[09:06:53] <arturo>	 (proxy entries)
[09:07:56] <dcaro>	 (as in, a domain 'label' legth has to be <63 characters, so the <label>.wmcloud.org can't be very long)
[09:10:03] <taavi>	 arturo: I'm not very concerned about that, redis is fast and the current setup is very easy to scale up both horizontally and vertically
[09:10:12] <arturo>	 taavi: ack
[09:18:03] <blancadesal>	 another question: the catalyst api will need to interact with the openstack api. Application credentials are tied to the user who created them though, is that right? Is that a problem other than that if that user leaves the project, the creds are no longer valid? Is there some way to get creds that aren't tied to a specific user (but still scoped to the project)?
[09:18:36] <arturo>	 mmmm
[09:19:18] <arturo>	 do you anticipate that happening? for example, Stef leaving the project?
[09:19:58] <arturo>	 service users can be created, though. So not a major problem I would say
[09:20:44] <arturo>	 so maybe start with a projectadmin-created set of credentials, and in case of problems (or in the next iteration) maybe switch to proper robot credentials
[09:21:04] <blancadesal>	 re stef leaving, you never know! a couple months ago, I would not have bet on you leaving ;)
[09:21:13] <arturo>	 heh, fair
[09:22:00] <blancadesal>	 but yes, I agree it's not a huge problem and that this is not very important for the proto to have a proper service account
[09:23:46] <blancadesal>	 and something like `environment-1-catalyst.wmcloud.org` is probably good enough for now
[09:24:09] <arturo>	 ok
[09:24:19] <arturo>	 so the catalyst UI should be hitting 3 APIs:
[09:24:39] <arturo>	 1) the openstack API to create a new VM instance from a k3s/minikube golden image
[09:24:59] <arturo>	 2) the kubernetes API in the previous VM instance to deploy the software (the mediawiki patch etc)
[09:25:12] <arturo>	 3) the web proxy API to create an entry to point to the previous VM instance
[09:25:41] <taavi>	 blancadesal: I would suggest creating a new developer account dedicated for catalyst if there will be credentials embedded in the project, instead of using someone's personal account
[09:26:08] <blancadesal>	 as a service account of sorts?
[09:26:14] <taavi>	 yeah
[09:26:24] <blancadesal>	 good idea
[09:26:58] <taavi>	 and then give that account the perms it needs and make an application credential for it
[09:27:53] <blancadesal>	 👍
[09:28:53] <blancadesal>	 where is the web proxy api documented?
[09:30:38] <arturo>	 I don't have that in the radar at the moment, but we can figure out together if others don't know either
[09:30:59] <taavi>	 right now nowhere. although there's a go client (https://pkg.go.dev/gitlab.wikimedia.org/repos/cloud/cloud-vps/go-cloudvps) and the python code in horizon is fairly re-usable and could be made into a standalone package if needed
[09:32:49] <blancadesal>	 catalyst backend will be written in go so that's perfect
[09:42:06] <blancadesal>	 last catalyst question (for now!)... what is the way to create the golden image? 
[09:42:56] <arturo>	 there are several ways
[09:43:56] <arturo>	 maybe the easiest is to create a new VM by hand, install the software needed (i.e, k3s/minikube), delete the stuff we don't need and create the image via openstack glance API
[09:44:04] <arturo>	 I think there is a script to do that
[09:44:14] <dcaro>	 we have one that does something similar yes
[09:44:23] <arturo>	 we could extend/fork that script to automate this case
[09:44:34] <dcaro>	 wmcs-image-create)
[09:44:52] <dcaro>	 in the puppet repo iirc
[09:46:22] <taavi>	 are custom images something we support now?
[09:46:44] <arturo>	 we don't usually do bring your own image
[09:46:54] <arturo>	 but we do have some special cases, for example for magnum there are special images too
[09:56:27] <blancadesal>	 found some docs https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/VM_images#Building_Pre-puppetized_Images_with_wmcs-image-create
[10:01:06] <dcaro>	 they might want to strip the puppet stuff from it, but it's an example on how to create an image
[10:45:36] * dcaro back in a bit
[11:35:54] * taavi paged
[11:36:43] <taavi>	 ceph slow ops, making instances inaccessible it seems
[11:37:32] <taavi>	 tools nfs server itself seems accessible
[11:38:13] <taavi>	 i think toolscheck itself is inaccessible, which is why it paged
[11:38:22] <taavi>	 dcaro: are you doing anything on ceph that might have caused it?
[11:38:59] <arturo>	 I'm around in case I can help
[11:39:40] <arturo>	 ceph status seems ok?
[11:39:41] <arturo>	 https://www.irccloud.com/pastebin/5VlsFCge/
[11:39:47] <taavi>	 just got a recovery
[11:40:01] <taavi>	 it was like this just a few moments ago: [WRN] SLOW_OPS: 93 slow ops, oldest one blocked for 793 sec, daemons [osd.106,osd.213] have slow ops.
[11:41:11] <taavi>	 toolsdb was also affected, looks like it's back
[11:42:41] <arturo>	 osd 106 is cloudcephosd1014
[11:42:49] <dhinus>	 looking at grafana, it seems toolsdb was out for about 10 mins and is now back
[11:42:55] <arturo>	 osd 213 is cloudcephosd1027
[11:43:01] <taavi>	 yeah, toolsdb alerted too
[11:43:14] <taavi>	 so far it seems like most stuff has recovered
[11:43:37] <TheresNoTime>	 currently `urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='api.svc.tools.eqiad1.wikimedia.cloud', port=30003): Read timed out. (read timeout=30)` when running `toolforge-jobs list`
[11:43:58] <taavi>	 this isn't looking good: https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview?orgId=1&viewPanel=2
[11:44:21] <arturo>	 we will need to do some reboots
[11:47:00] <taavi>	 TheresNoTime: can you please file a task for monitoring the toolforge api gateway?
[11:47:12] <TheresNoTime>	 taavi: sure :)
[11:47:36] <taavi>	 arturo: do you think we should do a general round of k8s/grid reboots or only the affected hosts? does the NFS server need a reboot?
[11:48:03] <arturo>	 I would _not_ reboot the NFS server unless we think it is 100% required
[11:48:16] <arturo>	 rebooting the NFS server means we will need to force-reboot everything
[11:48:32] <taavi>	 I know. I'm just concerned about the number of D processes it's showing
[11:48:58] <arturo>	 it is slowing down
[11:49:16] <arturo>	 (from 195 to 176 :-P)
[11:49:36] <taavi>	 I rebooted tools-sgeexec-10-19 which had the most D processes (79), let's see what happens
[11:49:43] <arturo>	 ack
[11:49:51] <taavi>	 surprisingly it was reachable just fine etc
[11:50:16] <arturo>	 yeah the problem is usually that the loadavg rises
[11:50:28] <arturo>	 which affects everything else, including grid/k8s scheduling capabilities
[11:50:46] <TheresNoTime>	 T348633 ftr
[11:50:46] <stashbot>	 T348633: Monitor the Toolforge API gateway - https://phabricator.wikimedia.org/T348633
[11:51:06] <arturo>	 TheresNoTime: is still down?
[11:51:28] <TheresNoTime>	 arturo: no, just filing as requested :)
[11:51:37] <taavi>	 yeah, thx
[11:51:52] <taavi>	 also filed T348634 for tracking any actions taken
[11:51:52] <stashbot>	 T348634: ceph slowdown 2023-10-11 - https://phabricator.wikimedia.org/T348634
[11:52:08] <arturo>	 thanks
[11:54:35] <arturo>	 let me check the switches
[11:54:58] <taavi>	 tools-sgeexec-10-19 is back
[11:56:48] <arturo>	 I don't think they are overloaded or anything
[12:00:29] <taavi>	 the grafana dashboard seems to be lagging behind for whatever reason, but I think the number of processes in D seems to be going down
[12:02:04] <arturo>	 ack
[12:02:40] <arturo>	 maybe reboot k8s-worker-75
[12:03:33] <arturo>	 I'm not finding anything useful in ceph logs about what happened
[12:16:01] <dcaro>	 sorry, I was away
[12:17:09] <dcaro>	 ceph was bringing in the last osds (rebalancing, has been doing so for the last couple of days)
[12:17:15] <dcaro>	 it's finished
[12:17:30] <arturo>	 found a smart error
[12:17:32] <taavi>	 when did it finish?
[12:18:07] <arturo>	 on cloudcephosd1027, it may have disk problems
[12:18:52] <arturo>	 https://www.irccloud.com/pastebin/ajbcx1ne/
[12:19:07] <taavi>	 "fun"
[12:19:25] <taavi>	 we should probably depool that disk
[12:19:43] <taavi>	 also we should probably have something to notify us about those
[12:19:46] <dcaro>	 aproximately 10:15 UTC
[12:20:53] <arturo>	 there are many disks affected by the smart error
[12:20:55] <arturo>	 let me generate a list
[12:21:08] <arturo>	 https://www.irccloud.com/pastebin/BbMKzFKk/
[12:23:22] <arturo>	 maybe depool that osd node entirely
[12:23:33] <arturo>	 and have DCops run a disk check or something?
[12:25:38] <dcaro>	 there's several hosts with this issue
[12:25:49] <taavi>	 that's very worrying :(
[12:26:27] <dhinus>	 I'd run that "grep" on all osd hosts with cumin
[12:26:32] <arturo>	 yeah
[12:26:38] <taavi>	 why is there no monitoring for this?
[12:27:04] <dcaro>	 I'm doing that
[12:27:12] <dcaro>	 I thought there was an alert for smart errors
[12:27:35] <arturo>	 maybe the alert config via puppet is not checking for the right disks?
[12:27:40] <dcaro>	 maybe
[12:28:07] <arturo>	 also, confusingly, smartctl reports no problems?
[12:28:09] <arturo>	 https://www.irccloud.com/pastebin/r6pFvB89/
[12:28:18] <dcaro>	 yen, 
[12:29:58] <dcaro>	 *it's not a continuous issue (that error at least), it's the hard drive having bad sectors and moving data arround, though it's not critical it's a clear sign of disk misbehavior
[12:29:59] <arturo>	 taavi: I'm not that worried. I think disk errors are the best problems. Easy for blaming, and also solving :-P
[12:30:30] <taavi>	 it's the "there's several hosts with this issue" part which worries me
[12:30:32] <taavi>	 a single disk, sure
[12:30:46] * arturo nods
[12:31:15] <taavi>	 anyway, since things are no longer on fire, I'm going to go eat lunch
[12:31:37] <dhinus>	 "Offline uncorrectable sectors" seems to imply some sectors are permanently offline, so I'm also surprised it doesn't show when running smartctl manually
[12:31:37] <dcaro>	 https://phabricator.wikimedia.org/P52903
[12:32:32] <arturo>	 dcaro: try with  `sudo journalctl | grep smart_failure | grep "Device: " | awk -F' ' '{print $7}' | sort | uniq`
[12:33:31] <dcaro>	 that's what I did, cumin does not like the quotes so much
[12:33:45] <arturo>	 ok, let me try it myself
[12:33:55] <dhinus>	 I count 14 hosts with issues
[12:34:33] <dhinus>	 is the paste complete? I would also expect a node group without problems :)
[12:37:15] <dhinus>	 this shows the "Offline_Uncorrectable" matching what we see in the logs: "smartctl --all /dev/sdi |grep Offline_Uncorrectable"
[12:38:10] <dcaro>	 usually, if that number does not change, it's okish, if it increases, the disk is degrading, and in the best case scenario, should be 0
[12:39:43] <dcaro>	 a more succinct list of host-drive: https://phabricator.wikimedia.org/P52904
[12:40:31] <dcaro>	 14 hosts, of which, most have errors on all the disks, and one on 7 out of 8
[12:40:50] <dcaro>	 coludcepd21-34
[12:40:55] <dcaro>	 feels like a bad batch
[12:41:05] <arturo>	 heh
[12:41:06] <dhinus>	 if the change in offline sectors is more interesting than the absolute number, we could also grep for "changed" in the logs
[12:41:20] <dhinus>	 I see some log lines like "(changed +8)"
[12:42:47] <dcaro>	 yep, not sure if that's since the last boot, or the last smartcl run
[12:45:56] <dcaro>	 the list of only changed ones https://phabricator.wikimedia.org/P52905
[12:46:21] <dhinus>	 it looks like it's since the previous check, I see some numbers going up in the same day
[12:47:30] <dhinus>	 e.g. "2312 Offline uncorrectable sectors", then a few hours later "2320 Offline uncorrectable sectors (changed +8)"
[12:47:55] <dhinus>	 24 hours later exactly, so maybe the check runs once a day
[12:48:49] <dcaro>	 those hosts are exactly two batches of servers
[12:49:01] <dcaro>	 https://phabricator.wikimedia.org/T291987
[12:49:07] <dcaro>	 and https://phabricator.wikimedia.org/T283888
[12:49:24] <arturo>	 wow! nice finding
[12:49:24] <dhinus>	 unfortunately the "(changed)" comment doesn't look reliable, the day after that same disk goes from 2320 to 2360, without any "(changed)" comment :/
[12:50:24] <dcaro>	 this might be the reason why ceph is unreliable when taking down a host without draining (crossing fingers)
[12:50:44] <dcaro>	 I'll open a task for the hardware
[12:56:14] <arturo>	 thanks!
[12:57:30] <dcaro>	 Please add any extra info you find, I'll fill up the logs and such from the hard drive failure wiki in a bit, got a meeting now
[12:57:31] <dcaro>	 T348643
[12:57:32] <stashbot>	 T348643: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643
[13:04:36] <arturo>	 I linked the procurement tasks
[13:04:40] * arturo food break
[14:07:39] <dcaro>	 I think the grid is starting to break due to NFS
[14:09:37] <dcaro>	 see the puppet error alerts (seems to be when doing Oct 11 13:34:32 tools-sgeweblight-10-16 puppet-agent[14886]: (/Stage[main]/Profile::Toolforge::Grid::Base/Exec[ensure-grid-is-on-NFS]/returns) change from 'notrun' to ['0'] failed: '/bin/false' returned 1 instead of one of [0] (corrective))
[14:27:36] <dcaro>	 I'm rebooting the weblight queue workers
[14:27:40] <dcaro>	 (seem to be the ones that got stuck)
[14:28:11] <balloons>	 Thanks.. Perhaps it too can feel it's demise is imminent 
[14:30:00] <dcaro>	 😈 ? xd
[14:33:35] <taavi>	 is that on the grid only or should I start the k8s reboot cookbook too?
[14:34:27] <dcaro>	 grid only
[14:34:36] <dcaro>	 I looked at a couple k8s workers but seemed ok
[14:34:52] <taavi>	 the puppet check you're seeing is grid specific, and the k8s ingress blackbox checks has been flapping
[14:35:16] <dcaro>	 yes, I was looking at the worker running the admin app
[14:35:22] <dcaro>	 but looked ok
[14:35:51] <dcaro>	 tools-k8s-worker-84
[14:36:29] <blancadesal>	 final call for the toolforge architecture decision request: https://phabricator.wikimedia.org/T346153
[14:37:00] <dcaro>	 and the alert was gone, so I did not look more in depth (there's some workers with 9 D processes, maybe there's some things stuck on those)
[14:37:50] <dcaro>	 do you think that if we use the nfs CSI plugin instead of host mounts, the pods will be able to self-heal? (restarting the pod would force a new nfs connection, and though will not free the stuck processes, the user service would recover I think)
[14:38:44] <arturo>	 would that result in D procs inside the pod if the NFS server is down. If so, is kubernetes able to remove such pod?
[14:39:19] <dcaro>	 it's the same as now, the process that gets stuck is the one inside the pod
[14:39:44] <dcaro>	 not sure if k8s would be able to delete the pod, or it would hang there as 'deleting' or similar though
[14:40:17] <arturo>	 https://github.com/moby/moby/issues/37886
[14:40:59] <arturo>	 maybe not
[14:41:24] <dcaro>	 but if we force the pod to restart, it would be able to start a fresh one correctly
[14:41:39] <dcaro>	 (instead of rebooting the host)
[14:41:57] <arturo>	 I'm not sure if you can cleanup a pod if it has a Dstate proc inside
[14:43:03] <dcaro>	 Not sure either, but for sure you can start a new one
[14:43:31] <dcaro>	 for example, this one is currently stuck:
[14:43:40] <dcaro>	 https://www.irccloud.com/pastebin/hTnBx7YA/
[14:44:09] <dcaro>	 https://www.irccloud.com/pastebin/8KJpH2A9/
[14:45:17] <dcaro>	 it's deleting the pod now (kubectl delete), and it started the new one:
[14:45:19] <dcaro>	 https://www.irccloud.com/pastebin/8m3glrTo/
[14:45:29] <dcaro>	 done
[14:45:39] <dcaro>	 process is gone on the server
[14:45:42] <dcaro>	 https://www.irccloud.com/pastebin/ComFwVQ3/
[14:45:44] <arturo>	 was it able to cleanup? wow
[14:45:47] <dcaro>	 I'm surprised actually
[14:45:55] <arturo>	 that's very nice of kubernetes
[14:46:14] <dcaro>	 and it's responsive again: https://wudele.toolforge.org/
[14:46:42] <arturo>	 then dcaro not using the hostmount sounds an order of magnitude better for toolforge k8s
[14:46:48] <taavi>	 can we easily automate that process?
[14:47:00] <dcaro>	 I guess we can :)
[14:47:43] <dcaro>	 !task [toolforge] automate cleanup of D state webservices on k8s
[14:47:51] <dcaro>	 xd /me dreaming
[14:49:00] <taavi>	 I might have a look at that
[14:49:21] <arturo>	 for pods that mount NFS via CSI, you can maybe introduce some liveness check to ensure NFS is working... to let k8s restart the pod itself
[14:49:34] <dcaro>	 that'd be nice yes
[14:49:54] <dcaro>	 not sure though if new pods on the k8s worker that has issues will get stuck too though
[14:50:41] <dcaro>	 it seems that they behave nicer than the grid ones, as at least from the shell I can access the NFS again
[14:51:04] <arturo>	 but with that scheme there wont be any worker-level NFS problems anymore, no?
[14:51:28] <dcaro>	 I think so yes
[14:51:52] <dcaro>	 I think that there's a chance that even with the hostmount, we can recover the pods by just recreating them though
[14:52:05] <arturo>	 there could be a readiness probe as well, to check if the NFS mount is working, and only consider a pod successfully started if NFS is healthy
[14:53:14] <dcaro>	 yep
[14:53:38] <dcaro>	 I mean, with a simple http probe with timeout you would already catch those no?
[14:54:12] <arturo>	 I don't know of hand how would I implement taht
[14:54:15] <arturo>	 maybe a sidecar?
[14:54:28] <arturo>	 does a sidecar container share the liveness probe with the main container?
[14:55:00] <taavi>	 no
[14:55:27] <arturo>	 does k8s consider a pod failed if one of the containers in such pod is in failed state?
[14:56:54] <dcaro>	 why not query the webservice itself?
[14:57:14] <arturo>	 maybe is a job and there is nothing listening on TCP
[14:57:28] <dcaro>	 yep, jobs are different
[14:57:54] <arturo>	 what about a sidecar container, sharing the proc namespace with the main container, which should be able to query for D procs inside the main container
[14:58:02] <dcaro>	 this look like what we were talking about
[14:58:02] <dcaro>	 https://github.com/kubernetes-csi/livenessprobe
[14:58:36] <dcaro>	 maybe not, but similar xd
[14:58:47] <dcaro>	 I think that's to check the liveness of the csi driver itself
[14:59:08] <taavi>	 we should make it possible to configure probes for jobs anyways.. T341919 and T348512
[14:59:11] <stashbot>	 T341919: Support probes in kubernetes webservices - https://phabricator.wikimedia.org/T341919
[14:59:11] <stashbot>	 T348512: Add health check support to toolforge-jobs - https://phabricator.wikimedia.org/T348512
[14:59:58] <arturo>	 this all sounds like a very nice and cloud-native project :-)
[15:00:02] <dcaro>	 ack, we should be careful not to bind users too much of the k8s api though
[15:01:42] <dcaro>	 T348662
[15:01:43] <stashbot>	 T348662: [toolforge,nfs] automate cleanup of D state webservices by deleting the stuck pod - https://phabricator.wikimedia.org/T348662
[15:01:47] <dcaro>	 for the automation
[15:13:05] * arturo merges https://gerrit.wikimedia.org/r/c/labs/private/+/964926
[15:40:38] * dcaro reverts sneakily xd
[15:44:59] <dhinus>	 something happened in cloudcephosd1025, "CRITICAL - degraded: The following units failed: user@0.service"
[15:46:55] <dcaro>	 oh
[15:47:22] <dcaro>	 the osds look up to the cluster
[15:54:18] <taavi>	 dcaro: can you review https://gerrit.wikimedia.org/r/c/operations/alerts/+/965154 too?
[15:55:25] <dcaro>	 done
[15:55:37] <taavi>	 thanks
[15:55:51] <dcaro>	 dhinus: manually started it and it worked now
[15:56:35] <dcaro>	 this is the only thing I found in the logs:
[15:56:40] <dcaro>	 https://www.irccloud.com/pastebin/jCk2zPlw/
[15:57:33] <dcaro>	 that seems to be the same as https://github.com/systemd/systemd/issues/23164
[15:58:56] <dcaro>	 it seems it might be solved in newer systemd
[16:04:45] <dhinus>	 dcaro: thanks! probably nothing to worry about
[16:10:33] <dcaro>	 gtg
[16:10:54] <dcaro>	 might connect later, cya
[16:28:08] <dhinus>	 I removed arturo from the WMCS team in victorops