[04:08:23] komla: best to stay consistent. I'm just worried, since there are still a lot of tools needing to migrate :( [07:34:25] morning! [07:39:51] greetings [08:42:01] unless there are objections I'll start progressively enabling dumps-nfs.w.o on tools, starting from static/mail/bastion then moving to nfs workers [08:57:50] ack [08:57:55] let me know if you want/need any help [08:58:02] thank you dcaro, will do! [10:07:25] ok tools done and failover tested, working as expected [10:07:44] I got https://gerrit.wikimedia.org/r/c/operations/puppet/+/1319053 out to enable the same in cloud as a whole [10:08:06] and working on paws, which will require a different but similar approach since there's no puppet [10:10:50] there aren't many pods running I think in paws, so should be faster/simpler [10:13:12] heh, so far I got a privileged daemonset to do the monitoring of nfs mountpoint and remount as needed [10:13:31] not sure if I'm ready to call that simpler/faster just yet :D [10:15:27] xd, I see wisdom there [10:20:07] overall though testing paws in cloudvps with minikube works well [10:30:00] I'm curious about that, is there a guide? how are you doing it? [10:31:56] yes I'm following https://github.com/toolforge/paws#settings-up-a-development-environment [10:32:09] * godog lunch [11:54:55] * dcaro lunch [15:00:48] I found a useful stat from cluebot-ng that has a tool running the functional tests in a loop, and it seems that somehow on the 23rd the tests started taking double the time [15:00:51] https://usercontent.irccloud-cdn.com/file/XHDrRfwb/image.png [15:02:00] that's right after the outage we had on haproxy side, maybe we are making things considerably slower on the API side? did we do anything specific that would affect the api requests? [15:03:52] neat, it seems it might be the config generate test [15:03:55] https://usercontent.irccloud-cdn.com/file/GZRmIjBo/image.png [15:04:59] meh, jobs-api tests increased all of them [15:05:01] https://usercontent.irccloud-cdn.com/file/BeYIKZy7/image.png [15:07:34] I don't see that change on the tests we run during deployment though, so might be specific to the tool or something [15:12:31] dcaro: hmm https://w.wiki/Srvu [15:13:30] hmmm, yep that's kinda suspicious [15:14:23] wow, add the method too, that's clearer [15:14:28] yeah was just about to say [15:15:29] ohhh, I see, I think I know what might be the issue, we started waiting for pods to finish differently, we might have messed up something (like waiting for each job before deleting the next), looking [15:16:02] is this UTC? [15:16:46] local time for me at least, it shows the offset in the tooltop when you hover over the graph [15:17:47] hmm.... UTC for me, though then it does not match the deployment of jobs-api [15:18:33] which deployment do you have in mind? [15:19:18] https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/releases/0.0.544-20260716120237-b69caa0d <- this release [15:19:46] there was one deployment that day, that is off by a few hours, but that one is the one that changed the delete behavior, and it's a few days before :/ [15:20:23] https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1347 <- this deployment is the one that's a few hours before the bump [15:21:19] https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1338 <- this is the deployment that changed the delete behavior (almost a week before) [15:22:11] i mean it starts climbing like directly after the last deployment of that day https://sal.toolforge.org/tools?p=0&q=&d=2026-07-16 [15:23:16] hmm, I see it climbing on the 23rd [15:23:19] https://usercontent.irccloud-cdn.com/file/7EmUYKHq/image.png [15:23:22] 12 UTC [15:23:25] *14 UTC [15:23:40] ... [15:24:26] the deployment was at 12 UTC right? (/me getting confused now that sal does not say UTC/notUTC) [15:24:27] yeah, 23rd indeed, zooming in shows it starting at around 1500Z [15:24:31] sal is UTC [15:24:55] seems to match more the outage? https://w.wiki/Srwf [15:25:22] why would it keep being high though? [15:26:02] good question [15:26:10] does jobs-api happen to log what exactly is taking so long? [15:26:51] (we should do distributed tracing for the toolforge api services one day, would make debugging stuff like this so much easier) [15:28:35] stern helps a bit there [15:28:48] (it pulls logs from all pods), but yep, +1 for distributed tracing xd [15:29:22] I think debug is not enabled, so unless there's an error it should not log much more than "GET took Xs" [15:29:24] kinda thing [15:32:47] i just tried launching a pod and just waiting for it to terminate normally is taking a while [15:33:13] hmmm [15:33:25] like directly, not using toolforge jobs? [15:34:07] might be the controllers then? [15:35:34] I'm slightly hoping it's the jobs api doing something silly though xd [15:36:51] with a `webservice shell` (which shells out to `kubectl`), yes. but now when trying again it's relatively fast again. so either that first one was a fluke or it is node-dependent [15:40:39] but yes, i think getting some better logs on what exactly is slow is the next step here [16:17:31] cloud-announce review request: https://etherpad.wikimedia.org/p/toolforge-opensearch-announcement [16:19:47] taavi: does this need to change? https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/blob/main/components/envvars-admission/values/tools.yaml?ref_type=heads#L11 [16:20:24] (bad choice using `elastic` on the variable names xd) [16:21:21] if that's something we support then we need to add an opensearch entry there and remove the elasticsearch one after the elastic cluster is shut down [16:22:16] yep, it's something we support yes [16:22:25] it might be in the docs too [16:22:31] it's not in the elastic/opensearch docs at least [16:22:55] probably in the envvars, looking [16:22:55] but the hostnames are stable except when we explicitely need the tool maintainers to check that their tool works with the newer version and move over the data, so if we ever change the hostname it'll practically need a new envvar as well [16:23:05] btw. the email looks ok for me [16:23:43] that's because of the .svc. names right? [16:24:12] .svc.'s are service names we can and will move around when needed, yes [16:24:22] ack [16:24:24] the docs are https://wikitech.wikimedia.org/wiki/Help:Toolforge/Envvars#Globally_set_environment_variables [16:25:02] should I create a task for the addition of the envvar or you are taking care of it? (and adding it to the docs) [16:25:16] i can take care of it, even if i would probably advice people to just use the hostnames directly [16:26:46] if we change port, protocol or have a second cluster at some point (eqiad1 -> eqiad2?) then the envvar is still useful imo, but sure [16:26:55] (it also helps when developing locally, etc.) [16:28:00] https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/1351 [17:17:47] * dcaro off [17:17:49] cya tomorrow! [20:29:57] https://public-paws.wmcloud.org/ is returning 504 timeouts [20:36:33] The console log in Horizon for paws-127d-mq2566hee6zs-node-0 is full of "Memory cgroup out of memory: Killed process NNNNNNN" errors. That is the backend for the public-paws proxy. I don't know anything about fixing PAWS. [20:45:12] bliviero: do you know who to bug about PAWS sadness? ^ [21:39:12] bd808: just restarted nbserve and renderer pods (last time that helped), will restart the node0 in case it's stuck [21:43:14] node1 seems more in trouble (kinda all seem to have been up for quite some time, probably a reboot might be good for all) [21:52:08] * dcaro off [21:52:23] I'll let andrewbogot.t take over when he's around in he's working hours [22:19:47] dcaro: thank you for doing this - yes unfortunately don't have coverage in US timezone when Andrew is out (until next week)