[07:50:48] morning! [08:50:42] morning [12:38:15] mmhh I'm trying to add a new image to paws' quay.io and failing, I copied the workflow from existing ones tho https://github.com/toolforge/paws/actions/runs/32369347430/job/96426042121 [12:39:22] not sure if I need to create the repo in quay first? I'm also failing to find the credentials [12:41:46] iirc when I last did that I logged in with personal credentials and then issued a service cred for my script. [12:42:13] So that would involve you making yourself a personal account and then me adding you to the wmf account [12:42:40] but let me doublecheck that that's really what I did :) [12:42:55] ack thank you [12:43:30] https://usercontent.irccloud-cdn.com/file/lLlnOGA1/image.png [12:44:02] so at least to check your work you'll want a redhat/quay account, do you have one? [12:44:19] not yet no, I'm creating one rn [12:46:18] ok done, 'fgiunchedi' [12:49:56] ok, I have sent you many membership invites [12:50:51] cheers, I'll poke around in the paws team [12:53:17] I'm trying to figure out where to define secrets in gitlab, seems to have moved since I was last here :/ [12:54:57] ok all done \o/ the github action worked as expected https://github.com/toolforge/paws/actions/runs/32369347430 [12:55:02] thank you for your help [12:55:16] ah, here they are. Settings -> CI/CD -> Variables [12:55:33] Oh great :) So you already did all the gitlab bits [12:59:18] github in this case yeah, I created the repo in quay and added read/write permissions to the github push robot account [13:00:31] oh, using an existing account! That's easier :) [13:02:24] yes this is "just" to add a new image to paws, mostly everything is already in place [13:18:54] can I have a quick stamp for https://gerrit.wikimedia.org/r/c/operations/puppet/+/1327544? [13:19:21] {{done}} [13:21:06] dcaro, at the risk of giving you painful flashbacks... I'm trying to figure out why particular cloudcephosds are showing slow ops. It is /not/ the same set of hosts you spent so on last year... those are working fine now :D [13:21:07] https://docs.google.com/spreadsheets/d/1YdP6qkH2shhdDTJ96kWU7VbSZ197likPvZtRsEf57jk/edit?usp=sharing [13:21:32] ^ There's one obvious thing to consider on that sheet but I'm interested in what I've forgotten to check [13:21:45] requested access [13:21:46] xd [13:21:57] interesting that the old ones work now xd [13:22:05] oh, I forgot to hit 'return' -- can you access now? [13:22:05] aand of course I forgot the SPDX headers :P [13:24:29] 1016-1022 show slow ops but they're about to be replaced so I'm ignoring them. The ones I'm interested in are 1042-1047 which were bought in a batch. [13:35:50] Is that "clouddb1025/mysqld" alert on purpose? [13:37:11] that's dhinus per -data-persistence [13:37:31] ok thanks. too many channels! [13:37:34] yes sorry just commented in -operations as well, forgot an icinga downtime [13:38:00] andrewbogott: hmm, is there any difference in load? all the operations that get stuck are write? [13:39:02] As far as grafana knows, load is never high enough on any of our osds to matter. [13:39:23] And I'm not sure if they're all write ops but the occasional stall reports that I've looked at have all been writes [13:39:44] so I'm thinking I need to drain two of those hosts and do some fio tests and possibly swap drives between servers to test... [13:39:54] but I'm hoping you have an easier suggesting :) [13:42:42] you could try checking if it's all the drives that have the issue [13:42:50] you can also just drain one drive to do the tests first [13:43:05] It is at least most of them. [13:43:32] keeping an eye also if after draining/repooling they don't have issues might indicate a similar issue as dell had, the "old data" [13:43:58] https://www.irccloud.com/pastebin/mZW9NV3z/ [13:44:08] ^ that is what I mean by 'most of them' [13:44:23] so you're saying /just/ a drain and repool and see if that fixes? [13:44:55] maybe yep, but fully draining the drives (recreating the osds essentially) [13:45:26] ok. I need to do that for the fio test anyway [13:45:33] yep [13:46:09] and, for what it's worth: I'm not sure this is an issue that truly matters. We haven't seen any user-facing symptoms at all, and I only noticed because a version upgrade added health checks for these occasional slow ops. [13:46:12] hmm, it's between 3-5 osds, what about 1020 and below? those are the "old" ones right? [13:46:18] But of course now I can't unsee them [13:46:33] right, I'm ignoring 1020 and below because we have replacements getting racked right now [13:46:48] ack, but that might point not to the hardware/batch then [13:46:58] (well.... or not to a single one xd) [13:47:03] you're right, I should at least check what drives are in those. Lets' see if I can find that... [13:48:14] bah, those have 6gps mixed use [13:48:21] the same drives as the newer healthy ones [13:48:28] :/ [14:02:37] I'm going to go ahead and drain/destroy a couple of servers, I guess that'll take a day or two before I can run more tests. [14:10:40] xd 💣 [14:11:37] are you still 'Puppet is disabled. dcaro fooling around with ceph.conf' on cloudcephosd2007-dev? [14:12:58] oh no [14:13:01] sorry, forgot [14:13:24] that was when we were checking if the options from the config file were picked up by the dameon or not [14:13:28] *daemon [14:19:45] ok, I'll reenable [14:31:52] godog: I've just stumbled upon modules/labstore/templates/tc-setup.sh.erb in Puppet with a bunch of outdated references to NFS server addresses, that will need either updating or removing :/ [14:33:02] taavi: 100%, I'm looking forward to clean all of that up next week [14:33:09] just got the paws bit deployed earlier today [14:33:26] ah great [14:34:30] do we have a precedent to terminate all paws singleuser containers? I pushed a new image [14:35:55] I tend to prefer blue/green deployment in paws, that effectively kills all the containers. So there's certainly precedent. [14:36:28] I don't know if it's formalized but as far as I know people don't really run PAWs things that they aren't actively interacting with. [14:37:21] the paws jupyterhub admin panel has a button to stop all running servers so i'm tempted to jfdi [14:38:46] taavi: if you have your pointer on the button already then please do, I don't think I have permissions [14:39:09] {{done}} [14:39:18] andrewbogott: certainly an option blue/green deployment yeah, I was after sth more lightweight to do before the meeting [14:39:22] cheers taavi [14:39:54] godog: I'm not saying you should do blue/green, just saying that I (and rook previously) have pulled out the rug plenty of times without prior announcement, and never got any pushback. [14:40:16] ah! thank you that's a good data pint [14:40:18] point even [14:40:27] Because the user experience in that case is... not much? Basically as soon as they interact with the webpage it will just reload and restart. [14:40:58] I guess they might get an unexpected progbar [14:41:43] taavi: as far as I can see that did terminate some singleuser pods, not all though, but anyways that's fine [14:41:55] yeah paws users seem to be pretty resilient, including the miners [14:43:17] :/ [14:44:18] just fyi. if you do a blue-green and rotate the k8s credentials, update the paws-nfs-2 and restart the miner searching process to pick them up [14:44:27] (currently manual, sorry) [14:45:24] ack [16:54:23] * dhinus off [17:04:15] * dcaro off [17:04:17] cya on monday!