[02:14:12] <bstorm>	 !log admin downtiming SMART alerts on dumps server labstore1007 for the weekend because it has been flapping T281045
[02:14:13] <wm-bb>	 <stashbot> T281045: labstore1007 crashed after storage controller errors--replace disk? - https://phabricator.wikimedia.org/T281045
[02:14:15] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[02:14:15] <stashbot>	 T281045: labstore1007 crashed after storage controller errors--replace disk? - https://phabricator.wikimedia.org/T281045
[02:23:24] <chlod>	 Anyone knows what's up with FontCDN on wmflabs? CSS files (https://tools-static.wmflabs.org/fontcdn/css?family=Roboto) are being served properly but font files (e.g. https://tools-static.wmflabs.org/fontcdn/s/roboto/v27/KFOmCnqEu92Fr1Mu72xKOzY.woff2) are 502'ing.
[02:24:21] <chlod>	 The mess can be visualized on https://fontcdn.toolforge.org/ where everything is just browser fallback font (for me, at least)
[02:39:00] <bd808>	 chlod: I'll see if I can figure anything out quickly. We may need to open a bug and look more tomorrow (/me is getting tired)
[02:39:24] <chlod>	 bd808: Got it. Thanks!
[02:42:47] <bd808>	 "no live upstreams while connecting to upstream" in the nginx error log. Now to check what it's really doing
[02:51:54] <bd808>	 !log tools Restarted nginx on tools-static-14 to see if that clears up the fontcdn 502 errors
[02:51:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[02:52:29] <bd808>	 chlod: turning it off and on seemed to work. It's like I'm a pro SRE or something :)
[02:53:02] <chlod>	 thank you so much! :)
[04:33:31] <wm-bb>	 <stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[04:34:56] <bd808>	 I wonder what we need to change to make the wm-bot on Libera.Chat relay the dologmsg notices instead of the Freenode bot?
[05:16:52] <wm-bb>	 <stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL
[14:32:17] <majavah>	 !log tools manually remove valhallasw yuvipanda admin certificates and from configmap and restart maintain-kubeusers pod T282725
[14:32:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:32:21] <stashbot>	 T282725: maintain-kubeusers: Remove admin access from users removed from tools.admin - https://phabricator.wikimedia.org/T282725
[14:47:31] <majavah>	 !log tools manually remove jeh admin certificates and from maintain-kubeusers configmap T282725
[14:47:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
[14:47:35] <stashbot>	 T282725: maintain-kubeusers: Remove admin access from users removed from tools.admin - https://phabricator.wikimedia.org/T282725
[16:50:05] <Poslovitch>	 Hi, I'm kinda new to the whole Toolforge and Kubernetes stuff. I'd like to setup a Python 3.8 bot to run on toolforge. I already set up the tool itself (https://toolsadmin.wikimedia.org/tools/id/lingua-libre-bot) and I'm ssh'd into the server with the proper tool account. I've been reading through the docs I could find (most notably:
[16:50:05] <Poslovitch>	 https://wikitech.wikimedia.org/wiki/Help:Toolforge/Kubernetes#Kubernetes_cronjobs), but before I start fiddling with Kubernetes, I thought I'd try to make the bot works at least once with the shell. Which I can't, because it requires Python 3.8 and python3 --version returns 3.5.3
[16:52:51] <majavah>	 Poslovitch: There's Python 3.7 on Kubernetes, I'm afraid there is no 3.8 or newer since we usually have what released Debian versions have (Debian 9/Stretch is running on the bastions and the grid with 3.5, Debian 10/Buster has 3.7, and Debian 11/Bullseye which is very close to release but not out yet will have 3.9)
[16:54:07] <Poslovitch>	 Okay. 3.7 should be fine. Now, how do I get myself a kubernetes installed? The page about Kubernetes seems to be about webservices, which my bot is not, I believe
[16:59:11] <majavah>	 Kubernetes is a platform for running containers across multiple servers, it's already installed but you just need to tell it what to do (which can be sometimes difficult due to all of its complex features). We have easy-to-use tooling for webservices, but similar for continuous jobs and cronjobs are still in development. You can talk to Kubernetes
[16:59:11] <majavah>	 directly using the `kubectl` command in the meantime. There's are a few examples on that page, and if you have any specific questions or issues I and others here can help with them.
[17:01:06] <Poslovitch>	 So, I can create a pod with kubectl and make it use this `image: docker-registry.tools.wmflabs.org/toolforge-python37-sssd-base:latest` as the docker image in which my code will run? Does that mean the bot will run on Python 3.7?
[17:03:38] <majavah>	 Basically yes. You should create a Deployment object instead of creating a Pod directly, so it makes sure your tool/pod is always running even if that specific pod gets deleted, for example if we have to restart a node for updates.
[17:04:28] <Poslovitch>	 Okay. Sounds a bit tough to do, but hopefully I'll figure out a way. Thanks for the hints!
[17:05:53] <legoktm>	 bd808: I know what to fix for the wm-bot !log relay, I will get to it in a bit
[17:11:49] <bd808>	 legoktm: awesome. thank you
[18:57:40] <wm-bot>	 I trust:  MacFan4000!.*@user/macfan4000 (2admin), .*@user/majavah (2admin), .*@user/bd808 (2admin),
[18:57:40] <legoktm>	 @trusted
[18:57:49] <majavah>	 want to be trusted too?
[18:58:17] <wm-bot>	 Successfully added .*@user/legoktm
[18:58:17] <majavah>	 @trustadd .*@user/legoktm admin
[18:58:19] <legoktm>	 ty
[19:12:55] * legoktm pokes wm-bot
[19:12:57] <wm-bot>	 Relay was enabled
[19:12:57] <legoktm>	 @relay-on
[19:12:59] <wm-bot>	 test
[19:13:06] <legoktm>	 yesss
[19:16:13] <legoktm>	 I submitted a patch to change the default port, I guess I could just set the hiera
[19:18:12] <legoktm>	 merged, it'll take a bit for puppet to roll it out
[19:32:07] <legoktm>	 puppet is really slow... :/
[19:34:32] <wm-bot>	 !log tools.admin <legoktm> dologmsg now logs to LiberaChat relay!
[19:34:34] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin/SAL
[19:34:49] <legoktm>	 there we go ^.^
[20:03:37] <gifti>	 i'm trying to mount a cinder volume but it just doesn't work. could be a volume deleted not long ago be a problem here?
[20:03:58] <gifti>	 +with the same name
[20:18:21] <gifti>	 no dice, wth
[20:21:23] <wm-bot>	 !log tools.integraality <jeanfred> Deploy b52079f
[20:21:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL
[20:24:27] <wm-bot>	 !log tools.integraality <jeanfred> Deploy 2f39621
[20:24:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL
[20:33:38] <bd808>	 gifti: hmmm... I haven't done much debugging with cinder volumes yet. andrewbogott or bstorm are either of you about?
[20:35:17] <majavah>	 gifti: can you be more specific than "doesn't work"? how are you doing that? any errors?
[20:40:06] <gifti>	 majavah: i had a 100G volume attached (it was mounted properly), i detached it, deleted it, created a new 80G one, attached it, forgot to edit fstab, ran wmcs-prepare-cinder-volume, edited fstab, ran the command again, it was successful, no errors, but it wasn't mounted, i tried it one more time, this time properly, same results, then i tried it with another volume name, same results again
[20:47:26] <gifti>	 i can mount it on another instance, so i think i broke something on the instance i want to attach it on 
[20:47:43] <majavah>	 what does /etc/fstab look like?
[20:48:42] <gifti>	 UUID=5eb86d23-f988-42e6-affd-9cc9409a40cb	/	ext4	defaults	11
[20:48:42] <gifti>	 /dev/vd/second-local-disk	/srv	ext4	defaults	0	0
[20:48:42] <gifti>	 UUID=308ee617-48e2-4ac9-8a38-a3821cb61793 /data ext4 nofail,x-systemd.device-timeout=2s 0 2
[20:49:26] <majavah>	 uh
[20:49:55] <majavah>	 does that vm also have a lvm-based extra disk?
[20:50:01] <gifti>	 yes
[20:50:08] <majavah>	 hmm
[20:50:18] <gifti>	 i wanted to migrate some data off of it
[20:50:42] <gifti>	 it worked two months ago
[20:57:39] <wm-bot>	 !log tools.integraality <jeanfred> Deploy d170ea1
[20:57:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL
[20:57:59] <wm-bot>	 !log tools.integraality <jeanfred> Deploy 67c42f43
[20:58:00] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL
[20:58:07] <wm-bot>	 !log tools.integraality <jeanfred> Deploy 258aae9a
[20:58:08] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL
[20:59:11] <andrewbogott>	 gifti: sorry I missed the ping before, I can have a look if you're still here
[20:59:52] <andrewbogott>	 can you tell me the fqdn of the affected instance?
[21:02:18] <gifti>	 andrewbogott: it's taxonbota-b.dwl.eqiad1.wikimedia.cloud
[21:04:46] <andrewbogott>	 gifti: ok, I'm going to start by removing this line from fstab:  'UUID=308ee617-48e2-4ac9-8a38-a3821cb61793 /data ext4 nofail,x-systemd.device-timeout=2s 0 2' that seems like a relic from a previous cinder mount, does that sound right?
[21:05:46] <gifti>	 yes, that was my last failed attempt
[21:06:18] <andrewbogott>	 ok
[21:06:30] <andrewbogott>	 I'm looking at 'lsblk' and I don't see the volume presenting to the OS
[21:06:41] <gifti>	 it's currently somewhere else
[21:06:47] <gifti>	 should i reattach it?
[21:07:10] <andrewbogott>	 this is the 'taxonbot' volume?  Horizon shows it as attached
[21:07:32] <andrewbogott>	 'attached to: /dev/vdb on taxonbot-b'
[21:07:40] <gifti>	 it's taxonbota-test
[21:08:59] <andrewbogott>	 horizon shows that one as '/dev/sdb on taxonbota-c'
[21:09:18] <gifti>	 that's right
[21:09:37] <andrewbogott>	 ok, you've lost me then
[21:09:54] <gifti>	 i want to attach it to taxonbota-b
[21:10:51] <gifti>	 which i have done now
[21:10:57] <andrewbogott>	 stop please
[21:11:01] <gifti>	 ok
[21:11:22] <andrewbogott>	 do you not see in Horizon that the 'taxonbot' volume is already attached to taxonbot-b?
[21:11:39] <gifti>	 i do
[21:11:46] <andrewbogott>	 Is it  your intent to have both volumes attached at once?
[21:11:46] <gifti>	 how is that relevant?
[21:11:54] <gifti>	 no
[21:12:20] <gifti>	 it's all named pretty similarly
[21:12:40] <gifti>	 i want taxonbot on taxonbot-b and taxonbota on taxonbota-b
[21:12:44] <andrewbogott>	 oh shit taxonbota-b and taxonbot-b are two different things
[21:12:46] * andrewbogott sights
[21:12:56] <andrewbogott>	 ok, so now I see the volume in lsblk, that seems good
[21:13:52] <andrewbogott>	 go ahead and try running wmcs-prepare-cinder-volume and let's see what it does :)
[21:14:37] <gifti>	 it asks me where i want to mount it
[21:14:48] <andrewbogott>	 that seems good
[21:15:16] <gifti>	 i will say /data
[21:15:49] <andrewbogott>	 ok
[21:15:55] <gifti>	 it's ready to prepare and mount and i continue
[21:16:12] <andrewbogott>	 yep, go ahead
[21:16:23] <gifti>	 it says it mounts and updates fstab
[21:16:27] <gifti>	 and is done
[21:17:03] <gifti>	 there is no mountpoint in lsblk, no corresponding output in mount and nothing in /data
[21:17:07] <andrewbogott>	 hm, fstab looks right but it didn't mount
[21:18:28] <andrewbogott>	 yeah
[21:18:38] <andrewbogott>	 and 'mount /data' returns success but doesn't do anything
[21:19:39] <andrewbogott>	 did wmcs-prepare-cinder-volume say it was going to format things?
[21:20:05] <gifti>	 no
[21:20:10] <gifti>	 it was already formatted
[21:20:19] <wm-bot>	 !log tools.lingua-libre-bot <poslovitch> Mom, am I really on TV?
[21:20:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lingua-libre-bot/SAL
[21:20:34] <andrewbogott>	 ok
[21:21:41] <andrewbogott>	 I tried mounting it as /waffle instead of /data and it worked
[21:21:50] <gifti>	 huh
[21:21:51] <wm-bot>	 !log tools.lingua-libre-bot <poslovitch> Mom, am I really on TV?
[21:21:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lingua-libre-bot/SAL
[21:21:55] <andrewbogott>	 so... that's interesting!  No idea why it cares unless there's some residual state from something being mounted there before
[21:22:20] <andrewbogott>	 Have you tried rebooting at any point in all this?  (Is this a server that's safe to reboot?)
[21:22:42] <gifti>	 i did not and it's not safe to reboot it at all
[21:23:08] <gifti>	 i would have to coordinate with doctaxon for that
[21:23:17] <andrewbogott>	 ok
[21:23:27] <andrewbogott>	 then... I guess we treat /data as cursed for now :(
[21:23:32] <gifti>	 if it works with another path, i'm happy
[21:23:45] <andrewbogott>	 I'm sorry I don't have an actual explanation
[21:23:52] <gifti>	 that's ok
[21:23:59] <gifti>	 thank you for your idea
[21:24:31] <gifti>	 the server will be scrapped soon hopefully
[21:38:03] <legoktm>	 when I was mounting stuff I had to fiddle with systemd because it wanted to control the mounts
[21:38:08] <legoktm>	 I don't remember what I actually did
[21:38:25] <yuvipand_>	 mmmm systemd
[21:38:29] <wm-bot>	 !log tools.translate-link <lucaswerkmeister> deployed d829336d71 (“show” mode and other improvements)
[21:38:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL
[22:04:33] <wm-bot>	 !log tools.ranker <lucaswerkmeister> pulled efa1cecac0 (README update, no webservice restart)
[22:04:35] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.ranker/SAL
[22:05:27] <wm-bot>	 !log tools.translate-link <lucaswerkmeister> pulled 11922d2df5 (README update, no webservice restart)
[22:05:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.translate-link/SAL
[22:35:10] <yuvipanda>	 chicocvenancio: do you think I can get rights to PAWS again if I wanted?
[22:35:14] <yuvipanda>	 (not that I'm asking for it now...)
[22:35:28] <yuvipanda>	 but sort of testing willingness of the cloud team :)
[23:28:59] <yuvipanda>	 bd808: what do you think I need to do to request more rights for the PAWS deployment?
[23:30:59] <majavah>	 yuvipanda: file a phabricator task and have no-one with similae rights oppose within a week, basically
[23:31:02] <majavah>	 https://wikitech.wikimedia.org/wiki/Help:Access_policies
[23:31:11] <yuvipanda>	 makes sense
[23:31:13] <bd808>	 yuvipanda: open a phab task, tag it with #wmcs-kanban, and let me know. :)
[23:31:43] <god-damn-sucker>	 this is a part of IRC you can't do anywhere else
[23:31:51] <bd808>	 god-damn-sucker: :)
[23:32:14] <bd808>	 god-damn-sucker: reminded me of https://bash.toolforge.org/quip/AVQ_K14_1EdYBLP_7qWd
[23:33:36] <god-damn-sucker>	 hahahahahahahah
[23:37:09] <yuvipanda>	 bd808: https://phabricator.wikimedia.org/T283443
[23:39:39] <yuvipanda>	 no promises on actually doing anything tho
[23:53:23] <yuvipanda>	 and hopefully https://github.com/toolforge/paws/pull/70 will setup automatic build + push of the user image, making chicocvenancio's life eaiser :)
[23:56:26] <bd808>	 yuvipanda: if you can trick^Wconvince madhu to come around too I'll give you root everywhere. ;)
[23:56:39] * bd808 misses the "old" gang
[23:56:41] <yuvipanda>	 bd808: I'm 100% convinced that isn't going to happen :)
[23:57:18] <yuvipanda>	 she's engineering management at hashicorp now, but really she wants to work on making clothes for a living eventually...
[23:58:46] <bd808>	 clothes are cool. bootstrapping physical manufacturing sounds meh, but I'm sure madhu can do it if she decides to do it