[00:03:21] !log toolhub Updated demo server to b417c3 [00:03:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolhub/SAL [08:46:34] Hi all! I'm going to bed shortly so I probably can't do anything about it right now if it's actually a problem, but wanted to raise nonetheless - about 30 minutes ago I got a bunch of emails saying that there are puppet failures on various Cloud VPS instances for the account-creation-assistance project. However, the last run summary included in the emails (4 total) all say that there were 0 [08:46:34] changes, 0 failure events (and 0 success events for that matter), "No failed resources", and "No exceptions happened". Is something actually wrong? [08:46:42] I just had a bunch of puppet failure emails... yeah, that [08:46:50] lol, hi [08:47:53] FastLizard4: yeah, there was a bug in the alert script that a recent refactoring triggered.. that's now fixed, sorry about that [08:48:14] in case you're curious, https://gerrit.wikimedia.org/r/c/operations/puppet/+/861805/ [08:48:34] Awesome, just wanted to confirm if this was an actual issue or a bug. Thanks very much! [08:54:25] (maybe worth an email to the cloud list, was just about to mention all the failure emails) [10:07:03] I agree that an email to the mailing list would be nice, btw [10:30:03] taavi: I can send such email if you want, let me know [10:34:17] arturo: sure, thanks. sorry I missed the initial messages [10:34:40] ok, sending email [13:56:03] Hello, who have any idea why the volume-admission-controller doesn't work on local machines? atleast in mine? [13:57:04] I tried installing it on my ubuntu vm on kind. the pod and everything was created but it's not yet playing nice with the jobs-framework-api [13:57:20] still can't create jobs. jobs are always stuck in pending state [14:00:57] you cat try checking the logs of the pods for the controller, and/or the jobs-framework-api [14:01:18] if you did, what was in the logs? [14:16:40] I already did. The most notable thing there is this line "0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector" [14:17:33] logs of what exactly? [14:18:37] this above is seen in the events of the jobs-framework-api jobs pod [14:19:01] oh I think I know what's happening [14:19:49] please help 😄, it's a mistery to me [14:21:08] try applying the kubernetes.wmcloud.org/nfs-mounted=true label to the node [14:22:41] ok let me try that and get back to you [14:51:54] LAST SEEN TYPE REASON OBJECT MESSAGE [14:51:55] 3m12s Normal Scheduled pod/normaljob-l2psx Successfully assigned tool-test/normaljob-l2psx to kind-control-plane [14:51:55] 3m9s Warning FailedMount pod/normaljob-l2psx MountVolume.SetUp failed for volume "dumps" : hostPath type check failed: /public/dumps is not a directory [14:51:55] 3m9s Warning FailedMount pod/normaljob-l2psx MountVolume.SetUp failed for volume "etcldap-yaml" : hostPath type check failed: /etc/ldap.yaml is not a file [14:51:55] 3m9s Warning FailedMount pod/normaljob-l2psx MountVolume.SetUp failed for volume "scratch" : hostPath type check failed: /data/scratch is not a directory [14:54:17] Mistakenly pasted the raw logs here @taavi, but this is the response I get now. The pod is stuck in containerCreating state [15:16:12] Guest6025: so it seems that you have issues with the volumes [15:17:23] (that you already knew xd), but it's not the admission webhook denying the pods, looking [15:25:04] Raymond_Ndibe: I don't see a set way or docs in the volume-admission-controller to change the paths or types of those volumes on deploy time, but you should be able to change the volumes-config configmap (that effectively changes the same) [15:25:38] Raymond_Ndibe: you might be able to just create an empty config and get away with that [15:26:52] it expects it to be json, so "{}" instead of just empty [15:28:27] hello, i got emails today like "[Cloud VPS alert][visualeditor] Puppet failure" for every instance in the visualeditor project. is this some sort of known error affecting everything and already resolved, or should i feel alarmed? [15:28:54] MatmaRex: see the follow-up cloud-announce email [15:29:02] MatmaRex: yes, it's already resolved, should go away before the next cron run [15:29:16] more details in the email yep [15:30:07] okay, thanks! [16:19:44] !log tools.sal restart, tool was not responding [16:19:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sal/SAL [19:08:01] !log ipfs deleted project T323991 [19:08:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ipfs/SAL [19:08:04] T323991: Request deletion of ipfs VPS project - https://phabricator.wikimedia.org/T323991 [19:52:48] !log tools clear puppet failure emails from exim queues [19:52:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:28:36] !log tools.lexeme-forms deployed b0ebae4629 (l10n updates: el) [21:28:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL