|
2022-11-29 00:03:21
|
<bd808>
|
!log toolhub Updated demo server to b417c3
|
|
2022-11-29 00:03:22
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolhub/SAL
|
|
2022-11-29 08:46:34
|
<FastLizard4>
|
Hi all! I'm going to bed shortly so I probably can't do anything about it right now if it's actually a problem, but wanted to raise nonetheless - about 30 minutes ago I got a bunch of emails saying that there are puppet failures on various Cloud VPS instances for the account-creation-assistance project. However, the last run summary included in the emails (4 total) all say that there were 0
|
|
2022-11-29 08:46:34
|
<FastLizard4>
|
changes, 0 failure events (and 0 success events for that matter), "No failed resources", and "No exceptions happened". Is something actually wrong?
|
|
2022-11-29 08:46:42
|
<stw>
|
I just had a bunch of puppet failure emails... yeah, that
|
|
2022-11-29 08:46:50
|
<FastLizard4>
|
lol, hi
|
|
2022-11-29 08:47:53
|
<taavi>
|
FastLizard4: yeah, there was a bug in the alert script that a recent refactoring triggered.. that's now fixed, sorry about that
|
|
2022-11-29 08:48:14
|
<taavi>
|
in case you're curious, https://gerrit.wikimedia.org/r/c/operations/puppet/+/861805/
|
|
2022-11-29 08:48:34
|
<FastLizard4>
|
Awesome, just wanted to confirm if this was an actual issue or a bug. Thanks very much!
|
|
2022-11-29 08:54:25
|
<TheresNoTime>
|
(maybe worth an email to the cloud list, was just about to mention all the failure emails)
|
|
2022-11-29 10:07:03
|
<Lucas_WMDE>
|
I agree that an email to the mailing list would be nice, btw
|
|
2022-11-29 10:30:03
|
<arturo>
|
taavi: I can send such email if you want, let me know
|
|
2022-11-29 10:34:17
|
<taavi>
|
arturo: sure, thanks. sorry I missed the initial messages
|
|
2022-11-29 10:34:40
|
<arturo>
|
ok, sending email
|
|
2022-11-29 13:56:03
|
<Raymond_Ndibe>
|
Hello, who have any idea why the volume-admission-controller doesn't work on local machines? atleast in mine?
|
|
2022-11-29 13:57:04
|
<Raymond_Ndibe>
|
I tried installing it on my ubuntu vm on kind. the pod and everything was created but it's not yet playing nice with the jobs-framework-api
|
|
2022-11-29 13:57:20
|
<Raymond_Ndibe>
|
still can't create jobs. jobs are always stuck in pending state
|
|
2022-11-29 14:00:57
|
<dcaro>
|
you cat try checking the logs of the pods for the controller, and/or the jobs-framework-api
|
|
2022-11-29 14:01:18
|
<dcaro>
|
if you did, what was in the logs?
|
|
2022-11-29 14:16:40
|
<Raymond_Ndibe>
|
I already did. The most notable thing there is this line "0/1 nodes are available: 1 node(s) didn't match Pod's node affinity/selector"
|
|
2022-11-29 14:17:33
|
<taavi>
|
logs of what exactly?
|
|
2022-11-29 14:18:37
|
<Raymond_Ndibe>
|
this above is seen in the events of the jobs-framework-api jobs pod
|
|
2022-11-29 14:19:01
|
<taavi>
|
oh I think I know what's happening
|
|
2022-11-29 14:19:49
|
<Raymond_Ndibe>
|
please help 😄, it's a mistery to me
|
|
2022-11-29 14:21:08
|
<taavi>
|
try applying the kubernetes.wmcloud.org/nfs-mounted=true label to the node
|
|
2022-11-29 14:22:41
|
<Raymond_Ndibe>
|
ok let me try that and get back to you
|
|
2022-11-29 14:51:54
|
<Raymond_Ndibe>
|
LAST SEEN TYPE REASON OBJECT MESSAGE
|
|
2022-11-29 14:51:55
|
<Raymond_Ndibe>
|
3m12s Normal Scheduled pod/normaljob-l2psx Successfully assigned tool-test/normaljob-l2psx to kind-control-plane
|
|
2022-11-29 14:51:55
|
<Raymond_Ndibe>
|
3m9s Warning FailedMount pod/normaljob-l2psx MountVolume.SetUp failed for volume "dumps" : hostPath type check failed: /public/dumps is not a directory
|
|
2022-11-29 14:51:55
|
<Raymond_Ndibe>
|
3m9s Warning FailedMount pod/normaljob-l2psx MountVolume.SetUp failed for volume "etcldap-yaml" : hostPath type check failed: /etc/ldap.yaml is not a file
|
|
2022-11-29 14:51:55
|
<Raymond_Ndibe>
|
3m9s Warning FailedMount pod/normaljob-l2psx MountVolume.SetUp failed for volume "scratch" : hostPath type check failed: /data/scratch is not a directory
|
|
2022-11-29 14:54:17
|
<Guest6025>
|
Mistakenly pasted the raw logs here @taavi, but this is the response I get now. The pod is stuck in containerCreating state
|
|
2022-11-29 15:16:12
|
<dcaro>
|
Guest6025: so it seems that you have issues with the volumes
|
|
2022-11-29 15:17:23
|
<dcaro>
|
(that you already knew xd), but it's not the admission webhook denying the pods, looking
|
|
2022-11-29 15:25:04
|
<dcaro>
|
Raymond_Ndibe: I don't see a set way or docs in the volume-admission-controller to change the paths or types of those volumes on deploy time, but you should be able to change the volumes-config configmap (that effectively changes the same)
|
|
2022-11-29 15:25:38
|
<dcaro>
|
Raymond_Ndibe: you might be able to just create an empty config and get away with that
|
|
2022-11-29 15:26:52
|
<dcaro>
|
it expects it to be json, so "{}" instead of just empty
|
|
2022-11-29 15:28:27
|
<MatmaRex>
|
hello, i got emails today like "[Cloud VPS alert][visualeditor] Puppet failure" for every instance in the visualeditor project. is this some sort of known error affecting everything and already resolved, or should i feel alarmed?
|
|
2022-11-29 15:28:54
|
<taavi>
|
MatmaRex: see the follow-up cloud-announce email
|
|
2022-11-29 15:29:02
|
<dcaro>
|
MatmaRex: yes, it's already resolved, should go away before the next cron run
|
|
2022-11-29 15:29:16
|
<dcaro>
|
more details in the email yep
|
|
2022-11-29 15:30:07
|
<MatmaRex>
|
okay, thanks!
|
|
2022-11-29 16:19:44
|
<wm-bot>
|
!log tools.sal <urbanecm> restart, tool was not responding
|
|
2022-11-29 16:19:53
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.sal/SAL
|
|
2022-11-29 19:08:01
|
<taavi>
|
!log ipfs deleted project T323991
|
|
2022-11-29 19:08:04
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ipfs/SAL
|
|
2022-11-29 19:08:04
|
<stashbot>
|
T323991: Request deletion of ipfs VPS project - https://phabricator.wikimedia.org/T323991
|
|
2022-11-29 19:52:48
|
<taavi>
|
!log tools clear puppet failure emails from exim queues
|
|
2022-11-29 19:52:50
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL
|
|
2022-11-29 21:28:36
|
<wm-bot>
|
!log tools.lexeme-forms <lucaswerkmeister> deployed b0ebae4629 (l10n updates: el)
|
|
2022-11-29 21:28:38
|
<stashbot>
|
Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL
|