[07:06:24] !log tools.docs Updating to version 0.1.0 [07:06:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.docs/SAL [12:46:28] Hello Cloudy People! Question: where (or how) can I observe the resource usage of my container? [12:46:45] Related quesiton: is it normal that k8s dashboard simply timeouts? [12:53:03] grin: hello! for 2), if you mean k8s-status.toolforge.org, then yes, unfortunately it's not optimized very well and the toolforge cluster is rather big [12:53:20] for monitoring resource usage, try `kubectl top pod` as the tool account [13:27:03] !log admin-monitoring removed a few leaked VMs to forestall a weekend page [13:27:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin-monitoring/SAL [16:38:10] grin, you can also use https://grafana-labs.wikimedia.org/d/toolforge-k8s-namespace-resources/kubernetes-namespace-resources?var-namespace=tool-commons-delinquent&orgId=1&refresh=5m [16:49:39] Did the login.toolforge.org host keys change recently?  I'm getting warnings this morning when I try to ssh in (though I didn't when I logged in a day or two ago).  Or should I be logging into a different host now?  (I'm trying to log in to update some data and to work on migrating over to the Kubernetes grid.) [16:52:30] the host keys changed last week https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/Z5S3KNIHAX3SJRZX5SASN2QTCSE2Q76H/ [16:52:43] or more accurately, the IP behind login.toolforge.org started pointing to a host with different keys [16:52:52] see also the channel topic :) [16:52:52] so it’s possible that a day or two ago, you had cached DNS still pointing to the old host [16:53:03] heh [16:53:36] Makes sense.   Yeah, OK, I see the new keys and they match the one that my ssh was concerned about.  I'll go ahead and update- thanks! [17:06:05] !log tools.wikibugs Updated channels.yaml to: 783e242631296d112bc37dc9ce1b3112c76a7009 channels: Community Tech [17:06:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikibugs/SAL [18:35:09] Who would I talk about installing a package that doesn't seem to be on the current server?  (I'm trying to use the Locale::Country module in Perl, which has been in the core Perl distribution in the past, but which is apparently now in a separate liblocale-codes-perl package.  I can manually code a replacement for the functionality I'm using, but [18:35:09] it'd probably be better to continue using a standard Perl module, in one of the standard library directories (which require superuser privs to install into, by the look of it)). [18:49:43] JohnMarkOckerblo: please file a Phabricator ticket in the https://phabricator.wikimedia.org/project/profile/3978/ project [18:50:20] if it's already packaged for Debian, it should be incredibly straightforward [18:57:10] Thanks!  It does look to be a standard package for Debian.  Do I need to be logged into Phabricator somehow to create a ticket? [19:01:55] JohnMarkOckerblo: yes, but you can use an existing Wikipedia user to login [19:02:14] or the wikitech wiki user [19:03:18] Yup; looks like I've got that.  Will file as a "feature or enhancement request"  (unless it's better to create.a "generic task" instead).  Thanks! [19:07:02] JohnMarkOckerblo: just "New Task" should be fine [19:12:54] Well, it seemed to want a particular type, so I went with the "feature or enhancement".  Hopefully it will get picked up by an appropriate admin. [19:17:51] JohnMarkOckerblo: it has the right tag, I think that is what matters [19:18:22] Thanks!  I see it in the backlog now. [19:22:56] Eventually I'll need to also figure out how to migrate to the proper kubernetes type for my webservice (looks like "lighthttpd" is no longer an option; "perl5.32" is, though I'm not sure what extra arguments or configuration it'll need).  But it looks like the absence of the module on kubernetes was giving error messages in any case, so I'll see [19:22:56] what happens once the module is in place. [19:46:35] !log tools Rebuilt toolforge-perl532-sssd-base & toolforge-perl532-sssd-web to add liblocale-codes-perl (T307812) [19:46:41] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [19:46:44] T307812: Please install the liblocale-codes-perl package on Toolforge - https://phabricator.wikimedia.org/T307812 [19:47:55] JohnMarkOckerblo: That package should be available in the kubernetes cluster now. Getting it on the grid engine nodes will happen when an SRE gets around to merging my puppet patch to add it there too. [19:54:21] Great; thanks!  Would that be likely to have happened by, say, Monday, or does that usually take longer (or shorter)?  (No rush, since I can still use the module on my current webservice type; I just couldn't use it with kubernetes.) [19:57:39] I can do it in ~15 minutes [20:11:48] JohnMarkOckerblo: it'll take 30-45 minutes to roll out to all the nodes I think [20:12:40] Thanks!  (Realizing I'm now involved in a late-Friday deploy-- hope that wasn't a big mistake :) [20:15:19] Fridays are the best time to distract people :p [20:42:30] thanks for that merge legoktm [20:51:10] :))