[09:31:51] !log admin-monitoring there's a bunch of novafullstack vms in error because it timed out when trying to allocate network, though there's a "successfully plugged vif" message from neutron, cleaning up for now [09:31:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin-monitoring/SAL [09:36:42] !log admin-monitoring actually, there's several different errors, will open tasks for each of them [09:36:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin-monitoring/SAL [10:20:18] !log tools.cloud-ceph-performance-tests Deploying addition of metadata [10:20:20] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.cloud-ceph-performance-tests/SAL [10:47:07] !log tools.wikiloves Deploy aebc1f9 (T284129) [10:47:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.wikiloves/SAL [14:08:37] !log toolsbeta update toolsbeta-test-k8s-control-4 to kubernetes 1.18 T280299 [14:08:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:08:40] T280299: Upgrade Toolforge Kubernetes to latest 1.18 - https://phabricator.wikimedia.org/T280299 [14:18:26] !log admin Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [14:18:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:23:24] !log admin Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [14:23:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:29:36] !log admin Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [14:29:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:42:12] !log toolsadmin remove toolsbeta-test-k8s-etcd-[15,16] from kubernetes, instances do not exist, likely leftovers from local storage work [14:42:13] majavah: Unknown project "toolsadmin" [14:42:21] !log toolsbeta remove toolsbeta-test-k8s-etcd-[15,16] from kubernetes, instances do not exist, likely leftovers from local storage work [14:42:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:57:02] !log admin Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [14:57:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:57:08] !log admin Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [14:57:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:57:30] !log toolsbeta continuing to update rest of k8s control nodes T280299 [14:57:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:57:32] T280299: Upgrade Toolforge Kubernetes to latest 1.18 - https://phabricator.wikimedia.org/T280299 [14:59:06] !log admin Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [14:59:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:59:57] !log admin Rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [14:59:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:02:20] !log toolsbeta continuing to update k8s ingress nodes T280299 [15:02:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:03:11] !log admin Finished rebooting node cloudcephosd2001-dev.codfw.wmnet - cookbook ran by dcaro@vulcanus [15:03:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:11:56] !log toolsbeta updating k8s worker nodes to 1.18 T280299 [15:12:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [15:12:01] T280299: Upgrade Toolforge Kubernetes to latest 1.18 - https://phabricator.wikimedia.org/T280299 [16:17:51] !log admin Rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T281248) - cookbook ran by dcaro@vulcanus [16:17:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:18:52] !log admin Rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T281248) - cookbook ran by dcaro@vulcanus [16:18:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:18:56] !log admin Rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [16:18:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:21:57] !log admin Finished rebooting node cloudcephosd2001-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [16:22:00] !log admin Rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [16:22:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:22:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:24:52] !log admin Finished rebooting node cloudcephosd2002-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [16:24:55] !log admin Rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [16:24:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:24:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:27:48] !log admin Finished rebooting node cloudcephosd2003-dev.codfw.wmnet (T281248) - cookbook ran by dcaro@vulcanus [16:27:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:28:05] !log admin Finished rebooting the nodes ['cloudcephosd2001-dev', 'cloudcephosd2002-dev', 'cloudcephosd2003-dev'] (T281248) - cookbook ran by dcaro@vulcanus [16:28:07] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:18:47] https://tools-static.wmflabs.org/bridgebot/0d53c6d4/file_4879.jpg [17:19:33] This (screenshot i posted in Telegram) is... not very useful. Any way to make the bridge bot ignore those, @bd808 ? [17:28:22] that is quite the one-sided conversation [17:29:26] can you even !log things from telegram anyway? if you can, then the stashbot messages might be useful. if not, then not so much [17:50:26] @jhosby: it's a dumb golang bot, not an AI. :) [17:56:19] wm-bot is excluded from the bridge, mostly because at some time way long ago folks found wm-bot to be "too noisy" in the bridged #wmhack telegram. [17:56:42] I can certainly exclude stashbot too. [17:58:40] but, there is a shared issue here that the exclusion lists are global. [17:59:21] so if we turn off stashbot mirroring by wm-bb here, we have to turn it off everywhere with the current setup [18:06:34] hehehe, yeah. i was mostly wondering about the option to turn it off, but if it is global then some further considerations would have to be taken indeed (re @wmtelegram_bot: @jhosby: it's a dumb golang bot, not an AI. :)) [18:07:25] (by "turn it off" i mean of course just this bot in this chat) [20:32:49] i am attempting to connect to toolforge but i am getting a nice error message from putty telling me "no supported authentication methods available (server sent: publickey,hostbased)" [20:32:53] maybe i should go to sleep [20:34:01] Guest73, make sure you are utilizing your SSH key to sign in [20:34:25] how do i make sure? i put my private key in connection -> ssh -> auth [20:34:29] I can't answer the question on if you should sleep or not :-) But in general, sleeping when tired is a good idea [20:35:39] Guest73, this page has some nice screenshots showing you where to put the key: https://wikitech.wikimedia.org/wiki/Help:Access_to_Toolforge_instances_with_PuTTY_and_WinSCP [20:35:45] Does that help? [20:36:01] i did put it in "private key for authentication" as shown though [20:36:08] it is the full path to the key yes [20:43:27] Guest73, what server are you connecting to? In general that error sounds like you don't have the right server/username/key combination [20:44:21] Or potentially the wrong key associated with your toolforge account.. [20:45:00] tools-login.wmflabs.org, i tried tools-dev.wmflabs.org too though [20:45:28] the username i set in connection -> data is the same as the name of my toolforge account thing [20:50:13] your account name or your shell username [20:50:47] Guest73, you can also test your ssh access to make sure something isn't off there. ssh USERNAME@login.toolforge.org [20:51:27] maybe the tool account has a permissions issue or something [20:51:29] make sure it's your shell username as listed in https://toolsadmin.wikimedia.org/profile/settings/accounts/ [20:51:39] haha oops it was my account name [20:51:50] that'll do it :) [20:52:13] it's funny how changing a single character will make it work! [20:55:34] yay! Thanks AntiComposite! [21:08:26] !log admin downtiming grafana-labs for maintenance [21:08:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [23:19:32] !log admin Downtimed cloudmetrics1002 in icinga until 2021-06-30 23:59:01 (T281881) [23:19:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [23:19:36] T281881: hw troubleshooting: server hardlocking for cloudmetrics1002.eqiad.wmnet - https://phabricator.wikimedia.org/T281881