[14:22:35] <andrewbogott>	 !log admin changing login.toolforge.org, bastion.toolforge.org, and dev.toolforge.org dns entries to refer to the new Buster bastions T277653 https://wikitech.wikimedia.org/wiki/News/Toolforge_Stretch_deprecation#Timeline
[14:22:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[14:22:38] <stashbot>	 T277653: Toolforge: add Debian Buster to the grid and eliminate Debian Stretch - https://phabricator.wikimedia.org/T277653
[14:28:14] <Lucas_WMDE>	 ohhh, nice
[14:42:15] <andrewbogott>	 DNS has a 60 minute ttl so you might not see the change for a bit. When you do you'll get a hostkey alert... new hostkeys are at https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/login.toolforge.org
[15:41:01] <wm-bot>	 !log tools.k8s-status <bd808> Hard stop+start of service.
[15:41:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.k8s-status/SAL
[15:52:29] * bd808 tries to grok why uwsgi threads are being sent SIGKILL inside the k8s-status webservice pod
[16:14:00] <wm-bb>	 <Pau> Hi
[16:14:34] <wm-bb>	 <Pau> I logged in login.tools.wmflabs.org
[16:15:01] <wm-bb>	 <Pau> and I received an strange message about dns spoofing
[16:16:04] <wm-bb>	 <Pau> Do I have to change the address to connect to my bot due to the buster migration?
[16:16:23] <wm-bb>	 <lucaswerkmeister> no, but you have to update the SSH host key, there was a cloud-announce email about it earlier today
[16:16:34] <wm-bb>	 <lucaswerkmeister> https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/Z5S3KNIHAX3SJRZX5SASN2QTCSE2Q76H/
[16:17:01] <wm-bb>	 <lucaswerkmeister> new host key fingerprints are at https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/login.toolforge.org
[16:17:14] <wm-bb>	 <Pau> thanks, lucas
[16:17:44] <wm-bb>	 <lucaswerkmeister> (it would also be a good idea to start using login.toolforge.org instead of login.tools.wmflabs.org, but i don’t think that would be related to the error, just a general thing 🙂)
[16:17:55] <wm-bb>	 <Pau> ok
[16:19:07] <wm-bb>	 <Pau> I jut fixed it
[16:19:09] <wm-bb>	 <Pau> Thanks
[16:19:15] <wm-bb>	 <lucaswerkmeister> great d)
[16:19:37] <wm-bb>	 <Pau> Thanks for the help!
[19:01:44] <wm-bot>	 !log tools.lexeme-forms <lucaswerkmeister> deployed fd45333563 (l10n updates, extra unit test)
[19:01:46] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL
[20:40:01] <mutante>	 I got one of those "puppet failed" mails but this one is special because it says:
[20:40:08] <mutante>	 ---- Failed resources if any:    No failed resources.
[20:40:17] <mutante>	  ---- Exceptions that happened when running the script if any:     No exceptions happened.
[20:41:20] <mutante>	 exactly 12 hours ago. will just see if it sends it again the next couple days I guess
[20:47:09] <andrewbogott>	 huh, curious
[20:47:25] <andrewbogott>	 I wonder if that's a failure so catastrophic that it doesn't know what to say :)
[20:48:14] <mutante>	 heh, yea. testing, sec
[20:49:11] <mutante>	 ah, you know. it's "the machine is not reachable by ssh but I tell you that by claiming puppet failed without failures ..because I could not reach it"
[20:49:25] <mutante>	 and "this is going to be fixed by clicking reboot on the instance in Horizon"
[20:49:46] <andrewbogott>	 unhelpful error but easy to fix :)
[20:49:50] <mutante>	 this happened once or twice lately
[20:49:55] <mutante>	 the "having to reboot but dont know why" thing
[20:50:19] <mutante>	 yea, I mean.. at least it's a notification about an issue there, yes :)
[20:50:28] <mutante>	 let me confirm
[20:55:04] <mutante>	 !log devtools - attempting to soft reboot instance deploy1004 (got the puppet fail mail and wasnt reachable by ssh), this happened lately as well to gitlab-prod-1001, same project, different instance, but this time it doesn't just come back yet
[20:55:06] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL
[20:56:08] <mutante>	 ah, _now_ it is back
[20:56:28] <mutante>	 uptime 5 min but only now I could get on it via ssh 
[20:57:18] <mutante>	 oh.. and puppet run does have an error as well, which I have never seen before and seems unrelated, heh
[20:57:59] <mutante>	 Unknown function: 'puppetdb_query'.. uhmm
[20:59:03] <mutante>	 !log devtools - restarting instance gitlab-prod-1001 -  No route to host
[20:59:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL
[21:03:03] <mutante>	 2 instances that were not reachable but are back after soft rebooting. one of them has an actual puppet error, the other does not and is fine
[21:03:49] <mutante>	 and the one that does have an issue is something with modules/wmflib/functions/resource_hosts.pp, line: 33  and puppetdb_query but ..dunno
[21:04:29] <mutante>	 will have to look at recent changes to that but probably not now