[12:42:08] Hey all -- I'm having trouble accessing ctt-rt-testing-01.wikitextexp.eqiad1.wikimedia.cloud -- is there a status page or something where I can see if the host is healthy? [12:42:39] ssh through bastion to the host just hangs, and https://parsoid-rt-tests.wmcloud.org/ (which should be hitting the host) just spins forever as well [12:43:32] cscott: yes check out https://grafana.wmcloud.org/d/0g9N-7pVz/cloud-vps-project-board?from=now-24h&to=now&timezone=browser&var-project=wikitextexp&var-instance=ctt-rt-testing-01 [12:43:44] looks like the host is deeply unhappy [12:45:27] yeah, although "instanceDown" firing for the past 2 months 27 days (!) isn't super useful either. [12:45:50] We kicked off an round-trip testing run yesterday at ~1600 UTC, so the CPU usage going up around then is expected. [12:46:42] lol re: instancedown for two months [12:46:52] It looks like maybe it wasn't complete yet when Raine upgraded one of the servers it was targeting to PHP 8.5, and that got it mad? [12:47:58] not sure, for sure it looks like the tests or sth else ate all the ram [12:48:18] hard reboot will bring the host back [12:49:01] I hit https://parsoid-rt-tests.wmcloud.org/commits at about 12220 UTC which does a db query, and that might have sent it over the roof? [12:49:33] no, wait, the time axis is my local time! So it really stopped responding 8 hours ago. That was well before the server upgrade or my query. [12:50:20] sigh I wish some day I'd figure out how to make UTC for real the default in every grafana dashboard, but yes that's browser time [12:50:25] I'll change it [12:52:19] so yes ~3:30 utc host stopped responding [13:15:45] thanks, i was trying to figure out if there was a setting to make UTC the default in my user profile or something. [13:22:35] yes it is a maze of possibilities, I generally leave my user preferences alone and change the dashboard's [14:06:24] !log tools reboot bastions to pick up kernel updates [14:06:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:41:52] The email spam I mentioned last week is back again. [14:41:53] > Login to Wikimedia Commons as KrinkleBot from a device you have not recently used [14:42:22] Logstash https://logstash.wikimedia.org/goto/69a16723f93757dc4fcbedbdd394ee72 [14:42:37] Login succeeded for elevated KrinkleBot from 172.16.16.38 - FileProtectionSyncBot/2.0 [[User:KrinkleBot]] https://gerrit.wikimedia.org/g/labs/tools/fileprotectionsync - :::::v4: [14:42:38] Login succeeded for elevated KrinkleBot from 172.16.21.114 - FileProtectionSyncBot/2.0 [[User:KrinkleBot]] https://gerrit.wikimedia.org/g/labs/tools/fileprotectionsync - :::::v4: [14:42:45] Email sent to "KrinkleBot" from "Wikimedia Commons" with subject Login to Wikimedia Commons as KrinkleBot from a device you have not recently used [14:43:05] I guess the IP change might have triggered it. [14:43:25] although I'm unsure if that's something we've never changed in ~5 years. [14:44:51] Krinkle: yeah, i remember looking into that a bit more when you last mentioned it but got distracted before I managed to file a task. basically to stop those, EmailAuth needs to be taught to treat all of the WMCS IP space as a single block (instead of the default, which is to treat each /24 as a separate block) [14:46:08] I see, so it already understands ranges but we happen to have a larger than default range [14:46:47] I don't see a flood in logstash, but I imagine a lot of this is very spread out for WMCS, and drowned out by non-wmcs [14:46:55] out of curiosity, how is the bot authenticating? a bot password instead of an oauth token, I assume? [14:48:01] That's right. [14:48:02] https://gerrit.wikimedia.org/g/labs/tools/fileprotectionsync/+/d37df57da2ce762b3b3c8d5d91c8ea5616b71af3/src/FileProtectionSyncBot.php#222 [14:48:09] it used to use pywikibot, botpassword then as well [14:49:20] https://gerrit.wikimedia.org/g/labs/tools/fileprotectionsync/+/d37df57da2ce762b3b3c8d5d91c8ea5616b71af3/src/FileProtectionSyncBot.php#56 [14:49:38] ah, and it's not persisting the cookie jar between runs either, so the cookie can't stop the emailauth notices either [14:54:33] Yeah, I'd rather not put that on NFS to preserve between scheduled jobs [14:55:10] but I can see how that may narrow the issue :) [14:55:27] more state is more things that can go wrong usually [14:56:39] I've avoided OAuth for this so far as I'd prefer to keep this simple, but there's presumalby a way to do OAuth via fixed header in a way that's equivalent to a botpassword but without the need for a login step and cookies. [14:57:56] We have decent docs for the end-user OAuth flows. I'm not sure we have good docs on how to creating a simple header to set. I should know :) [19:31:21] !log bd808@tools-bastion-14 tools.gitlab-content Restart to pick up new container built from 86606dbd (T441864) [19:31:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.gitlab-content/SAL