[03:01:44] !log copypatrol copypatrol-backend-prod-01 deploy d896631..af4b5f5 to address CVE-2024-1135 [03:01:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Copypatrol/SAL [04:59:28] رباط جستجوگر ثنا [08:31:31] !log bsadowski1@tools-sgebastion-10 tools.stewardbots Restarted StewardBot/SULWatcher because of a connection loss [08:31:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [10:48:35] !log tools rebooting tools-k8s-worker-nfs-1 [10:48:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:51:46] !log lucaswerkmeister@tools-sgebastion-10 tools.bridgebot Double IRC messages to other bridges [11:51:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.bridgebot/SAL [12:34:06] !log hoo@tools-sgebastion-10 tools.stewardbots ./stewardbots/StewardBot/manage.sh restart # RC reader not reading RC [12:56:33] !log melos@tools-sgebastion-10 tools.stewardbots SULWatcher/manage.sh restart # SULWatchers disconnected [12:56:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [14:12:48] !log admin deleting dns leaks [14:12:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:20:59] !log tools swapped login.toolforge.org to point to tools-bastion-13 [15:21:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [18:13:24] !log lucaswerkmeister@tools-bastion-13 tools.lexeme-forms pulled fa6c094165 (templates CC BY-SA 3.0 → 4.0; no webservice restart needed) [18:13:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [18:16:55] !log anticomposite@tools-bastion-13 tools.stewardbots ./stewardbots/StewardBot/manage.sh restart # RC reader not reading RC [18:16:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [19:03:08] is it a good idea for the example in https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Configuring_health-checks_for_jobs to be referencing an NFS directory instead of /tmp [19:21:08] my script is not running on Kubernetes. May there be an I/O issue about the cookie? This used to work fine with the old bastion [19:35:23] and... is there any chance you can install PHP on the new bastions? [19:44:57] !log lucaswerkmeister@tools-bastion-13 tools.lexeme-forms deployed e5d2281cea (l10n updates: krc) [19:45:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [19:57:46] @Yetkin: today you can use login-buster.toolforge.org to get an old style "fat" bastion. For the PHP question, file a phabricator task explaining your use case please and thank you. [19:58:52] @Yetkin: I need more information about what is failing and how to say if there would reasonably be any difference from grid to kubernetes for the script you say is having problems. [20:00:43] Any ideas why I would get `Connection closed by 185.15.56.62 port 22` when trying to ssh to toolforge? i the key is right since when i throw -vvv it says "debug1: Server accepts key: Connection closed by 185.15.56.62 port 22" but maybe i setup the key wrong? [20:02:26] the fingerprint for login.toolforge.org changed today, see https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/login.toolforge.org for the current ones [20:05:12] bd808: login-buster works fine. Is there an enf-of-life for this? [20:07:14] @Yetkin: none announced yet. See https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/UAMLGQ42CVHLRZ5W2CZBJDJFRNSBT4DC/ for more details. [20:11:11] derenrich: `access denied for user `derenrich'`. It doesn't look to me like that user account is a Toolforge project member. [20:18:11] derenrich: you need to apply for membership in the Toolforge project and be approved by an admin before you can ssh into a Toolforge bastion. See https://toolsadmin.wikimedia.org/ for the application process. [20:47:45] bd808: ah ok thanks [20:48:57] !log tools In response to stuck processes (NFS?), running sudo cookbook wmcs.toolforge.k8s.reboot --hostname-list tools-k8s-worker-nfs-50 --cluster-name tools [20:48:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [21:11:41] it's not possible to change the amount of RAM a cloud VPS instance gets without recreating it, is it? [21:13:36] you should be able to resize it in horizon [21:14:02] JJMC89: ooh, nice.. I tried "edit instance" hmmm [21:14:27] that's not it though [21:14:42] resize instance [21:15:34] JJMC89: thanks, gotcha, just found it [21:15:48] yay, I dont have to start over :) [21:16:03] gives more RAM to new puppetserver [21:16:28] well, a flavor up [21:16:57] puppet7 seems to be a bit of a hog [21:17:10] yea, unfortunately I had to keep soft rebooting it [21:17:15] more java, more ram I guess [21:17:22] yea, seems like it [21:18:38] !log devtools - resizing puppetmaster-1003 from g3.cores1.ram2.disk20 to g3.cores2.ram4.disk20 - T360470 [21:18:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [21:18:43] T360470: Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470 [21:19:08] have the quota for it because we deleted buster stuff :) [21:21:08] mutante: if you run into the ram problem anywhere else, we are pretty quick at turning around quota increases for stuff like that. [21:21:58] derenrich: you should be a Toolforge member now. Hopefully your ssh access will just work™ at this point [21:22:42] bd808: ack, thank you! [21:24:46] I think I can delete the old puppetmaster now, it's not like we are going to revert and I think the old puppet role is already gone [21:25:02] was just playing it safe and kept it around (shut down) [21:25:47] oh, wait, maybe not. there is still the change in "volatile", revisiting later [21:26:40] Just to know, how much is the default memory of a pod of kubernetes? I just got a OOM error minutes ago, but I tried again and worked. [21:30:36] @albertoleoncio: I think 512MiB? Let me see if I can find that on a Help page on wikitech... [21:33:49] @albertoleoncio: per https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Runtime_and_cpu_memory_limits and https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Job_quotas both say 512MiB RAM and 0.5 CPU are the default hard limits [21:39:27] Oh, good to know that I can increase myself. I was using `npm install` before and wasn't working, then I switch to `yarn install` and it went through. Today even `yarn build` wasn't working, but it did at second try. [21:42:53] @albertoleoncio: yeah, if you are launching a job or interactive shell to do npm/yarn installs you will probably have a better time if you add `-mem 2G -cpu 1` to the command. [21:43:54] it is nice when folks are conservative about the ram and cpu for long running jobs, but for short term setup things ram and cpu generally help make things move faster [21:46:06] bd808: I'll write it down for next time. Thanks! =D [22:33:12] WARNING - POTENTIAL SECURITY BREACH! The server's host key does not match the one PuTTY has cached in the registry. This means that either the server administrator has changed the host key, or you have actually connected to another computer pretending to be the server. The new ssh-ed25519 key fingerprint is... If you were expecting this change and trust the new key, press "Accept [22:33:13] " to update PuTTY's cache and continue connecting. If you want to carry on connecting without updating the cache, press "connect once". If you want to abandon the connection completely, press "Cancel". Pressing "Cancel" is the ONLY guaranteed safe choice. [22:33:42] First time I've ever seen that message. I don't have a clue. Advice please? [22:36:16] This is when attempting my usual connection to wbm1058@tools-sgebastion-10 after rebooting my desktop PC after it shut down due to a local power incident. Stormy weather here today [22:41:56] I'm running PuTTY Release 0.76 .... suppose I could update to the latest version 0.81 [22:42:28] Bill: see "[Cloud] [Cloud-announce] New dev.toolforge.org bastion" [22:43:02] Bill: you can compare the fingerprint to https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/dev.toolforge.org and if it matches then accept it [22:44:09] https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/message/IXEIFD3ADOJFRQUQ2KCPAE2DNBZ53JLQ/ [22:44:25] alternative, you can also use login-buster.toolforge.org to get the old server [22:50:21] @Bill: if you do not yet subscribe to the cloud-announce@lists.wikimedia.org mailing list, please do consider signing up. The Toolforge admins expect that posting to that low volume, announce only list will reach all active Toolforge users. https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.org/ [22:51:13] !status login.toolforge.org ssh key changed; see cloud-announce@ for details [23:02:50] Thanks all. The key I see at https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/dev.toolforge.org doesn't match the key that PuTTY is showing me, however the key I see at https://wikitech.wikimedia.org/wiki/Help:SSH_Fingerprints/login.toolforge.org does match, so I guess I'm good to go and Accept to update PuTTY's cache [23:04:25] I guess it's called a fingerprint, not a key [23:06:28] Bill: yes, it's about login.toolforge.org - my bad, dev.toolforge was swapped in March and today login.toolforge was forwarded to it [23:33:29] I'm pretty sure that today a new vm was given the login.toolforge.org name and not that that service name was applied to the same vm as the dev.toolforge.org service name. [23:38:04] yes, two different VMs