[00:30:57] gry: puppet (in theory) dictates what the state of a server should look like, including potentially installing packages, creating config files, etc. if the VM dies, you just recreate it and puppet will set it up exactly the same as before (again, in theory) [00:31:20] because puppet is maintained in git it also provides a public record of changes made to servers and (in theory) a way to revert back [00:31:22] interesting. what is the use case for that? when for example would a vm die? [00:31:47] ok, i will take notes of what i do in case someone wants to help with creating a puppet script [00:32:12] i'm going through a few docs about packages to install for the webchat, hopefully it'll be done in the next couple days [00:33:02] hardware failures are rare but ocassionally happen. I think there's a lot more redundancy than there used to be. also when we're upgrading OS versions, like from stretch -> buster, it is often easier to just delete the old VM and create a new one with the newer OS and let puppet set it up again instead of manually dist-upgrading it [00:33:33] also the value of puppet really comes when you're managing 2+ of the same server, since it keeps them all in sync and you don't have to manually update each one with the change you want [00:34:46] e.g. https://gerrit.wikimedia.org/g/operations/puppet/+/b5c043b00891d31df7569c7ddec33ba86a7dcea3/modules/profile/manifests/lists.pp and https://gerrit.wikimedia.org/g/operations/puppet/+/b5c043b00891d31df7569c7ddec33ba86a7dcea3/modules/mailman3/manifests/web.pp are two parts of the lists.wikimedia.org puppet config which we also reuse on lists.wmcloud.org [00:36:18] okay, thank you! i think i won't do it on two servers, but having notes for re-install for when a vps upgrade happens, sounds useful. [04:15:57] hm, started webchat on port 80 on the vps, i can telnet to it from shell in the vps, but when i add web proxy, it gives error 502 [04:20:18] changed to port 9000 on backend, same issue [04:54:01] gry: did you add the security group (firewall) rule to allow traffic to the port? [04:54:25] majavah: no; how do i do that? [04:55:12] https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reach_Cloud_VPS_servers_from_the_internet#Security_groups [04:57:51] majavah: ok, it works now, thanks. [05:14:40] happy to help [05:16:22] gry: don't forget to add a floating ip address to the instance to have outgoing traffic use it and not the shared nat address, https://wikitech.wikimedia.org/wiki/Help:Manage_floating_IP_addresses_assigned_to_Cloud_VPS_instances#Manage_floating_IP_address(es) [05:31:43] gry: one more side note: the proxy will only pass X-Forwarded-For on a proxy named ircwebchat.wmcloud.org, not ircwebchatdev1. [07:12:25] majavah: ok, i think it has error 'Service worker failed to fetch an url: Request failed with HTTP 502'. what proxy is it using? [07:12:35] i changed name of the proxy as you suggested [07:13:11] gry: https://ircwebchat.wmcloud.org/ is 502ing, if that's what you're looking for [07:14:07] majavah: yes; that's an improvement over error 502 and it not showing its face at all, but it's not fully working, yet [07:15:01] not sure what you mean, that page only shows a "502 Bad Gateway" for me [07:15:36] the lounge registers a service worker, so if you already visited that page it might show its own error page instead [07:16:57] gry: but the proxy is somehow failing to connect to your instance, are you sure thelounge is running and you are using the correct port? [07:19:10] majavah: if you hard-refresh, do you only see the 502 bad gateway error with nothing else? [07:19:50] gry: yes, and a hard refresh will not clear the service worker [07:22:06] I still only see the 502 bad gateway [07:24:51] right, i think i saw something else because of browser cache, i'll check [07:25:11] does this nginx proxy support websockets? [07:26:31] https://thelounge.chat/docs/guides/reverse-proxies#nginx [07:26:33] gry: it should, but currently your thelounge instance does not even respond to normal http requests [07:26:37] ok [07:28:58] one issue, thelounge will keep scrollbacks only if the users log in; for users who didn't log in,it doesn't keep the scrollback [07:29:08] just found it when reading the docs [09:54:12] !log tools clear error state from tools-sgeexec-0913, tools-sgeexec-0950 [09:54:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:56:29] !log admin fix PTR record for 185.15.56.1 (T248025) [09:56:32] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:56:32] T248025: VisualEditor overwrites "edit source" link with JavaScript even when unnecessary, breaking the link in some skins - https://phabricator.wikimedia.org/T248025 [09:57:29] !log admin fix PTR record for 185.15.56.1 (T284025) [09:57:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [09:57:31] T284025: WMCS NAT address missing forward dns records - https://phabricator.wikimedia.org/T284025 [10:10:05] !log tools properly clean up deleted vms tools-k8s-haproxy-[1,2], tools-checker-03 from puppet after using the wrong fqdn first time [10:10:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:01:22] yay, i got theloungechat to work for the first time. permissions issues with ident will be fixed tomorrow. [13:12:02] !log admin Changed the ceph osd_memory_target on eqiad pool to 6Gi (we were reaching the limit, swapping at some points) [13:12:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [19:38:26] !log tools.notwikilambda deployed function-evaluator as another service (only Node supported, not Python) [19:38:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.notwikilambda/SAL [19:41:30] !help can someone investigate what's going on with cyberbot-db-01. I issued a command to reboot, but it hasn't come back up. [19:41:30] If you don't get a response in 15-30 minutes, please create a phabricator task -- https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?projects=wmcs-kanban [19:45:01] Still down [19:46:44] Oh neat. I discovered the option to hard reboot the instance on horizon. [19:47:05] Alright, it's back. :D [19:47:16] \o/ [19:49:58] Is skynet cyberpower? [19:50:06] Ye [19:50:26] yuvipanda how could you tell? [19:51:22] Skynet, glad you found the power cycle button :p [19:51:39] Yes. Very nice that Horizon has that. [19:53:55] I'm not sure, it just all seemed very familiar. [19:55:38] yuvipanda, Skynet is my proprietary alternate username. [19:58:45] I may still need some sysadmin to help investigate the VM. Something is very clearly wrong. [19:59:29] sudo reboot will cause the VM to hang indefinitely, and space is not getting reclaimed on my cinder storage [19:59:56] I just dropped a massive 160 GB table that is no longer needed, but the space never reclaimed on the cinder storage. [20:00:55] I see. did you pick it up on the move to libera.chat because it hadn't been claimed yet? [20:01:03] or did you have it on freenode too? [20:01:29] Nvm. Looks like the VM is getting back to normal. Storage has been reclaimed. [20:01:38] yuvipanda, I have it on both. [20:02:33] As soon as the mass move happened, I swiftly registered on Libera and grabbed it as I knew someone would take it. [20:02:40] mysql does not shrink its data files on disk unless you ask it to [20:03:18] majavah: when I drop a table, normally the disk usage goes down. [20:12:37] are we aware of the phising attempts from "tools.wmflabs.org IT Support"? [20:13:48] not sure if I was the only one to receive one of these. Should I forward it to security@ ? [20:14:30] this was sent to tools.xtools@tools.wmflabs.org so I suspect I'm not alone [20:15:12] uh, good question [20:15:47] I haven't seen any phishing [20:16:05] Nah spam empty [20:29:06] musikanimal looks like normal automated phishing [20:31:25] Platonides so you got one too? I forward mine to security@ [20:31:35] musikanimal: no [20:31:39] well, I don't think so [20:31:53] what I mean, those kind of mails are "normal" [20:33:09] sure, but if it's widespread probably worth warning people not to click on the link. I guess most subscribers to cloud-announce@ are tech-savvy and probably are able to tell this email is fake. They didn't even spoof the sender address [20:33:32] what was the from? [20:34:10] tools.wmflabs.org IT Support via tools.wmflabs.org [20:34:49] the email body says you need to confirm your account to keep it active. That link looks very suspicious. Obviously I didn't click it [20:35:47] really obvious [20:35:55] it is [20:36:22] also the "© 2021 tools.wmflabs.org. All rights reserved." at the bottom, lol [20:37:24] they take the email domain and replace it on a few places [20:39:22] it looks very standard phishing [20:39:39] the "Dear tools.xtools," really gives it away [20:39:49] lol [20:39:51] people who are actually support@compart.com.br https://www2.compart.com.br/site2018/wp-content/uploads/2020/08/our-differentials-melhor-exp.png [20:40:17] it may not even come from support@compart.com.br [20:40:54] yea, but maybe they should know [20:41:15] maybe their mail server is owned? [20:42:48] at least they have spf [20:43:08] have a look at the source ip