[10:54:48] !log admin running SQL command `update domains set master="208.80.154.11:5354 208.80.154.148:5354 10.64.151.4:5354";` on all 3 cloudservices nodes (T345240) [10:54:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [10:54:52] T345240: cloudservices1006: put into service - https://phabricator.wikimedia.org/T345240 [11:00:49] !log tools restarting mariadb on toolsdb-2 (replica) to test slave_parallel_threads (T345450) [11:00:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:00:52] T345450: [toolsdb] ToolsToolsDBReplicationLagIsTooHigh - 2023-09-01 - https://phabricator.wikimedia.org/T345450 [12:18:52] !log admin updating pools.yaml in all cloudservices designate nodes (T345240) [12:18:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:18:57] T345240: cloudservices1006: put into service - https://phabricator.wikimedia.org/T345240 [12:19:16] !log paws Reduce memory request for single user container T345467 [12:19:18] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [12:19:19] T345467: Reduce memory request for singleuser - https://phabricator.wikimedia.org/T345467 [12:34:47] !log admin synced pdns database from cloudservices1004 to cloudservices1006 (T345240) [12:34:51] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:34:52] T345240: cloudservices1006: put into service - https://phabricator.wikimedia.org/T345240 [12:45:57] !log testlabs moved all VMs to ns-recursor.openstack.eqiad1.wikimediacloud.org via project puppet (T345240, T342621) [12:46:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Testlabs/SAL [12:46:03] T345240: cloudservices1006: put into service - https://phabricator.wikimedia.org/T345240 [12:46:03] T342621: eqiad1: cloudlb: transition DNS clients (VMs) to the new BGP-based recursor VIP - https://phabricator.wikimedia.org/T342621 [12:49:58] Can somebody please approve our kind user MauroBianchiInmagik in Wikitech? I'm not sure how this thing should be requested [12:51:37] 'approving a Wikitech user' is not a thing. perhaps you're confusing it with some other system? [12:52:56] that user is saying "if I login in wikitech.wikimedia.org it says pending approval since a week" [12:54:02] what's the exact error message and the url of the page that's on? a screenshot would be nice too [13:24:26] OK it seems the messasge "your account is pending approval" comes from gitlab.wikimedia.org [13:24:58] and that's not wikitech :-) in that case you want https://phabricator.wikimedia.org/project/profile/6554/ [13:25:08] Thaaanks [18:32:34] Can somebody help me figure out why some of my VMs become unreachable after some time with the error "Connection closed by UNKNOWN port 65535" [18:32:56] I've got two of them doing that, and one of them is a freshly spun up image with nothing done to it yet. [18:33:11] The only fix is to reboot the whole VM. [18:33:18] (That I know of) [18:33:47] (Sounds like all the ports are being opened for something and then never closed. [18:33:57] do you have an example VM that's currently in that state? [18:34:01] Yes [18:34:19] cyberbot-web-iabot-01.cyberbot.eqiad1.wikimedia.cloud [18:37:16] so that one is unreachable because the first puppet run is failing which means it can't actually apply the config to let you log in [18:37:41] Interesting. Do you know what's causing it to fail? [18:38:35] your debian version isn't listed here: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/profile/manifests/cyberbot/exec.pp#5 [18:38:40] that's an easy fix, thankfully [18:39:22] Ah. I'm bad with gerrit, may I ask you to fix it? :-) [18:40:41] yep, https://gerrit.wikimedia.org/r/c/operations/puppet/+/954984/, just waiting for the CI to pass [18:41:06] Thank you. :D [18:42:29] taavi: should I reboot the VM, or will it become reachable on it [18:42:32] 's own [18:42:55] it would do that by itself, but I can make it do that a bit faster [18:44:01] Another thank you. :-) [18:46:04] Skynet: try now? [18:46:06] It's working. [18:46:20] Yay [19:03:46] What is considered the best way to move files from Toolforge to a cloud VM? [19:03:53] Specifically, within a Toolforge tool [19:11:58] `scp `apparently doesn’t work [19:12:07] (due to key authentication requirements) [19:16:50] if the files aren't secret, then you can use https://wikitech.wikimedia.org/wiki/Help:Toolforge/Web#Static_file_server and pull them over HTTP [19:18:10] There are a handful that *are*, but I am not sure if they’re worth moving over [19:18:27] (and they have file permissions accordingly, so I can’t just gzip the directory from my personal Toolforge account) [19:19:21] I've definitely also used the "copy and paste text between two terminals" method before :) [19:19:30] Too many files :( [19:49:22] gzip it as the tool account? [20:08:57] fyi i got nonsense lottery spam to wikitech email [23:22:30] hey, mathoid-beta.wmflabs.org is giving 504 [23:23:25] this is causing default configurations of Math to time out on Special:MathStatus and possibly failing to render math functions [23:28:26] Error: Unable to retrieve sudo rules. Details [23:28:26] {'result': -11, 'desc': 'Connect error', 'ctrls': [], 'info': '(unknown error code)'} [23:28:29] One of my team members can't use sudo, so when I went to check on his access on Horizon, I got ^ [23:32:36] wmflabs questions go here, right? [23:36:44] other than there not being anything called wmflabs anymore, yes [23:42:59] you could try contacting the Members or Viewers listed at https://openstack-browser.toolforge.org/project/math (Physikerwelt being the one I’d most associate with this area) [23:56:00] AntiComposite: there's some urls on wmflabs such as mathoid-beta and phabricator test instance [23:56:37] i mean servers