[12:50:43] !log admin rebooting each eqiad1 cloudcontrol node in hopes of getting a baseline re: openstack instability [12:50:46] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:01:21] Hey all, I am trying to download files from my PAWS account using a Python script. It seems that I need to be logged in to PAWS in order to be allowed to do so. Do we have an API to do so? Is it documented somewhere? Thanks! [15:02:14] Perhaps pulling from the public endpoint would help. For example here's my files: [15:02:14] https://public.paws.wmcloud.org/User:VRook_(WMF)/ [15:02:33] MisterSynergy: not clear to me what you're trying. Using a python script in PAWS? Locally? [15:05:06] MisterSynergy: answered your phab task about page table [15:06:40] i will do some poking [15:08:14] It's a schema change in progress. I'm not sure if there's any reason the view hasn't been changed or how the logic really works. [15:10:19] MisterSynergy: DBAs are looking [15:11:18] Rook: thanks, the public endpoint works [15:11:28] šŸ‘ [15:11:29] I hadn't thought of this [15:13:07] The Python script is a local one. It pulls data files generated by a script on PAWS to my local machine in order to do some memory-demanding operations that do not fit into PAWS memory limits [15:14:15] MisterSynergy: page table issue is fixed! [15:15:38] Thanks, I will be looking into this in a moment ... [15:17:47] i quickly checked altwiki and the error was gone [15:17:52] For those who wonder: the "page table issue" is tracked at https://phabricator.wikimedia.org/T311148 and not related to my initial question. [15:22:54] Confirm, the error is gone on all reported wikis. Thanks for the prompt fix! [15:24:11] Thank marostegui, I just asked pointed the ticket to the right people [15:26:36] Not sure where to go for this but I am still getting email notifications for https://phabricator.wikimedia.org/T255319 even though Iā€™m unsubscribed [15:27:14] are you added to one of the projects in it? [15:27:41] MisterSynergy: there's also a convenience link for public endpoint on context menu [15:29:58] https://phabricator.wikimedia.org/project/view/1778/ has you as member and is tagged @harej [15:39:52] Should I be able to auth to horizon apis from inside a cloud vm? I was curious whether I could get https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/terraform working, but I always get auth denied [16:41:50] Some new instances I launched don't seem to have dns records, so the bastion can't find their ip addresses. Did I miss a step, or is something broken? [16:43:11] jhathaway: something is indeed likely broken on our side :/ [16:43:32] taavi: thanks, hmm, I wonder what! [16:43:52] if you know, please let us know too [16:44:25] will do [16:44:44] !log admin restart nova-conductor on all the cloudcontrol nodes [16:44:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:48:22] !log admin restart designate-*.service on both cloudservices nodes [16:48:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [16:49:29] (TIL you can use wildcards on systemd commands like that) [16:49:39] yeah that is nice [16:50:03] as long as nothing in your current directory matches that glob! [16:50:55] that might've fixed things [16:51:04] at least my test VM testdnscreate.testlabs.eqiad1.wikimedia.cloud got a record [16:51:19] jhathaway@bastion-restricted-eqiad1-02:~$ host mx2.ldap-dev.eqiad1.wikimedia.cloud [16:51:21] Host mx2.ldap-dev.eqiad1.wikimedia.cloud not found: 3(NXDOMAIN) [16:51:28] ^^ I'm having the DNS issue too, last instance launched was ~10m ago, should I delete/respin? [16:51:35] do I need to destory & relaunch? [16:51:38] probably [16:51:45] thanks [16:52:04] +1, will delete/respin [16:52:26] !log admin-monitoring remove some failed nova-fullstack instances now that DNS problems are hopefully fixed [16:52:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin-monitoring/SAL [16:55:16] at least nova-fullstack is happy now, so please let me know if you see any issues with newly launched issues [16:57:06] looks like my instance got a DNS record, thanks taavi ! [16:58:06] as did mine, thanks taavi! [16:59:08] great! [17:10:58] Hi cloud services folks! [17:11:19] I'm trying to apply an option to the sshd config via puppet and it doesn't seem to be taking effect [17:11:30] over in the Horizon UI for project fr-tech-dev [17:11:53] I've added this to the Hiera config [17:11:59] profile::base::ssh_server_settings: [17:12:10] gateway_ports: true [17:12:32] I added that about a half hour ago, and the puppet agent has just run again [17:13:01] it's supposed to add 'GatewayPorts yes' to the /etc/ssh/sshd_config [17:13:28] the option was added in https://gerrit.wikimedia.org/r/c/operations/puppet/+/724816 [17:13:42] ejegg: hey! looks like the setting names were changed in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/acaf2557d89f6745fb6cc5fef5665dfde965e900%5E%21/#F20 [17:13:48] thanks taavi! [17:13:53] the new hiera key would be `profile::ssh::server::gateway_ports: true` [17:14:20] great, I'll try that [17:18:36] thanks taavi, that did it [17:19:05] awesome [19:18:59] is labtestwikitech.wikimedia.org something search should be concerned with at all? Some of our scripts are complaining that that trying to source data from it errors out (and indeed, the domain doesn't host a working wiki atm) [19:19:13] i can have it ignore it, but is that appropriate? [19:21:43] ebernhardson: feel free to just ignore it [19:22:26] taavi: ok, thanks [19:23:13] it really only exists to validate 2fa for labtesthorizon (T310795), we don't care about the other features [19:23:14] T310795: Abolish labtestwikitech - https://phabricator.wikimedia.org/T310795 [20:50:07] !log quarry 806474: Get non-coincidental history entries. | https://gerrit.wikimedia.org/r/c/analytics/quarry/web/+/806474 [20:50:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [23:53:31] !log suggestbot started all cron jobs on suggestbot-02 [23:53:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Suggestbot/SAL [23:56:28] !log suggestbot shut down and deleted suggestbot-01 instance [23:56:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Suggestbot/SAL