[01:06:20] I have an issue with a `.htaccess` ErrorDocument directive not working, and I suspect some proxy shenanigans. I get errors like [01:06:21] [Tue Jan 18 23:59:35.695556 2022] [:error] [pid 24689] [client 172.16.1.26:35262] script '/var/www/wikiminiatlas/tiles/missingmapnik.php' not found or unable to stat, referer: https://wma.wmflabs.org/iframe.html?wma=69_-52_700_500_en_6_en&globe=Earth&lang=en&page=Disko%20Bay [01:06:21] in my apache2 error log [01:06:33] host 172.16.1.26 [01:06:34] 26.1.16.172.in-addr.arpa domain name pointer maps-proxy-01.project-proxy.eqiad1.wikimedia.cloud. [01:07:24] the script exists and is successfully triggered in some cases, but not in others [01:08:04] for example it is not triggered by https://wma.wmflabs.org/tiles/mapnik/11/1660/tile_1660_1155.png [01:08:34] even though the file does not exist and the .htaccess directive is in https://wma.wmflabs.org/tiles/mapnik/ [01:09:30] then again it is triggered by https://wma.wmflabs.org/tiles/mapnik/11/1660/tile_166s0_1155.png (the filename format error comes from that ErrorDocument php script) [01:11:18] actually strike that. it is triggered [01:11:20] :-/ [01:12:03] but a `file_get_contents` of a file in /tmp returns an empty string (file is owned by www-data, and apache  runs as www-data) [01:21:20] yeah, that's crazy... fopen returns a resource ID but writing to a file fails, file_exists returns false even though teh file is tehre, and file_get_contents returns an empty string [01:25:33] Crap, could it be the `PrivateTmp=true` option in the apache2.service systemd file?! [01:35:41] yep, that was it. using /var/run now for my pid files and fifos [01:36:07] Ladies and gentlemen, thank you for being 147 very effective rubber ducks :-D [01:52:22] hehehe, I set apache’s PrivateTmp to false on my dev system ^^ [01:52:39] file_put_contents("/tmp/blah", var_export(…, true)) is just too convenient sometimes [03:24:36] dschwen: glad it worked out :) [06:11:04] I currently store a ton of stuff on the /data/project NFS (?) mount. Would a "volume" be a better (faster) choice? [08:14:52] dschwen: depends on the use case. If you are sharing the data for read/write across different VMs, a cinder volume can't help you. Think on a cinder volume like a virtual "pen drive". Can only be active on one VM at a time. A volume is nice to decouple data from a particular VM [14:11:38] !log toolsbeta craeted 'automated-toolforge-tests' tool account following https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolsbeta#create_a_tool_account_in_toolsbeta [14:11:40] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:12:08] !log tools.automated-toolforge-tests created account [14:12:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.automated-toolforge-tests/SAL [16:38:11] !log admin moving all scratch mounts to scratch.svc.cloudinfra-nfs.eqiad1.wikimedia.cloud [16:38:13] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [17:33:49] !log maps rebooting maps-wmanew to recover from (presumed) fallout from the scratch/nfs move [17:33:50] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Maps/SAL [17:34:28] !log tools rebooting tools-sgeexec-0913.tools.eqiad1.wikimedia.cloud to recover from (presumed) fallout from the scratch/nfs move [17:34:29] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [17:36:17] !log wikidata-query rebooting wcqs-beta-01.wikidata-query.eqiad1.wikimedia.cloud to recover from (presumed) fallout from the scratch/nfs move [17:36:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidata-query/SAL [17:36:47] !log devtools - added brennen, aokoth and jelto as users and projectadmins (T297411) [17:36:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [17:36:49] T297411: Migrate gitlab-test instance to puppet - https://phabricator.wikimedia.org/T297411 [19:46:47] n00b here setting up openstack-client on cloudcontrol1003 . Does anyone know which password I use? Wikitech dev, maybe? [19:47:04] ^^ trying to use openstack client with deployment-prep env, if it matters [19:50:59] what are you trying to do? [19:51:49] generally you shouldn't touch the cloud* hardware on production unless you're directly working on the cloud vps infra itself [19:52:40] What's the right place to access the Openstack APIs in deployment-prep? For context, I am more comfortable with the openstack CLI/APIs vs. Horizon as I am a former Openstack cloud operation [19:54:16] I also noticed that keystone is an HTTP endpoint, so trying to avoid sending credentials in the claer [19:54:22] err...clear [19:55:47] this is a "nice-to-have" as opposed to a hard requirement, FWiW [19:55:50] the general answer to that is "you don't until we sort out tls and some other form of authentication than passwords" [19:56:33] tls is mostly done now (everything currently supports both http and https), but the second one is still a work-in-progress [19:57:26] in particular we don't want to have ldap passwords (which also grant access to gerrit and other sorts of systems that are dangerous in the wrong hands) lying around in random vps instances or similar [19:58:30] understood, that is why I am asking. I know keystone can generate EC2-like creds, maybe that could be a solution at some point? Would like to help once I get a little more familiar with the env [19:59:06] yeah, I haven't had time to look into them yet [20:00:06] no worries, it will probably be awhile before I'm useful anyway ;) [20:50:55] Yeah, I have a question (already asked last night, was looking for a reply in the public logs, but those stop in May of last year - any reason the channel isn't logged anymore?). [20:51:09] I'm storing a ton of data that I access frequently under /data/project (effectively /mnt/nfs/secondary-maps/project). Is a volume a better solution here? [20:51:31] it’s still logged here: https://wm-bot.wmcloud.org/browser/index.php?start=01%2F19%2F2022&end=01%2F19%2F2022&display=%23wikimedia-cloud [20:52:11] you were probably looking at the Freenode logs on wmflabs.org? (the wm-bot interface switched from wmflabs to wmcloud together with the Freenode→Libera migration IIRC) [20:53:02] if you only access the data from one VM, then a volume might be better I think [20:53:04] yeah, maybe, that means teh links on meta need to get updated... [20:53:13] thx! [21:02:52] I updated some of the links I found on Meta [21:07:49] dschwen6: if you only use the data from one instance, then a volume is likely a better solution. If you need to access the data from more than one instance NFS is currently the only solution. [21:09:05] Yeah, is only accessed from one VM. Volume it is! [23:45:37] I need to restart a job running in grid engine, I used jstop but it didn't stoped, the state column in qstat says 'dr' [23:45:37] so I followed the steps described in https://wikitech.wikimedia.org/wiki/Help:Toolforge/Grid#Stuck_jobs , I ssh to the sgeexec node where the job is running, but the job is not in the 'ps ux' list, the job name is ptwikisbot and the job-ID is 3103931