[06:02:33] Job name:          Job type:               Status: [06:02:34] -----------------  ----------------------  ---------------------------------------- [06:02:34] job-archive        schedule: 0 10 * * *    Last schedule time: 2024-01-31T10:00:00Z [06:02:35] is there any way to know why my job didn't run on 1 Feb? [08:29:35] leloiandudu: how long does it take to run usually? [08:43:42] Looking at the logs, I'm having trouble trying to find anything between 01-31 8:30 and 02-01... weird [08:54:15] Is there a mechanism to periodically clear up files from a directory in a custom container ? (Or do I just build my own) [09:02:50] sohom_datta: not really, restarting the container will do so though (might be a good practice too), can you elaborate a bit more or your use case? [09:11:57] croptool comes with a script that clears out the files/ directory out periodically as a job, but I've unsure wrt how that would work in a custom container (re @wmtelegram_bot: sohom_datta: not really, restarting the container will do so though (might be a good practice too), can you elaborate a ...) [09:18:24] sohom_datta: yep, any job will run on it's own container. A way of forcing a restart would be through liveness probes, though we don't set those yet so needs a bit of development (k8s would do a request to a port/path, and if not what it expects will restart the container), or making the process exit, k8s will restart it too. Or running your own cleanup from the main process as you said. [09:19:34] Got it, that makes sense [11:08:36] is there any way to execute a command before every job? I can create a .sh file for each job and write it there but since what I'm trying to do is just a simple git pull for all jobs, is there a way to do it once for all jobs? [11:12:41] ashotjanibekyan: you can create another job that does the pull periodically [13:54:41] !log admin [codfw1dev] cleanup /etc/network/interfaces on cloudlb2003-dev from puppet leftovers [13:54:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:44:45] I think there is (still) a typo the bottom of the Cloud Services Terms of use. It's in one of the hyperlinks. https://wikitech.wikimedia.org/wiki/Wikitech_talk:Cloud_Services_Terms_of_use [14:45:27] Better link: https://wikitech.wikimedia.org/wiki/Wikitech_talk:Cloud_Services_Terms_of_use#c-Whym-20230909022100-Possible_typo [15:10:34] leloiandudu: I was unable to find the logs for the period in which that job should have triggered :/, usually if a run is skipped, is either because the job took too long (so it does not trigger if it's still running) or because there was some other k8s level issue (of which I don't have logs... so maybe?) If it happens again, can you open a task to follow up? [16:16:37] @Yusuke you're right, thank you! [16:42:46] According to https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Listing_your_existing_jobs there is no image that runs PHP 5.x versions. Is that right? [16:46:11] Yetkin: PHP5 feels very very old [16:47:00] 5.6 is EOL since 31 Dec 2018 [16:54:22] * dhinus feels old remembering when PHP5 was the shiny new version :D [16:55:55] dhinus: I know right? PHP 5 was so fancy with it's first class objects and new syntax. :) [16:59:50] I feel dirty pointing this out, but we actually do still have a set of unmaintained PHP5 containers in the registry (https://docker-registry.toolforge.org/). I would not encourage anyone to use them; please update your stuff to work with something that has been patched in this decade. :) [17:05:45] bd808: you may even be able to them from k8s, is marked as deprecated, but present: https://gitlab.wikimedia.org/repos/cloud/toolforge/image-config/-/blob/main/deployment/chart/values.yaml?ref_type=heads#L255 [17:06:57] Yetkin: so the actual answer to your question is: yes, there is an image to run PHP5.6 on toolforge kubernetes. It is considered deprecated, but should be 'usable' [17:10:23] arturo: oh you very much can run them in Toolforge. I can see on https://k8s-status.toolforge.org/images/ that there are currently 155 active "toolforge-php5-sssd-web" containers. `ebservice --backend kubernetes php5.6 shell -- php -v` outputs "PHP 5.6.33-0+deb8u1 (cli) (built: Jan 5 2018 15:46:26)" [19:12:13] !log devtools deleting web proxy phorge.wmcloud.org which pointed to 172.16.7.98 which doesn't exist anymore T356530 [19:12:17] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [19:12:18] T356530: clean up phabricator test host situation - https://phabricator.wikimedia.org/T356530 [19:29:39] !log devtools deleting web proxy phabricator-prod.wmflabs.org which pointed to port 443 on the buster instance and timed out T356530 [19:29:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [19:29:43] T356530: clean up phabricator test host situation - https://phabricator.wikimedia.org/T356530 [19:55:26] how to run a docker image on a bastion? [20:00:36] !log quarry rebuilding trove instances with new antelope guest image [20:00:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [20:13:55] !log devtools running "scap deploy" in /srv/deployment/phabricator/deployment on deploy-1004 which deploys to phabricator-bullseye and phabricator-prod-1001 T356530 [20:13:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [20:13:59] T356530: clean up phabricator test host situation - https://phabricator.wikimedia.org/T356530 [20:16:57] !log editing proxies phab.wmflabs.org and phab-prod-usercontent.wmflabs.org to point to bullseye instance instead of buster T356530 [20:16:57] mutante: Unknown project "editing" [20:17:03] !log devtools editing proxies phab.wmflabs.org and phab-prod-usercontent.wmflabs.org to point to bullseye instance instead of buster T356530 [20:17:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [20:38:34] !log devtools deleting proxy phab-bull.wmcloud.org after previous proxy names are switched to bullseye backend T356530 [20:38:38] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [20:38:38] T356530: clean up phabricator test host situation - https://phabricator.wikimedia.org/T356530 [20:41:44] !log devtools shutting down instance phabricator-prod-1001 (buster), replaced by phab-bullseye (bullseye) T356530 [20:41:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [20:45:23] @Yetkin: you cannot run a container on the bastion. You can only run our provided containers on the Kubernetes grid in Toolforge. If you have access to another Cloud VPS project then you could in theory install Docker or Podman on a vm you administer and run an arbitrary container there. [20:57:30] bd808: it's a good thing to delete any proxy using "wmflabs.org" and replace it with wmcloud.org in general? you still want to get rid of the labs domain entirely? or they are just equal [20:59:51] !log devtools - changing phabricator domain in instance Hiera of phabricator-bullseye to phab.wmflabs.org and running puppet to update apache config/rewrite rules T356530 [20:59:55] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [20:59:55] T356530: clean up phabricator test host situation - https://phabricator.wikimedia.org/T356530 [21:00:24] mutante: I would recommend migrating to the newer TLD wmcloud.org as soon as is convenient for your project. The wmflabs.org domain is not going anywhere in the foreseeable future, but it is "legacy" these days. Any foo.wmcloud.org proxy should also respond when addressed as foo.wmflabs.org too assuming the downstream vhost also accepts the wmflabs.org hostname. [21:04:25] bd808: ACK! yea, I need to cleanup both sides, the proxies in Horizon and also the vhost config on the backend [21:04:41] ok, thanks, let me get rid of anything "labs" [21:06:16] * bd808 is no curious how long it has been since we made wmcloud.org available... [21:07:52] Creation Date: 2017-06-07T22:58:08Z [21:10:17] Feburary 2020 is when I'm seeing the first commits mentioning wmcloud.org in ops/puppet.git so it was probably around then that we started to actually use it. [22:26:59] !log devtools - deleted proxies phab.wmflabs.org, phab-prod-usercontent.wmflabs.org, phabricator.wmflabs.org - created proxies phabricator.wmcloud.org, phab-usercontent.wmcloud.org - wmflabs names are legacy and should migrate T356530 [22:27:03] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [22:27:03] T356530: clean up phabricator test host situation - https://phabricator.wikimedia.org/T356530 [22:31:58] !log devtools - phabricator-bullseye, instance hiera: setting phabricator_domain to phabricator.wmcloud.org and phabricator_altdomain to phab-usercontent.wmcloud.org T356530 [22:32:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [22:37:21] !log devtools - phabricator-bullseye - /srv/deployment/phabricator/deployment/phabricator/bin$ sudo ./config set phabricator.base-uri 'http://phabricator.wmcloud.org/' T356530 [22:37:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [22:37:26] T356530: clean up phabricator test host situation - https://phabricator.wikimedia.org/T356530 [22:53:05] !log devtools - phabricator-bullseye sudo ./config set phabricator.base-uri 'http://phabricator.wmcloud.org/' | sudo ./config set security.alternate-file-domain 'https://phab-usercontent.wmcloud.org' | delete proxy phab-prod-usercontent, create proxy phab-usercontent.wmcloud.org, restart apache T356530 [22:53:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [22:53:10] T356530: clean up phabricator test host situation - https://phabricator.wikimedia.org/T356530 [22:53:35] oh well, somehow the new proxy doesnt exist, but the old one pointing to the same backend does [22:54:39] it's in DNS but "We can’t connect to the server at" [23:15:20] Hey mutante could you help me with something please? [23:15:33] I created a new instance and I'm trying to ssh in using ssh -J jdlrobson@bastion.wmcloud.org jdlrobson@skins-2024.reading-web-staging.eqiad1.wikimedia.cloud but its complaining about my key.. [23:15:42] Also tried ssh -J jdlrobson@bastion.wmcloud.org jdlrobson@skins-2024.reading-web-staging.eqiad1.wikimedia.cloud -i ~/.ssh/id_wikitech but that didn't work either [23:15:52] it says skins-2024.reading-web-staging.eqiad1.wikimedia.cloud: Permission denied (publickey,gssapi-keyex,gssapi-with-mic). [23:22:05] Jdlrobson: it won't let me in with my root key either. Let me see what's on the Horizon logs for that instance... [23:23:05] okay glad it's not me this time bd808 :) [23:23:10] I usually do something dumb [23:24:50] It's built with an image that I don't think will ever work here: Fedora-CoreOS-38. andrewbogott any idea why the reading-web-staging project would have access to a Fedora image? [23:25:51] Jdlrobson: you will need to build your instance with one of the Debian base images in order to get ssh to work. [23:26:17] I think that Magnum uses fedora. I wouldn't have expected that to be visible in Horizon, though, probably I need to do some monkeying with service roles. [23:26:18] ahh so that was the dumb thing I did haha [23:27:44] While I have you bd808 do you also know if there is any HTTP throttling going on on cloud vps? I have a script that runs hourly and using puppeteer opens 100 browser windows (100 HTTP requests plus any associated JS/CSS/IMG resources). It gets stuck after about 20 of them and starts triggering timeout errors. I was going to revise the code to add sleep commands in between calls but wanted to check if there are any other [23:27:44] options (it works fine locally) [23:27:47] andrewbogott: Magnum / Trove was my guess at why we would even have those [23:27:55] okay I'm rebuilding that machine now... [23:28:15] * andrewbogott makes T356547 [23:28:28] It's messy, Trove requires Ubuntu, Magnum requires Fedora [23:28:42] But Trove Vms are contained to the Trove project so it's hidden nicely away [23:28:59] bd808: any reason not to use eeny, meeny, miny, moe when choosing between bulleye or bookworm? [23:29:13] Jdlrobson: use bookworm, it's newer so will get killed off later [23:29:28] unless you have specific need for bullseye [23:30:09] the buster -> bullseye -> bookworm Debian 10 -> 11 -> 12 codename progression is oh so confusing to me too. [23:30:56] I think that horizon displays #s as well as names though? [23:31:11] great [23:31:12] * andrewbogott accidentally created a new irc chatroom [23:31:44] Jdlrobson: there is not any outbound traffic shaping in Cloud VPS or Toolforge, but I think the inbound HTTP proxies for both have some per ip rate limits to try and prevent a few trivial attacks. [23:32:00] bd808: it got slightly better for me once I realised (or was told?) that it’s in reverse alphabetical order ^^ [23:32:10] but I’m still very cross with whoever decided to have three b* releases back to back [23:33:15] bd808: okay so now I'm getting the same problem with the new instance.. ssh -J jdlrobson@bastion.wmcloud.org jdlrobson@skin24.reading-web-staging.eqiad1.wikimedia.cloud -i ~/.ssh/id_wikitech [23:33:34] I guess I have to wait for something to happen? [23:34:24] everything less than bookworm is technically old or "oldold" by now, bookworm is 'stable' [23:34:40] Jdlrobson: are you able to access other cloud-vps VMs with that key? [23:34:45] Jdlrobson: yeah, you can watch the console logs at https://horizon.wikimedia.org/project/instances/099c5638-065d-46ae-824f-4885409c8dac/ You should be able to ssh in once puppet has run once and that log shows a login prompt on the console. [23:34:56] !log devtools phabricator-bullseye:/srv/deployment/phabricator/deployment/phabricator/bin$ sudo ./auth unlock [23:34:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [23:35:53] ack [23:37:13] !log devtools phabricator-bullseye configured auth provider for simple passwords, letting users register users, locked auth config option again [23:37:15] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL [23:39:39] Jdlrobson: you can probably ssh in to that instance now [23:43:35] yep it's working but now it's not picking up the puppet changes (vagrant) [23:44:22] (going to follow instructions at https://wikitech.wikimedia.org/wiki/Help:MediaWiki-Vagrant_in_Cloud_VPS#Setting_up_your_instance_with_MediaWiki-Vagrant) [23:45:39] hmm Error: /Stage[main]/Lxc/File[/etc/lxc/default.conf]: Could not evaluate: Could not retrieve information from environment production source(s) puppet:///modules/lxc/bookworm/etc-lxc-default.conf [23:45:42] Jdlrobson: ugh. It looks like the Vagrant role has not be updated to work on Bookworm. The puppet run is failing for a missing file that varies on os version. [23:45:52] 😭 [23:46:00] Should I create a ticket? [23:46:49] yes please. and then if you need to move forward now go ahead and build yourself a 3rd instance using bullseye as the base. [23:49:54] * bd808 wonders how hard it is going to be to get role::labs::mediawiki_vagrant working on bookworm [23:51:01] Hashicorp did licensing violence to Vagrant too I think when they relicensed Terraform and some of their other stuff [23:52:53] sigh, yeah it is "Business Source License" these days [23:55:30] eh, Debian is still packaging an older version that actually had an OSI-approved license.