[00:00:30] Forwarded from wmtelegram_bot: it`s asking to add one like node16 in terminal, can you tell me the command to start lightpd webservice command to start node16? [00:13:15] !log switched toolforge mounts on labstore1004 to read only, T333477 [00:13:16] andrewbogott: Unknown project "switched" [00:13:16] T333477: Migrate tools nfs from labstore1004 server to a ceph-backed VM - https://phabricator.wikimedia.org/T333477 [01:05:01] Time to figure out what normal people do on Sunday nights instead of deploying new code to Toolforge 🤔 [01:15:55] sorry legoktm! [01:16:09] :p [04:29:14] can anyone help me to deploy react in tool forge? [04:30:33] https://github.com/wikimedia/WikiContrib/blob/master/DEPLOYING.md [04:30:34] i used this link, but now i think its not updated [04:44:37] Rishav_a2z91: most of toolforge is offline for the next 12 hours or so, so step one is 'wait until tomorrow' [06:52:52] !log tools stop jobs-framework-emailer to prevent spam due to NFS being read-only T333477 [06:52:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [06:52:57] T333477: Migrate tools nfs from labstore1004 server to a ceph-backed VM - https://phabricator.wikimedia.org/T333477 [07:39:28] Why my wikimedia cloud account is in read-only mode [07:40:13] Oh got it [08:25:55] Still getting loads of emails from failed grid jobs (re @wmtelegram_bot: !log tools stop jobs-framework-emailer to prevent spam due to NFS being read-only T333477) [08:26:29] glad to see I'm not the only one. I deleted hundreds of grid mails since morning :-(. [08:27:19] hmmm [08:28:18] taavi: maybe the mailserver is having troubles with coping with its queue? [08:28:19] !log tools stop exim4.service on tools-sgecron-2 T333477 [08:28:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [08:28:24] T333477: Migrate tools nfs from labstore1004 server to a ceph-backed VM - https://phabricator.wikimedia.org/T333477 [08:28:51] urbanecm: the component I stopped is only about k8s emails, not grid emails [08:28:55] ah [08:29:07] ignore me then :) [08:29:14] but now I also stopped the mail relay on the cron server, which should stop the grid mails too [08:29:25] thanks! [08:31:17] * urbanecm received two grid mails now [08:32:46] define 'now' please? [08:33:33] Received: from tools.heritage by tools-sgecron-2.tools.eqiad1.wikimedia.cloud with local (Exim 4.92) (envelope-from ) id 1pjFcc-0000Wm-0M for tools.heritage@tools.wmflabs.org; Mon, 03 Apr 2023 08:33:02 +0000 [08:33:52] (the previous one was Mon, 03 Apr 2023 08:30:10 +0000) [08:38:10] ok now it really should be fixed [08:38:32] turns out local mail sending does not need exim4.service to be running.. so I just edited the config file to blackhole everything [08:39:17] thanks again! [11:41:58] hi folks, I need a little help! I'm trying to update an app on toolforge but the system seems to be read-only for some reason I am not aware - the same happens with other toolforge projects I access. For more context: I access via ssh and then `become TOOL_NAME`. Any hints? thanks in advance :) [11:42:43] isaranto: Scheduled toolforge NFS outage in progress [11:43:11] isaranto: yes, there is ongoing maintenance. see the latest cloud-announce@ post for details, and subscribe to that list if you have not already to get notified in advance in the future [11:43:12] clear, thanks! sry I missed that [11:59:19] isaranto: subscribe here: https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.org/ [12:05:08] done, thanks! [13:19:46] !log forcing puppet run on all toolforge VMs [13:19:48] andrewbogott: Unknown project "forcing" [13:19:56] !log tools forcing puppet run on all toolforge VMs [13:19:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:29:56] from today i am facing the issue of a read-only file in my tool. can anyone know how to fix this. thanka is advance:) [13:30:13] context toolforge [13:37:06] Rishav_a2z18 see https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/Q4C4QW5X4ATG3ANRO6CMDIWCG42YM6NJ/ for details [13:38:05] ohh thanks for info Wurgl [13:38:41] and subscribe here https://lists.wikimedia.org/postorius/lists/cloud-announce.lists.wikimedia.org/ so you get notified next time [13:39:06] sure [13:40:03] https://github.com/wikimedia/WikiContrib/blob/master/DEPLOYING.md [13:40:03] i think above instructions are not working anymore [13:40:39] ok, you should contact the person who made the instructions [13:41:58] yeah i will raise the issue. [14:12:46] stashbot, how's it going? [14:12:56] See https://wikitech.wikimedia.org/wiki/Tool:Stashbot for help. [14:12:59] !log tools test log to see if stashbot is back working [14:13:20] hmmm [14:13:22] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:13:26] ah! [14:22:00] !log tools.stewardbots Restarted StewardBot and SULWatchers which dropped out of IRC [14:22:04] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [14:31:55] !log paws pywikibot version bump c385d673b9a9fedd29f7ef6e586f527a93ae2f50 T333482 [14:32:00] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [14:32:00] T333482: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T333482 [14:47:24] Hello, I think something is weird with my recently created "duct" openstack project (this is my first time using it.) On the overview page, I can't see any quota information, and I'm getting this error message: "Error: Unable to retrieve limits information." I'm also unable to create a new instance (the "Launch Instance" button doesn't appear.) I'm wondering if it's because my user for this project has the roles [14:47:24] "projectadmin_renamed_for_T330759, user_renamed_for_T330759" and is notably missing "member" and "reader" which I have in other projects. [14:47:47] try logging out and back in? [14:49:58] Same error [14:50:04] oh right, I see the roles are actually wrong, give me a sec [14:50:14] Thanks! [14:51:43] try now [14:54:20] Works! Thanks taavi <3 [15:16:14] Hello, how would I query the wikidata replica database for labels? I want to get all items where the "en" label starts with a given string, but I have not a clue which table contains the raw label text. [15:17:51] Are you looking for help getting access to the replica or documentation on the database schema? [15:18:15] Documentation on the schema [15:19:36] https://www.mediawiki.org/wiki/Manual:Database_layout is my go-to resource. [15:19:37] RPI2026F1: https://doc.wikimedia.org/Wikibase/master/php/docs_storage_terms.html [15:20:05] Thank you. I was using `SHOW TABLES;` and tinkering around with SELECT * FROM TABLE LIMIT 1 so far. [16:09:14] Hey Cloud admins/folks -- I wanted to check in around an exception to hosting open-source on Cloud VPS. The documentation suggests reaching out to cloud admins (https://wikitech.wikimedia.org/wiki/Wikitech:Cloud_Services_Terms_of_use#What_uses_of_Cloud_Services_do_we_not_like?) but I wasn't aware of a listserv etc. so am looking for guidance on the best next steps. [16:09:34] A little context: As part of the upcoming May hackathon, we'd like to showcase some AI models that tool developers might think about using in their work. While some models are unequivocally open (code is open-source; model is also explicitly open-source), others that we are curious about because they have better performance and lower footprint than known existing open models have some use restrictions for the models [16:09:49] e.g., non-commercial clauses (https://ai.facebook.com/blog/large-language-model-llama-meta-ai/) or ethical use restrictions such as not using for the purposes of creating disinformation (https://bigscience.huggingface.co/blog/the-bigscience-rail-license). These models do not currently meet the OSI definitions but OSI's recent report on this (https://deepdive.opensource.org/report/) indicates there is active discussion about what Open [16:09:50] and AI should look like. [16:09:53] While I don't think we're ready to take a stance on whether these restrictions are acceptable for our systems, I do think they qualify as a reasonable temporary exception for the purpose of testing / exploring for the hackathon. For clarity, we would only be using OSI open-source code (MIT, Apache, etc.) -- this is just about the model weights which are the more debatable area. [16:10:38] in summary: I don't need a decision right now but wanted to know the best way to request one [16:12:28] isaacj: I think a phab task (with cut-n-paste) is probably the best place for that. My instinct is that we'll tell you no but maybe there's some loophole because it's data and not code. [16:13:32] thanks andrewbogott -- having it documented in a public, easily-accessible place would be nice even if that's the outcome [16:13:41] yep! [16:13:49] (to the extent that phab is those things) [16:13:55] :) [16:14:53] any particular tag? I'd just default to Cloud-Services [16:15:22] there's also https://phabricator.wikimedia.org/tag/cloud_services_proposals/ which is usually for internal things but has some process attached to it [16:15:32] cloud-vps and/or cloud-services tags [16:17:56] :thumbs up: I'll do cloud-services because that seems like the appropriate umbrella even if i'm thinking cloud vps for this particular exception [17:16:37] we're back up I believe, please let us know if you see anything odd [17:24:23] \o/ [17:27:39] NFS back working then? [17:28:32] looks like both the webservice and background runner in the quickcategories tool came back to life, nice [17:29:18] herzog: yes! [17:29:43] thanks taavi, reloading toolforge-jobs yaml then [17:39:08] taavi: did you restart the mail stuff? I didn't receive emails for my k8s jobs. [17:43:17] Thanks for the help earlier, but now I have a new problem, which is that I have no idea how to query all the statements for a given item. Let's say I have the id for 'Q5', I can get its labels and aliases but I'm not sure how to get all of the claims. With the query service I would do `wd:Q5 qdt:PXXX ?value` but I'm not sure in which format claims [17:43:17] are stored in the database. [17:43:33] `wdt:PXXX`* [17:44:27] Nevermind I think I have it [17:46:58] that would surprise me, statements aren’t available in the SQL replicas [17:47:36] (the pagelinks table can sometimes be used as an approximation but you should be aware of the limitations of that approach) [17:50:02] JJMC89: not yet, I'm planning on waiting a bit for things to stabilize [17:50:50] Before you unleash the flood of cron emails? (re @wmtelegram_bot: JJMC89: not yet, I'm planning on waiting a bit for things to stabilize) [17:53:34] Statements are not available? Is the query service the only way to get it? I just want to query items that have no statements [17:54:06] That's in SPARQL. You can just look for items that have that set to zero (re @wmtelegram_bot: Statements are not available? Is the query service the only way to get it? I just want to query items that have no s...) [17:54:40] www.wikidata.org/entity/Q5.rdf : 202 [17:56:09] searching for items with *no* statements should be possible via the page_props table [17:56:30] using the wb-claims page prop [17:57:21] but you can also do it in a query, e.g. https://w.wiki/6Xkq [17:57:40] or even in the normal search, by searching for -haswbstatement:* [18:03:08] That's not completely true because search doesn't index all types of statements. For example date statements are not in the search index. Not sure why these are not indexed. (re @lucaswerkmeister: or even in the normal search, by searching for -haswbstatement:*) [18:05:57] hello! grid engine job 1523844 for `tools.eranbot` has been stuck in the deletion state (`dr`) for a long time. Could someone kill it for me? [18:22:49] Hello o/ I am not able to sudo on VMs created in my CloudVPS project, duct. It also appears that cloud-final.service is failing on the initial boot; however, I can't see why because I can't sudo. ;) Logs from Horizon: https://phabricator.wikimedia.org/P46011 [18:25:48] kindrobot: puppet/cloud-init look ok to me. I think something didn't get set with the initial sudo rules though, let me investigate. Happen to know when this project was created? [18:28:42] Roughly two weeks ago. I'll try to find the phab ticket. [18:29:16] andrewbogott: 2023-04-03T18:13:12Z per https://openstack-browser.toolforge.org/server/jump.duct.eqiad1.wikimedia.cloud [18:29:24] thx [18:29:32] oh, that's instance not project :) [18:30:10] March 10 was when the project was created. [18:30:20] https://phabricator.wikimedia.org/T331741 [18:30:35] Instance was approximately 30 minutes ago. :D [18:31:10] I think taavi did a role fix for kindrobot earlier today. see backscroll around 14:47 UTC [18:31:35] That's right. [18:31:55] The instance was launched after the role fix. [18:43:19] kindrobot: I can't reproduce the error that caused this. Mind if I delete that project and try all over again? [18:43:24] project, vm, everything [18:43:42] Sure thing. :) [18:43:45] thx [18:46:37] kindrobot: done, mind trying again? [18:47:06] * kindrobot tries again [18:54:35] It's working. Thanks andrewbogott <3 [18:55:12] sure thing. No idea why it didn't work the first time. [19:40:41] musikanimal: done! [19:40:54] thank you! [19:45:31] !log xtools bump quotas by 5 VCPUs and 10GB RAM T205232 [19:45:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Xtools/SAL [19:45:35] T205232: Request increased quota for xtools Cloud VPS project - https://phabricator.wikimedia.org/T205232 [20:50:35] Hi, looks like my grid engine job 1172179 ("dyk-counts", tool name sdzerobot) is stuck in a being deleted stage. Could someone force-delete it? [21:00:33] sure, give me a sec [21:01:09] sd0001: gone [21:02:09] taavi: thanks! [22:41:19] I have a very stuck queue job since the downtime yesterday and using the method in Wikitech I'm unable to kill it. I have been away from wiki for sometime and would like to migrate to Kubernetes for jobs, but just need to get this old one stopped first. Anyone able to assist? [22:46:33] JamesR: I have the super powers to kill stuck grid engine jobs. What's the job id? [22:47:37] * bd808 notices that the `!help` magic trick is missing from the current topic [22:47:59] bd808: 40794 [22:48:58] Tried everything in https://wikitech.wikimedia.org/wiki/Obsolete:Toolforge/Grid#Stuck_jobs also [22:49:45] !log tools.aivhelperbot Force deleted job 40794 via `sudo qdel -f 40794` per IRC request from JamesR [22:49:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.aivhelperbot/SAL [22:50:01] Thank you, most appreciated [22:50:18] JamesR: no problem, there are ways that things can get wedged which require root user magic.