[02:14:28] Southparkfan: thank you for the offer! I just created T442482 and will make my way through the staff-assigned items soon in case you want to be among the (TBD) assignees :) [02:14:28] T442482: Secure deployment-prep against php2shell and restore access - https://phabricator.wikimedia.org/T442482 [02:49:12] !log anticomposite@tools-bastion-15 tools.yifeibot webservice stop [02:49:14] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.yifeibot/SAL [02:49:23] !log anticomposite@tools-bastion-15 tools.yifeibot webservice php8.4 start [02:49:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.yifeibot/SAL [09:25:27] !log admin restore Horizon access [09:25:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [13:04:38] I guess the topic could be switched back to saying Up, if we are pretty much back up [13:18:18] url shortened so that the CoC link fits again [13:24:24] !log tools.paws migrate to https://wikitech.wikimedia.org/wiki/Tool:Containers#Redirect_container [13:24:26] taavi: SAL for "tools.paws" is deprecated. Use "!log paws" instead. [13:25:13] no, I meant tools.paws [13:52:11] I think I may have put a specific hack in stashbot to whine about tools.paws… [13:58:18] https://gerrit.wikimedia.org/r/c/labs/tools/stashbot/+/732418 — it’s all my fault [13:59:23] !log paws migrate tools.paws to https://wikitech.wikimedia.org/wiki/Tool:Containers#Redirect_container [13:59:25] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Paws/SAL [13:59:26] let's just do that then [14:10:43] !log tools.admin-beta migrate to php 8.4 [14:10:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.admin-beta/SAL [14:11:34] !log tools.stewardbots-legacy migrate to php 8.4, filed T442540 [14:11:36] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots-legacy/SAL [14:11:36] T442540: Fix stewardbots election tool to run properly on PHP 8.4 - https://phabricator.wikimedia.org/T442540 [14:41:03] !log admin "openstack role create secgroupadmin" for T442554 [14:41:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [14:41:09] T442554: Create temporary secgroupadmin keystone role - https://phabricator.wikimedia.org/T442554 [14:41:43] !log admin openstack role add --project paws --user magnum secgroupadmin T442554 [14:41:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [15:09:43] any API or wrapper for toolforge-jobs system? [15:12:40] Yetkin: https://wikitech.wikimedia.org/wiki/Help:Toolforge/API [15:15:37] !log tools round of pod restarts to pick up new pre-built images [15:15:42] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [15:18:34] I'm not sure why but builds-api still seems to not be very happy (T442194), list is a lot quicker since T442364 but components-api is still timing out a bunch (https://github.com/cluebotng/component-configs/actions?query=is%3Afailure). I have a workaround to delete all the builds then re-run a deployment manually, but that is somewhat sub-optimal. [15:18:35] T442194: [builds-api] very slow to response to image list / [components-api] has constant timeouts to api.svc.tools.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T442194 [15:18:35] T442364: [builds-api] returns incorrect parameter values for use_latest_versions & use_deprecated_versions causing [components-api] to not re-use existing builds - https://phabricator.wikimedia.org/T442364 [15:19:05] Is there a way to easily list all Toolforge tools of one maintainer nickname? Potentially useful when an user account is cryosleeping, inactive, or compromised. [15:19:06] Maybe a new epic Striker task [15:20:34] https://ldap.toolforge.org/user/damian (see 'service groups') [15:20:45] bozzy: https://ldap.toolforge.org/user/dcaro the service groups `tools.*` are the tools that user is member of (note it will not resolve nested groups) [15:20:55] cool ❤️ [17:27:21] !log codesearch codesearch10: re-enabling puppet to apply 1358873 (hound containers with --init) - restarts all hound-* containers, search incomplete for ~1h while indexes rebuild. Fixes leaked zombie git processes (houndd as PID 1 did not reap them; prod hosts reached 160k+ and ran out of threads) T26819 [17:27:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL [17:27:28] T26819: Default minor/not minor edit preference is forced to all replies/new threads edits - https://phabricator.wikimedia.org/T26819 [17:27:49] oops, wrong ticket because my message got cut off [17:27:54] !log codesearch codesearch10: re-enabling puppet to apply 1358873 (hound containers with --init) - restarts all hound-* containers, search incomplete for ~1h while indexes rebuild. Fixes leaked zombie git processes (houndd as PID 1 did not reap them; prod hosts reached 160k+ and ran out of threads) T268199 [17:27:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL [17:27:59] T268199: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199 [19:09:38] could someone with an existing roster of PAWs notebooks please log into paws and confirm that your old notebooks are available? [19:11:08] * lucaswerkmeister looks into paw [19:11:24] ty [19:11:29] it takes ages to launch right now [19:11:31] meanwhile, is anything wrong with k8s right now? ww-monitor has a deployment in CrashLoopBackOff and no logs and I don’t know what might be wrong :/ [19:11:56] nothing known to be wrong [19:11:58] (`kubectl logs deployment/monitor` returns empty) [19:12:00] hm ok :/ [19:14:29] I see Untitled.ipynb, Untitled1.ipynb, Untitled2.ipynb from 6y ago ^^ [19:14:43] and they still have content? [19:14:56] I can open them and look at them, at least [19:15:00] haven’t tried running anything yet [19:15:04] but I see the old commands and output [19:15:13] great! thank you for checking [19:15:27] np [19:15:28] lucaswerkmeister: tools.ww-monitor@tools-bastion-14:~$ ls -lah ./www/rust/target/release/masto-monitor [19:15:28] ls: cannot access './www/rust/target/release/masto-monitor': No such file or directory [19:15:28] tools.ww-monitor@tools-bastion-14:~$ toolforge jobs dump [19:15:28] - command: ./www/rust/target/release/masto-monitor [19:16:36] that's being logged on to monitor.err, btw [19:16:52] aha, filelog [19:16:53] thanks [19:17:07] One Day we will get around to getting rid of file logs, I home [19:17:07] www/rust/target/artifacts/masto-monitor does exist [19:17:12] (and is an ELF) [19:17:29] did rust change the path? [19:17:31] * lucaswerkmeister to the duckduckgomobile [19:17:50] I have vague memories that this all is automated by some ansible playbook that legoktm has somewhere [19:19:37] Oh [19:20:05] trying a local cargo build --release now to see where it’ll end up [19:20:10] but happy to leave the production part to legoktm ^^ [19:20:12] Sorry I know what it is [19:20:33] glibc and trying to run a trixie build on bookworm [19:20:55] hello everyone [19:21:07] lucaswerkmeister can I bother you regarding m3api ? (maybe after this) [19:21:47] Gouvernathor: if it’s a specific question, maybe [19:22:03] if it’s about the whole m3api oauth / auto-import area in general, I’ll get to it eventually, but there are some other things on the TODO list at the moment :) [19:22:49] (my last recollection is that the auto export seemed to work quite well in wdactle, so I’ll probably include that as a semi-experimental feature in the next minor release, with an eye towards making it the default export in the next major release) [19:23:12] it's not about the auto-import, but about T437364, checking if an auth is still valid by requesting the user-info route [19:23:12] T437364: [oauth2] testing validity of the grant and requesting the userinfo/profile route - https://phabricator.wikimedia.org/T437364 [19:23:20] handy thing, that [19:24:28] anyway : I don't know how to solve this cleanly, so I would appreciate your input, but I can hack my way into it and bypass it if you're not likely to work on it in the short term [19:25:27] taavi: lucaswerkmeister: ok, fixed now sorry. Yes I did change paths to target/artifacts/ and forgot to update the jobs.yaml. And also fixed the bookworm->trixie which would've bitten once the path was correct [19:25:49] thanks legoktm \o/ [19:25:55] https://gitlab.wikimedia.org/legoktm/toolforge-ansible/-/blob/main/roles/tfrust/files/fetch-rust-binary.py?ref_type=heads [19:26:14] it's on my list to change this all to push-to-deploy so it's harder to screw up [19:26:49] Gouvernathor: I think the honest answer is I’m not likely to work on it in the short term tbh [19:27:02] interaction between m3api-rest and m3api-oauth2 is probably an open question in general, not sure I was aware of that before [19:27:28] like, back when OAuth errors were Action API errors, this didn’t really make sense as a consideration, because the REST API has its own errors anyway [19:27:48] but now, at least in Wikimedia production, OAuth errors are actually HTTP-level errors, so I had to add support for that to m3api-oauth2 [19:28:16] at which point it should *in principle* be possible to make that work for REST requests as well, but how to actually make that work out in the stack of request/fetch network methods… [19:28:48] (the changed OAuth error is https://phabricator.wikimedia.org/T419034 ftr) [19:29:35] lemme leave a note in that direction on the task at least [19:30:40] ok, thanks and no worries :) [19:37:50] legoktm: do you have time to poke at https://toolsadmin.wikimedia.org/tools/id/php-security-checker or mind if I do? it's down because of php 7.4 and blocking libup runs [19:38:05] oh go for it [19:38:49] I officially added you as a maintainer :) [19:38:55] oh dear [19:39:28] let's see of 'webservice stop ; webservice php8.4 start' is enough [19:39:55] !log tools.php-security-checker port to php 8.4 [19:39:57] tbh that should probably be rewritten to use composer/packagist's new API. Also no reason for the webservice itself to be in PHP [19:39:58] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.php-security-checker/SAL [19:41:26] or just have libup shell out to `composer audit` the same way it does for npm? [19:41:57] yeah, there's a bunch of libup stuff i should eventually get to but haven't yet [19:42:54] libup's not crashing on that at least so i'm calling it a success [19:43:10] <3 [19:43:30] !log libraryupgrader arm keyholder [19:43:31] taavi: Unknown project "libraryupgrader" [19:43:31] taavi: Did you mean to say "tools.libraryupgrader" instead? [19:43:49] !log library-upgrader arm keyholder [19:43:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Library-upgrader/SAL [20:04:55] So I'm working on some experimental global templates-related stuff for Wikiproject Medicine Foundation and we need to have a couple of test wikis hosted where we can run with a small extension installed to hack some stuff in. My understanding is that putting whole-ass MediaWikis is still not a good fit for Toolforge and that cloud VPS remains the best way to go for these sorts of things? [20:05:34] (I have some separate things I want to put on Toolforge that will do client-side video and image processing, so they don't have heavy server-side requirements. I think those will fit there well, but I won't have time to get to this for a few weeks.) [20:29:41] bvibber: you should be good on toolforge these days. see https://gitlab.wikimedia.org/toolforge-repos/mwdemo for an example [20:31:02] oh sweet! [20:31:13] hah and i just finished a phab ticket for cloud vps lol [20:38:41] hm [20:38:45] $ become globaltemplatestest [20:38:46] sudo: a password is required [20:38:51] what.... does that mean [20:39:04] it wanst you to log out and back in, probably [20:39:30] huh weird [20:39:34] that worked, i have no idea why [20:54:05] taavi or bliviero, the quota tool is indisposed but can I get a +1 on https://phabricator.wikimedia.org/T442643 so we get subbu unblocked on a nodejs thing? [20:59:32] nm, doing [21:01:37] +1 [21:03:16] thx! [21:05:20] !log quarry redeploying from git 'main' branch in /home/root/quarry on quarry-bastion.quarry.eqiad1.wikimedia.cloud [21:05:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Quarry/SAL [21:13:42] So, "sudo wmcs-prepare-cinder-volume" says: "This tool will partition, format, and mount a block storage device.". But, I am trying to attaching a volume detached from an old VM to a new VM. Presumably, this will only mount it, not reformat it. Correct? [21:16:38] yes. It should also prompt you before doing anything too drastic. [21:16:50] ok. [23:23:34] !log codesearch - up and running and now has 0 git zombie processes - since it now runs with docker --init to tini can reap git zombies, which hound as pid 1 would not do. before it was about 1000 new git processes per day. in prod it ran out of pids within days. [23:23:39] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Codesearch/SAL [23:25:41] Hmm, ok so I'm trying the MediaWiki-on-toolforge installer forked from https://gitlab.wikimedia.org/toolforge-repos/mwdemo ... modified my config file with the project name... running the installer, it checks out all the repos and then tries to run the MediaWiki installer I think which craps out with `Cannot access the database: :real_connect(): (HY000/1045): Access denied for user 's58293'@'172.16.5.0' (using password: YES).` [23:26:04] It appears to pick up the db username and password from the environment, but I guess something isn't right. [23:26:21] Not sure if I've missed a step, gonna double-check stuff. [23:28:30] same error connecting with `mariadb --defaults-file=$HOME/replica.my.cnf -h tools.db.svc.wikimedia.cloud` [23:28:54] or with `sql tools` [23:31:55] same on a fresh login [23:32:27] I think that means the db credentials didn’t get provisioned correctly and someone has to fix it, but I don’t have the permissions for that 😔 [23:32:38] ooh, an adventure :D [23:35:13] (https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin#Regenerate_replica.my.cnf is the link I have bookmarked fwiw) [23:36:32] oh, or try this https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Rotate_your_credentials [23:36:45] (found via https://lists.wikimedia.org/hyperkitty/list/cloud-announce@lists.wikimedia.org/thread/FUKPBXANMVX63DGBNQ2HMNG35PQ3ZBGF/ → https://wikitech.wikimedia.org/wiki/News/2026_Rotating_Credentials) [23:37:05] maybe that’ll work ^^ [23:38:04] thanks, giving the rotation thingy a try :D [23:39:34] well it gave me a different password but it still doesn't appear to work [23:42:44] aww :( [23:54:19] filed a task https://phabricator.wikimedia.org/T442660 feel free to recategorize etc :)