[04:26:32] !log wikisp Mars instance launched (again) [04:26:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikisp/SAL [06:58:25] legoktm: that should be set now too [07:40:26] !log cloudinfra-nfs remove "arturo" from project members and instead add "aborrero" which is the real one [07:40:28] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Cloudinfra-nfs/SAL [11:13:07] !log tools removed tons of duplicate qw jobs accross multiple tools [11:13:10] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:32:30] !log tools depool tools-sgeexec-0910 T294228 [11:32:34] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [11:32:34] T294228: urbanecmbot's continuous jobs are getting restarted too frequently - https://phabricator.wikimedia.org/T294228 [12:25:30] quit [14:33:16] !log tools copy nginx-ingress controller v1.0.4 to internal registry T292771 [14:33:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [14:33:20] T292771: upgrade to ingress-nginx 1.0 - https://phabricator.wikimedia.org/T292771 [14:41:08] !log toolsbeta deploy ingress-nginx v1.0.4 to toolsbeta via helm, diff only changes the image T292771 [14:41:11] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [14:41:11] T292771: upgrade to ingress-nginx 1.0 - https://phabricator.wikimedia.org/T292771 [15:26:05] majavah: is there any easier solution to this puppet signing certs issue on my gitlab-runner-addshore-1004 instance that nuking the wheol thing? [15:27:14] addshore: yes, integration has a local puppetmaster, just follow https://wikitech.wikimedia.org/wiki/Help:Standalone_puppetmaster#Step_2:_Setup_a_puppet_client [15:28:00] really I should just apply for a cloudVPS project of my own for this 1 mwcli related gitlab runner! [15:29:10] I thought there already was a project for gitlab runners? [15:29:45] that is for shared runners & this instance was created before the birth of that project and is right now still only used by a single gitlab repo [15:51:45] majavah: ty! I got an email "[FIRING:1] InstanceDown library-upgrader (upgrader-06 node warn)" - I'm not sure what that means, the host looks fine to me [15:51:58] Also the link in the email goes to http://metricsinfra-alertmanager-1:9093/#/alerts?receiver=library-upgrader_libup [15:56:29] oops, that link needs to be fixed, https://vpsalertmanager.toolforge.org/?q=project%3Dlibrary-upgrader would probably be the best for now [15:56:55] legoktm: anyhow, prometheus thinks the LibUp VM and the web interface were unavailable for some time last night: https://prometheus.wmcloud.org/cloud/graph?g0.range_input=1d&g0.expr=up%7Bproject%3D%22library-upgrader%22%7D&g0.tab=0 [15:57:29] o.O [15:58:22] that might be correct, given the weird spike at https://nagf.toolforge.org/?project=library-upgrader&range=week [15:59:29] yay for monitoring working! [16:29:40] Hello world. My tool on grid is stuck. qdel, qdel -f is not working. "dt" status. What should I do? "iluvatarbot" toollacc [16:31:22] Iluvatar: I deleted it with a more powerful force, it seems to have disappeared now [16:31:40] Thanks! [16:40:30] a more powerful force than Iluvatar? :O [16:40:48] sudo -9 force [16:42:49] running the exact same command (qdel -f $JOB_ID) as root on the grid master node is somehow more powerful than running it as a normal user [17:00:18] !log library-upgrader temporarily stopped libup-celery service to test new monitoring/alerting [17:00:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Library-upgrader/SAL [17:07:55] !log toolhub Updated demo server to 5a4d3e2 [17:07:57] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolhub/SAL [18:38:12] Replag on s3 is https://phabricator.wikimedia.org/T294295 [18:47:44] !log tools.integraality Deploy 5dfef39, e940ae7, 765b314, 3e4c2de, c859c45, 5526cb3 [18:47:47] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [18:57:16] majavah: is https://phabricator.wikimedia.org/T294295 worth a heads up to anyone else [19:29:45] !log tools.lexeme-forms deployed 754342b9a3 (language name for bn-x-Q6747180) [19:29:48] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [19:31:14] !log tools.lexeme-forms belay that, the new pod hasn’t actually started properly. investigating [19:31:16] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [19:31:43] `kubectl logs` says: !!! UNABLE to load uWSGI plugin: /usr/lib/uwsgi/plugins/python_plugin.so: cannot open shared object file: No such file or directory !!! [19:32:11] maybe something changed in the container since the last pod started 7d6h ago? [19:32:46] that's expected I think [19:33:13] we tell uwsgi to load both python and python3, it uses what's inside the container [19:33:43] looks like the pod is running successfully now after all [19:33:57] just took a bit longer for whatever reason (and 1 restart, apparently) [19:34:03] false alarm [19:34:25] !log tools.lexeme-forms deployment was successful after all 🤷 [19:34:27] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [19:36:19] maybe failureThreshold 3 on the startupProbe is a bit low, I’ll bump that up a bit [19:38:04] majavah: if you have op here, it might be wise to change Status: Up to something that points to T294295 [19:38:05] T294295: db1112 (s3 contribs/rc replica) is down - https://phabricator.wikimedia.org/T294295 [19:38:23] !log tools.lexeme-forms deployed 0f5b5de66a (bump startupProbe failureThreshold 3→10) [19:38:26] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.lexeme-forms/SAL [19:39:55] like that? [19:42:20] looks good [19:42:21] thanks [20:15:15] !log tools.integraality Deploy 977236b (T284183) [20:15:19] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [20:22:02] !log tools.integraality Deploy ef0f537 (T284684) [20:22:05] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [21:11:22] discussion around archiving the classic notebook interface in favor of jupyterlab https://github.com/jupyter/notebook/issues/6210 [21:40:51] I'm trying to access mediawiki's database replicas. I've followed a guide and created an account on toolsadmin.wikimedia.org and added my SSH key under `profile/settings/ssh-keys`. I'm not sure where I can get `replica.my.cnf` file, I couldn't find that at the homepage of toolsadmin.wikimeida.org, so when I try to SSH through `ssh -v -L 4711:enwiki.analytics.db.svc.wikimedia.cloud:3306 [21:40:57] login.toolforge.org` I'm getting premission deined (publickey, hostbased) error. Could someone help me out? My goal is to access English Wikipedia's database replica throuhg commandline. [21:41:21] !log tools.integraality Deploy 7e3343a (T278156) [21:41:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.integraality/SAL [21:42:59] fentanyl: Most of this should be documented in https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#Connecting_to_the_database_replicas [21:43:28] I would suggest trying to connect to the replicas first via a SSH session directly, before trying to do it locally with ssh tunnelling [21:46:54] Reedy: You mean SSH to dev.toolforge.org? I tried to ssh through my toolsadmin username to dev.toolforge.org, but when the authentican succeded, it gives me `Connection closed 22` [21:47:52] Has your account actually been approved on toolsadmin? [21:48:14] * Reedy looks [21:48:17] I'm not sure. [21:48:26] Reedy: My toolsadmin username is wikilinuz [21:49:31] When did you signup? [21:50:09] Like within 30 mins [21:50:21] It's not even 30 mins [21:51:12] (If you mean signup for an account at toolsadmin.wikimedia.org) [21:52:10] There's no membership request for that username (at least, not today) [21:52:52] Have you done https://toolsadmin.wikimedia.org/tools/membership/apply ? [21:53:10] Oh, no. I'll apply now. [21:53:25] That'd explain why you couldn't SSH in at least ;) [21:54:15] Ok, I've submitted a requet. [21:54:31] I see it now [21:55:38] >Once you are added as a Toolforge member, you must log out and then log in again at https://toolsadmin.wikimedia.org/ [21:55:59] alright [21:56:44] Then.. If you added your ssh key correctly, and have your SSH config sorted/key loaded into an agent... SSH-ing in should work [21:56:56] Yeah, I can access a shell [21:57:08] progress! [21:58:09] Yeah, I can now see replica.my.cnf [22:05:22] Hmm, I've connected to the database. But `show tables;` outputting some list of hex values. [22:05:44] Like this: https://paste.debian.net/plain/1216871 [22:06:58] Some weird client options? [22:07:26] Reedy: This was my option: `mysql --user=uNUMBER --host=127.0.0.1 --port=4711 --password enwiki_p` [22:08:28] what does `status;` say? [22:08:49] fentanyl: does sql enwiki not work? [22:09:01] This is the status: https://paste.debian.net/plain/1216872 [22:09:12] Spookreeeno: I'm getting ready to go to work now [22:10:09] SQL: don't pick such an interesting name [22:10:45] lol [22:11:18] Maybe don't edit enwiki? :P [22:11:54] SQL: or don't touch databases [22:12:02] which feels like a good idea after today [22:12:34] Hmm, any idea? [22:13:00] fentanyl: i just ssh'd into dev.toolforge.org and did sql enwiki and it looks fine [22:13:16] https://www.irccloud.com/pastebin/F2x9oKld/ [22:14:21] I see, I don't know about that `sql` tool. Mine just ssh into the instance and use it there. [22:14:39] Client characterset: binary [22:14:39] Conn. characterset: binary [22:14:43] But, the experieance is too slow... [22:14:52] fentanyl: from your shell once ssh'd, you should be able to just do sql [22:14:54] vs [22:14:56] Client characterset: utf8 [22:14:56] Conn. characterset: utf8 [22:14:56] it's on path [22:15:09] Reedy: also the version [22:15:25] mysql Ver 8.0.26-0ubuntu0.20.04.2 for Linux on x86_64 ((Ubuntu)) [22:15:31] Spookreeeno: Yeah, that works, but the response is just slow. [22:15:34] atleast for me. [22:15:41] fentanyl: it wasn't 10 seconds ago [22:16:03] I guess your connection is better. [22:17:12] fentanyl: maybe try login.toolforge.org instead of dev [22:18:21] I'm trying to follow Reedy's suggestion. like trying to change binary to utf8 [22:19:44] Ok, I did `SET character_set_results = utf8` and it worked. [22:20:08] you can change it in your my.cnf [22:20:29] (rather than having to a SET every time you connect etc) [22:20:41] So, I need to append this line at the end right? character_set_results = utf8 [22:21:11] (I'm new to mysql and stuff) [22:21:16] [client] [22:21:16] default-character-set=utf8 [22:21:16] [mysql] [22:21:17] default-character-set=utf8 [22:21:20] like this ^ [22:24:22] mutante: Do I need to add this to the replica.cnf file found at login.toolforge.org? [22:24:31] (That file is read-only) [22:24:54] You probably only need it in your local version [22:25:39] Hmm, I didn't use that replica file to login into the database though. I'm not sure how that'd help. [22:26:01] Because I entered `mysql --user=uNUMBER --host=... --port=... --password DBNAME` [22:26:19] here, I didn't use the replica file itself (except for knowing the username and password) [22:26:27] which I typed manually [22:26:47] Or, am I mistaken something? [22:27:22] I followed: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database#SSH_tunneling_for_local_testing_which_makes_use_of_Wiki_Replica_databases [22:27:49] Sorry, I mean this: "Connecting to the database replicas from your own computer" [22:27:49] fentanyl: it would look automatically for certain config file names, do you see a file called .my.cnf where you run mysql ? [22:28:12] nope [22:28:39] * fentanyl reading https://wikitech.wikimedia.org/wiki/Help:Toolforge/Database [22:29:00] when you actually type mysql, is this on your local computer? [22:29:02] "You might need to copy over the replica.my.cnf file to your local machine for this to work." [22:29:31] fentanyl: is the actual mysql client on your own machine? you just connect via toolforge as tunnel? [22:30:28] then it matters what is in /etc/mysql/ and/or .my.conf in your home dir on your local machine [22:30:53] the docs seem to say that's why you need to copy replica.cnf or the settings from it [22:31:07] Yeah, its on my local machine [22:31:26] Yeah, I open an ssh tunnel and use that port [22:31:31] look inside replica.conf if it has the setting you want [22:31:45] copy that to a file called .my.cnf in your home dir at home and try again [22:33:21] Yeah, thanks! [22:36:01] make sure it starts with the . I gotta got, good luck. bbl [22:40:33] I'm learning about web proxies & Cloud VPS and had a question about how web proxies work. Is the Wikimedia Cloud team running a reverse proxy that connects the internal IP address with the externally accessible host + domain names? [22:41:12] let me find the chart [22:43:46] ariutta, there are multiple reverse proxies https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Kubernetes/Networking_and_ingress#Network_topology [22:44:24] Perfect, that explains it well [22:47:40] Would I be able to file a Phabricator ticket to request a web proxy for the domain wikipathways.org? (My group is working to port our biological pathway project to Cloud VPS.) [22:48:25] sorry, I didn't notice you mentioned Cloud VPS, not Toolforge [22:49:39] We could possibly use either Cloud VPS or Toolforge, but we'd like to continue supporting published wikipathways.org URLs [22:50:57] I think Item 7 here suggests it would be possible to create a web proxy for wikipathways.org: https://wikitech.wikimedia.org/wiki/Help:Using_a_web_proxy_to_reach_Cloud_VPS_servers_from_the_internet#Creating_a_web_proxy [22:51:40] "If you want a domain that is not already present in the menu, a cloud admin (most likely a staff member) will need to create it for you." -- I'm assuming this means to file a Phabricator ticket. [22:51:52] yup, I'd go to Phabricator for that [22:52:13] Cool, thanks! [22:53:18] https://www.wikipathways.org/index.php/Special:Version [22:53:20] Well, htat's scary. [22:53:23] >MediaWiki 1.13.1 [22:54:17] ariutta: Just to point out... Cloud VPS et al isn't arbitary web hosting [22:58:10] Reedy: good point. In this case, it's a project that has been approved as fitting within the Wikimedia mission -- curation and sharing of biological pathways. [22:58:21] aha, fair enough :) [22:58:29] :) [22:58:41] Just wanted to make sure, rather than you making requests that may otherwise not be approved [22:59:41] appreciated! If we've already been approved to have Cloud VPS instances and such, should we be good to go? Or should I point out our purpose further in the Phabricator request? [23:01:02] you should be all good. Just might need to make some different requests for other non standard features (like your own domain - I don't know if that's a thing we actually do) [23:02:47] but don't ask, don't get etc! [23:03:23] Will do! If we were starting out new, we could just use wikipathways.wmcloud.org, but since people have already published papers linking to wikipathways.org, it'd be nice to not make those URLs break :) [23:10:17] ariutta: When you get round to it too... I'd highly suggest getting your mediawiki upgraded to a supported version [23:10:59] Absolutely! That's part of why we're migrating to Cloud VPS -- need to update quite a few things. [23:15:54] keep in mind that MediaWiki 1.36 or later don’t even support upgrading from 1.13, you’ll need an intermediate release first https://w.wiki/4HRD [23:29:22] Good point. I think we'll probably start fresh and port over just the specific data we need.