[02:18:10] 06Machine-Learning-Team: ml-lab should have documentation - https://phabricator.wikimedia.org/T376974 (10calbon) 03NEW [06:42:59] (03PS2) 10Kevin Bazira: Makefile: add support for article-country [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1077747 (https://phabricator.wikimedia.org/T371897) [07:08:46] hello o/ [08:50:34] morning! o/ [08:55:51] 06Machine-Learning-Team: ml-lab can't install rocm torch - https://phabricator.wikimedia.org/T376967#10220698 (10klausman) > Most likely reason is that download.pytorch.org is prohibited Ah, yes, this is intentional on production systems: no outside web access without a proxy in-between. [[ https://wikitech.wik... [08:56:47] Morning! For those of you who want to tinker with the lab machines, the above phab ticket has an important comment regarding outside web access (for e.g. pip install) [09:06:52] hello! [09:07:07] ack [09:12:25] I have a suggestion for the ml-lab machines and the environments. We can do things in a similar way as we do in the production images for pytorch-rocm [09:13:13] we can either install it in the system and then issue the --use-system-packages arg when creating the virtualenv. That way the huge torch-rocm files won't be copied [09:13:28] xAh, good point [09:13:40] I'll give that a whirl in a bit [09:13:54] or if we use a venv (in this case the one in your dir) we could just expand the PYTHONPATH so that would work too [09:15:20] I'm not quite sure yet how best to manage a system-wide install of pytorch [09:16:19] maybe the second option is easier. I'm trying it out now [09:16:31] so shall I go ahead and create a venv in /src/MYUSER ? [09:16:36] */srv/ [09:17:45] yep [09:28:51] ok it works [09:29:28] shall we start with a phab task write stuff over there and then take it and put it into docs? we can use the above task tha chris created [09:30:00] or another one , that one seems too specific [09:30:20] I'm on it [09:30:22] works for me. A goo doc would also work [09:30:35] aa I just saw this one https://phabricator.wikimedia.org/T376974 [09:31:00] I am also trying out if we can make the venv just use symlinks to the humumgous .so and co/hsaco files [09:34:20] Seems like that method can shrink the venv from 16G ot 5G [09:39:44] 06Machine-Learning-Team: ml-lab should have documentation - https://phabricator.wikimedia.org/T376974#10220817 (10isarantopoulos) In order to avoid reinstalling rocm versions of pytorch that take too much space we can follow one of the following approaches: - Install torch+rocm in the system python. Then use `py... [09:44:19] 06Machine-Learning-Team: ml-lab should have documentation - https://phabricator.wikimedia.org/T376974#10220824 (10klausman) In the long term, I'd want this to be a more independent location (like `/opt/pytorch` or something) of course. This would also allow further reducing disk space usage by symlinking redunta... [09:51:18] we could do the same for the torch image in productions images. If we have the correct rocm version that would also help reduce the image by a lot [09:52:30] ack. we didn't pursue that approach at the time so as to reduce the number of variables, but it boils down to a few lines of shellscript. It will need rigorous testing though as some files are not identical [09:54:17] * isaranto nods [09:54:33] it is risky indeed as we may end up having prod issues [10:30:41] 06Machine-Learning-Team: ml-lab can't install rocm torch - https://phabricator.wikimedia.org/T376967#10220899 (10elukey) It is easier :) ` elukey@ml-lab1001:~$ set_proxy Proxy set elukey@ml-lab1001:~$ env | grep HTTP HTTPS_PROXY=http://webproxy:8080 HTTP_PROXY=http://webproxy:8080 elukey@ml-lab1001:~$ unset_... [10:40:15] 10Lift-Wing: Article topic model produces unexpected results for general topic areas such as art - https://phabricator.wikimedia.org/T376987 (10Pginer-WMF) 03NEW [10:45:55] elukey: TIL --^ thanks! [10:46:20] is this for all servers? [10:46:30] IIUC yes [11:06:17] (03PS5) 10Nik Gkountas: Use category search to find campaign pages instead of template [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1076020 (https://phabricator.wikimedia.org/T373132) [11:06:17] (03PS11) 10Nik Gkountas: Fetch campaign metadata and return them with recommendations [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1070308 (https://phabricator.wikimedia.org/T373132) [11:06:17] (03PS1) 10Nik Gkountas: Replace "campaign" term with "collection" or "page_collection" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1079467 [11:06:18] (03PS1) 10Nik Gkountas: Support Default collections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1079468 [11:06:36] (03CR) 10CI reject: [V:04-1] Use category search to find campaign pages instead of template [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1076020 (https://phabricator.wikimedia.org/T373132) (owner: 10Nik Gkountas) [11:06:39] (03CR) 10CI reject: [V:04-1] Fetch campaign metadata and return them with recommendations [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1070308 (https://phabricator.wikimedia.org/T373132) (owner: 10Nik Gkountas) [11:06:42] (03CR) 10CI reject: [V:04-1] Replace "campaign" term with "collection" or "page_collection" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1079467 (owner: 10Nik Gkountas) [11:06:45] (03CR) 10CI reject: [V:04-1] Support Default collections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1079468 (owner: 10Nik Gkountas) [11:07:21] (03PS5) 10Nik Gkountas: Support Default collections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1072175 (https://phabricator.wikimedia.org/T374597) (owner: 10Santhosh) [11:07:34] (03CR) 10CI reject: [V:04-1] Support Default collections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1072175 (https://phabricator.wikimedia.org/T374597) (owner: 10Santhosh) [11:08:07] (03Abandoned) 10Nik Gkountas: Support Default collections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1079468 (owner: 10Nik Gkountas) [11:26:42] (03CR) 10Nik Gkountas: "recheck" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1076020 (https://phabricator.wikimedia.org/T373132) (owner: 10Nik Gkountas) [11:26:48] (03CR) 10Nik Gkountas: "recheck" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1079467 (owner: 10Nik Gkountas) [11:27:46] (03CR) 10Nik Gkountas: "recheck" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1076020 (https://phabricator.wikimedia.org/T373132) (owner: 10Nik Gkountas) [11:37:45] (03CR) 10Nik Gkountas: "recheck" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1070308 (https://phabricator.wikimedia.org/T373132) (owner: 10Nik Gkountas) [11:37:52] (03CR) 10Nik Gkountas: "recheck" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1072175 (https://phabricator.wikimedia.org/T374597) (owner: 10Santhosh) [12:50:06] * isaranto late lunch! [12:58:25] 10Lift-Wing, 10Recommendation-API: Article topic model produces unexpected results for general topic areas such as art - https://phabricator.wikimedia.org/T376987#10221302 (10SBisson) [15:43:34] going afk folks, have a nice weekend! [15:54:02] (03PS6) 10Nik Gkountas: Support Default collections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1072175 (https://phabricator.wikimedia.org/T374597) (owner: 10Santhosh) [15:55:02] (03CR) 10CI reject: [V:04-1] Support Default collections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1072175 (https://phabricator.wikimedia.org/T374597) (owner: 10Santhosh) [15:55:03] \o heading out as well [15:56:46] (03CR) 10Nik Gkountas: "recheck" [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1070308 (https://phabricator.wikimedia.org/T373132) (owner: 10Nik Gkountas) [16:02:40] 06Machine-Learning-Team, 10ORES, 10Moderator-Tools-Team (Kanban), 07Spike: [SPIKE] Investigate how to install ORES in idwiki [8HRS] - https://phabricator.wikimedia.org/T374077#10221916 (10Kgraessle) @Ladsgroup do you know if we need to do anything database related to install ORES extension on idwiki? Also... [16:07:23] (03PS12) 10Nik Gkountas: Fetch campaign metadata and return them with recommendations [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1070308 (https://phabricator.wikimedia.org/T373132) [16:16:55] (03PS7) 10Nik Gkountas: Support Default collections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1072175 (https://phabricator.wikimedia.org/T374597) (owner: 10Santhosh) [16:17:40] (03CR) 10CI reject: [V:04-1] Support Default collections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1072175 (https://phabricator.wikimedia.org/T374597) (owner: 10Santhosh) [16:36:57] 10Lift-Wing, 10Recommendation-API: Article topic model produces unexpected results for general topic areas such as art - https://phabricator.wikimedia.org/T376987#10222124 (10Isaac) This is excellent -- thank you @Pginer-WMF for documenting! Detailed thoughts below about each example you brought up and what we... [16:39:00] (03PS8) 10Nik Gkountas: Support Default collections [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/1072175 (https://phabricator.wikimedia.org/T374597) (owner: 10Santhosh)