[07:54:09] good morning! [07:54:11] nice accraze :) [07:54:17] I am trying to deploy ores in beta and I get [07:54:18] Installing setuptools, pkg_resources, pip, wheel...done. [07:54:19] gensim-3.8.3-cp37-cp37m-manylinux1_x86_64.whl is not a supported wheel on this platform. [07:54:22] lovely [07:56:27] that kinda makes sense, cp37 should by python 3.7 and it is not supported on stretch (we use 3.5) [08:01:09] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10elukey) Tried to deploy in Beta (deployment-... [08:04:08] (03CR) 10Elukey: nlwiki articlequality, hiwiki editquality, ores observability (031 comment) [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/755731 (https://phabricator.wikimedia.org/T300195) (owner: 10Halfak) [08:05:49] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10elukey) @Halfak there is a problem that I ju... [08:06:06] the last patch seems to also revert mwparserfromhell to its previous version, that caused the celery segfaults.. [08:11:59] last commit in the wheels repo.. [08:12:00] "Updated wheels for revscoring-2.11 and python 3.7" [08:12:02] .... [08:12:34] yeah https://gerrit.wikimedia.org/r/c/research/ores/wheels/+/748390 [08:13:36] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10elukey) Yes the dependencies got bumped to P... [08:17:10] so I think that tomorrow's deployment is a no-go :D [11:41:24] * elukey lynch [11:41:31] * elukey err lunch [11:49:58] Was worried there for a moment :) [13:36:16] ml-serve2005 up and running with overlay fs! [13:48:00] 10Machine-Learning-Team, 10DC-Ops, 10SRE, 10ops-eqiad, 10Patch-For-Review: Q2:(Need By: TBD) rack/setup/install ml-serve100[5-8] - https://phabricator.wikimedia.org/T294949 (10elukey) Already set the partman recipe (we are going to use a new one). [13:53:21] 10Machine-Learning-Team, 10DC-Ops, 10SRE, 10ops-eqiad, 10Patch-For-Review: Q1:(Need By: TBD) rack/setup/install dse-k8s-worker100[1-4] - https://phabricator.wikimedia.org/T291579 (10elukey) I've set the partman recipe (we are using a newer one for these nodes). [13:54:05] 10Machine-Learning-Team, 10DC-Ops, 10SRE, 10ops-eqiad, 10Patch-For-Review: Q1:(Need By: TBD) rack/setup/install dse-k8s-worker100[1-4] - https://phabricator.wikimedia.org/T291579 (10elukey) [13:57:07] dse-k8s-worker nodes (train wing) in eqiad! https://phabricator.wikimedia.org/T291579 [13:57:22] I see that the ml-cache nodes have tasks as well [14:19:53] Veyr nice [14:20:02] Does this mean they have been delivered already? [14:34:23] I think delivered but not racked yet [15:09:02] Still quicker than I expected [15:09:18] But then again, time has been flying in some respects recently [15:10:23] Morning! [15:10:31] Heyo Chris. [15:10:40] no deploy tomorrow? [15:11:20] ah I see it [15:18:18] chrisalbon: morning! I am personally -1 with my current understanding of what is being deployed, but I may have missed something [15:18:35] Yeah it makes sense, good catch [15:22:36] https://www.applyconf.com/apply-meetup-february-2022/ is scheduled for thursday [15:23:00] it is a little late for EU folks, but it contains a lot of interesting talks [15:23:09] I think/hope they will be added to youtube [15:38:11] o/ [15:46:40] o/ [15:49:41] just catching up on the deploy discussion, i agree on holding off [15:54:41] i know the articlequality models won't run on python3.7 [15:55:52] accraze: I was very puzzled this morning, I didn't get the bump to python 3.7 [15:56:28] yeah im not sure why it happened either, nice catch elukey [15:56:29] also mwparserfromhell is downgraded [15:56:42] that gets us back to the segfault area sigh [15:56:46] (for celery) [15:57:01] yeah that's a no-go [15:57:39] it might just be an error, the wheels repo geos out of sync pretty easily in the past [15:58:18] (the downgrade that is) [15:58:18] https://gerrit.wikimedia.org/r/c/research/ores/wheels/+/748390 this is the original code change [15:58:51] that downgrades also mwparserfromhell [16:00:02] this could be related to the pip version ranges across all the various ores repos, i ran into old versions of stuff getting installed due to pip backtracking [16:01:45] sure but Aaron's intent seemed to bump everything to python 3.7 anyway, not sure if there was the idea to move to Buster [16:01:57] ohhh i see [16:02:15] the wheels that need cpython are all py37 [16:02:22] they cannot run in a python35 venv [16:02:28] pip refuses to install them [16:02:55] hmmmm [16:13:27] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10Halfak) Oh! The move the 3.7 is kind of old... [16:17:00] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10elukey) @Halfak my understanding is that mov... [16:53:34] so ORES deployments are currently blocked, the problem with python wheels seems big [16:59:35] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10Halfak) It shouldn't cause problems for unpi... [17:31:21] * elukey afk! [17:32:07] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10elukey) >>! In T300195#7693843, @Halfak wrot... [18:17:21] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10Halfak) It looks like the mwparserfromhell c... [18:18:31] (03PS1) 10Halfak: Reverts back to python 3.5 wheels. [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/761015 (https://phabricator.wikimedia.org/T300195) [18:20:22] (03PS2) 10Halfak: Reverts back to python 3.5 wheels and includes mwparserfromhell 0.6.3 [research/ores/wheels] - 10https://gerrit.wikimedia.org/r/761015 (https://phabricator.wikimedia.org/T300195) [18:20:28] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10Halfak) OK I think that patchset is good for... [23:22:25] awesome, just got a lime explainer attached to enwiki-goodfaith isvc on ml-sandbox, its hacky but does alright job at giving feature value stores [23:23:35] * feature importances scores [23:24:18] https://www.irccloud.com/pastebin/Cj0KG8Ji/enwiki-goodfaith-lime-explainer [23:26:50] ok going afk - will make a WIP explainer CR tomorrow