[06:59:17] 10Machine-Learning-Team, 10Bad-Words-Detection-System, 10revscoring: Add language support for Thai (th) - https://phabricator.wikimedia.org/T304045 (10PatsagornY) [07:18:49] good morning! [07:19:33] I am trying to run pytests for revscoring, just to test https://github.com/wikimedia/revscoring/pull/517 [07:19:37] but it is a real pain [07:20:40] everything is outdated, and pip on bullseye seems to like only more recent versions of packages [07:21:26] I know that we already done a lot of work to pin dependencies for revscoring in our inference docker images, we could use that work if we want to upgrade revscoring as well [07:22:41] mmm in theory even on our docker images we use the current revscoring lib, that has its own messy requirements.txt [07:26:30] yeah I see that the libenchant version for debian 11 is not the correct one for the pinned version of pyenchant, it is only available on debian 10 [07:26:33] sigh [07:37:59] myc - even if we'll not upgrade revscoring on ORES (and hence move it to python3.7), we should work on upgrading deps in the library to fully support python3.7 and more up-to-date dependencies [07:38:13] testing the new version in our docker images etc.. [07:38:43] I may be wrong but any update to our deps of the lift wing images is a major pain at the moment [07:38:49] and we'll have to support them for a while [07:40:13] need to run some errands, bbl! [09:44:03] back! [09:44:07] so my idea is the following: [09:44:57] 1) we create a python35 (or any name, legacy, etc..) in wikimedia/revscoring, branching from the current master. This is the branch that we'll use for ORES builds [09:45:33] 2) we use the master/main branch of wikimedia/revscoring to upgrade everything to python 3.7 and more up-to-date deps, that we'll deploy in liftwing [09:45:50] I can open a task with some ideas, but please lemme know your thoughts! [09:46:17] in my opinion we need to do basic maintenance to the revscoring lib, it will run on lift wing for quite some time [09:59:40] Sounds like a good plan. I'm not familiar with the build pipeline for ORES, does it allow specifing the branch to build from? [10:01:16] klausman: o/ I think it is a regular pip requirements file, so in case we'll have to specify repo+branch or similar in there [10:01:34] I think at worst, a tag might workl [10:02:27] If we can sortof separate the two targets (ORES as-is vs. a py3.7+ future), that would likely be least disruptive to work on either side. [10:02:39] Of course we don't want to keep that legacy branch around forever [10:04:29] yes definitely, I think that its lifetime will be the ORES's one [10:04:37] so we could backport patches etc.. if needed [10:04:44] or just use it in case of emergency [10:10:05] Yeah. It's probably also easier to switch between the two versions in a given env, e.g. run the main branch on new machines (updated ORES machines or otherwise) and legacy where it is needed. [10:10:33] ideally we could upgrade the os on ores, keeping 3.5, and then moving to 3.7 [10:10:37] if we wanted I mean [10:10:47] but in the meantime we'd be more free to modify the lib for liftwing [10:10:55] Yeah, I think that is most likely to work. [10:11:05] for example, on ORES we have revscoring 2.11.x, and on liftwing 2.9.3 [10:11:17] will prep a task for the team to decide [10:14:16] folks https://phabricator.wikimedia.org/T303928 track Andy's leftovers on stat100x nodes [10:14:30] I think that it is all related to past experiments, so ok to drop [10:14:39] but maybe worth to double check [10:59:48] Went over it quickly (I am not all that familiar with most of the data), but I spotted nothing that seemed save-worthy [11:02:32] ack thanks! [11:46:45] 10Machine-Learning-Team: Revscoring library branching proposal - https://phabricator.wikimedia.org/T304063 (10elukey) [11:47:10] opened the task with the branching proposal --^ [11:47:52] * elukey lunch! [12:04:30] same [14:25:55] *grmbl* with friends like Debian testing, who needs enemies? [14:28:27] elukey: if something instructs you to run `dpkg-fsys-usrunmess`, I _strongly_ suggest you only do it when a) you have some time to fix the breakage it causes and b) have a recovery plan that covers having to reinstall. [14:43:04] I have no idea what it is but I'll keep it in mind :D [14:49:53] Apparently recent installers for Testing have created a mess by symlinking stuff between / and /usr (e.g. sbin). But... dpkg can't handle that [14:50:18] cf. https://manpages.debian.org/bullseye/dpkg/dpkg-fsys-usrunmess.8.en.html [14:52:14] ah snap [14:53:16] I've spot-checked a few of our (WMF) machines, and they all seem fine. My private machines and work laptop, not so much [14:54:44] Morning all! [14:54:56] heyoo chris [15:00:52] o/ [17:57:05] klausman: as FYI https://gerrit.wikimedia.org/r/c/operations/puppet/+/771676, I think it is not a problem with the staging masters if already done, but for the future we'll have nodes without the need to turn off the swap (otherwise the kubelet refuses to start) [18:37:45] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks): Set up the ml-cache clusters - https://phabricator.wikimedia.org/T302232 (10elukey) Plan after a chat with Eric and Filippo - let's start with one instance for each node, and then we can experiment with keyspaces and data access patterns. In case w... [18:42:46] * elukey dinner! o/ [18:50:55] elukey: the "no swap allowed" of kubelets always puzzled me. I wonder what actual engineering defect is the root cause there [18:51:50] We can wipe and re-do the ml-s-ctrl nodes first thing tomorrow, there's nothing save-worthy there yet [18:52:07] s=staging, that is [18:52:22] (I just realized that -s- is not a useful abbrev there :D)