[06:59:32] good morning! [08:16:14] RECOVERY - ORES worker production on ores.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 985 bytes in 1.167 second response time https://wikitech.wikimedia.org/wiki/ORES [08:17:09] \o/ [08:17:21] so this is the alarm that I was talking about earlier on --^ [08:17:41] it was not working fine, it is now fetching from ores.discovery.wmnet (instead of going through the public caches etc..) [08:18:05] it is also publishing alerts on our chan now, previously it was on wikimedia-ai [08:37:54] going afk for a bit! [09:56:40] I think that we should create something like https://wikitech.wikimedia.org/wiki/Data_Engineering/Team/Onboarding [10:01:19] 10Lift-Wing, 10artificial-intelligence, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Revert editquality isvc architecture to predictor-only - https://phabricator.wikimedia.org/T301412 (10elukey) ` elukey@deneb:~$ docker-registryctl delete-tags docker-registry.wikime... [11:19:42] klausman: o/ [11:20:19] we have the ml-serve staging nodes racked https://phabricator.wikimedia.org/T294946 [11:20:36] they need to be reimaged as bullseye, and then we'd need to create another dedicated cluster [11:20:50] (etcd, control plane, puppet horror, etc..) [11:21:22] if you are interested it could be a good task to see all the steps to deploy/configure something from scratch [11:36:15] hey guys! It is Aiko here. So happy today is my first day joining the ML team \o/ I'm setting up my wikimedia account and other communication platforms. I'm wondering do we have channels on slack? [11:38:21] aiko: hellooooooo [11:38:25] welcome! [11:38:33] We do have one [11:38:37] lemme invite you [11:41:06] aiko: I see two users on slack, I pinged you on (what I think is) the new one, lemme know if it is the right one [11:41:19] we may want to delete the other to avoid confusion [11:50:15] yes! that's me [11:50:27] added :) [11:52:20] miriam: o/ do you recall how Aiko's slack account was added? Since there are now two I think we could ask to remove the old one [11:52:42] aiko ooooooooooooo [11:53:29] elukey: yes, I asked someone from IT, let me check [11:55:55] elukey: I think you can write an email to techsupport@ or directly ask Nina Bertoni [11:56:20] miriam: thanks :) [11:56:38] aiko: when you have a moment, can you send an email to techsupport@ to ask for the removal of the old slack account? [11:59:17] ok! I'll send Nina a message to remove the old one [12:03:54] thanks :) [12:06:29] aiko: we still don't have an onboarding page, I'll try to create one similar to https://wikitech.wikimedia.org/wiki/Data_Engineering/Team/Onboarding later on [12:06:50] in the meantime, please take it super easy, no rush in doing anything this week :) [12:07:40] also I just remember that you have already a lot of things set up [12:07:46] ssh account, etc.. [12:07:54] those will need to be updated/converted probably [12:07:59] we'll figure it out :) [12:17:33] elukey: Already racked? excellent. The setup (etcd and so on) should be (relatively) easy. [12:17:52] (... if I remember where I put my nodes from last time :-S) [12:21:40] klausman: if you have time to do it, feel free to open a task will all the infos etc.. (haven't done it yet) [12:22:02] aye [12:22:29] I presume we do the same VM setup for the control plane? [12:23:43] 10Machine-Learning-Team, 10ORES, 10Discovery-Search, 10Growth-Team (Current Sprint): Include Argentina (and any other individual countries that are ready to include) in ORES topic modeling - https://phabricator.wikimedia.org/T301671 (10kostajh) [12:30:01] 10Machine-Learning-Team, 10ORES, 10Discovery-Search, 10Growth-Team (Current Sprint): Include Argentina (and any other individual countries that are ready to include) in ORES topic modeling - https://phabricator.wikimedia.org/T301671 (10kostajh) p:05Triage→03High From chatting with @gehel it sounds like... [12:33:41] klausman: I think so yes, you can check the kubestage* nodes and see what sre did [12:33:53] should be the same tiny specs, nothing really different [12:34:02] Roger [12:39:13] 10Machine-Learning-Team, 10ORES, 10Discovery-Search, 10Growth-Team (Current Sprint): Include Argentina (and any other individual countries that are ready to include) in ORES topic modeling - https://phabricator.wikimedia.org/T301671 (10kostajh) @isaac as I understand your comment in T301030#7690294, your w... [12:40:01] 10Machine-Learning-Team, 10ORES, 10Discovery-Search, 10Growth-Team (Current Sprint): Include Argentina (and any other individual countries that are ready to include) in ORES topic modeling - https://phabricator.wikimedia.org/T301671 (10kostajh) [12:48:57] going out for lunch [15:04:09] elukey: o/ one thing is I'll receive Foundation's laptop late, probably at the end of March, according to Apple's order page, since there have been major shipping delays :( so currently I will set up everything on my personal laptop. Hope it's not a big problem [15:06:21] aiko: not at all if you don't mind! [15:06:44] be careful with the ssh settings, protect your key with password etc.. [15:06:46] usual things :) [15:08:28] aiko: do you have a wikitech account? [15:08:57] I am wondering what's best since you are transitioning to full employee status, it may be easy to just convert what we have [15:09:52] yes you already have everything [15:10:27] elukey: yes I have a wikitech account. AikoChou [15:10:40] I'll ask to other SREs what's best and fix what's needed, but I think it should be a matter of changing your email in the various puppet/ldap/etc.. configs and nothing more [15:10:52] (in case I'll take care of it) [15:12:29] Ok!! thanks :D [15:12:53] aiko: just added you to the few meetings that we do [15:13:09] there is one this evening, if you are busy and you can't make it don't worry [15:13:24] if you can join we'll say hello and chat all together :) [15:40:17] created https://phabricator.wikimedia.org/T301681 [15:40:35] elukey: I can join this evening :) [15:40:39] nice! [15:40:53] I created the above task for your accounts, I am waiting for the green light from SRE [15:46:05] yep seems the right thing to do, going to do it in a bit [15:49:28] elukey: does it only change my LDAP username? Will the Wikimedia email also be changed? [15:50:43] aiko: nono only the email info the ldap user metadata [15:50:55] I just changed your email with the WMF one [15:50:59] no change from your side [15:51:43] Morning Aiko! [15:53:14] Good morning Chris 😃 [15:56:16] Elukey will by your onboarding guide for the technical things, I will help you be onboarded for everything else. [15:57:00] aiko: I am almost done in modifying your accounts, from your side nothing will change, but you'll be in the right groups [15:57:37] chrisalbon: just for paperwork, can you review/approve https://phabricator.wikimedia.org/T301681 ? [15:59:30] (need to get some groceries, bbiab :) [15:59:34] chrisalbon: got it! [16:03:26] aiko I put some time in your calendar this week where we can talk about the team and its role, but for now just keep working on getting yourself set up. No hurry, take your time. [16:09:01] o/ [16:10:27] chrisalbon: alright thank you that's nice! [16:24:24] (03CR) 10Accraze: [C: 03+2] editquality: remove transformer blubberfile [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/761788 (https://phabricator.wikimedia.org/T301412) (owner: 10Kevin Bazira) [16:24:57] elukey: sorry for being a bit tardy on the data.yaml review [16:27:25] (03CR) 10Accraze: [V: 03+2 C: 03+2] editquality: remove transformer blubberfile [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/761788 (https://phabricator.wikimedia.org/T301412) (owner: 10Kevin Bazira) [16:33:12] 10Lift-Wing, 10artificial-intelligence, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Revert editquality isvc architecture to predictor-only - https://phabricator.wikimedia.org/T301412 (10ACraze) [16:41:09] 10Lift-Wing, 10artificial-intelligence, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks): Upload model binaries to storage - https://phabricator.wikimedia.org/T301413 (10ACraze) a:03ACraze [16:41:19] 10Lift-Wing, 10artificial-intelligence, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks): Upload model binaries to storage - https://phabricator.wikimedia.org/T301413 (10ACraze) 05Open→03In progress [16:41:21] 10Lift-Wing, 10artificial-intelligence, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks): Migrate editquality models - https://phabricator.wikimedia.org/T301409 (10ACraze) [16:41:23] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10Halfak) I understand your hesitation to mini... [17:07:34] klausman: np! There is really no rush, it can be reviewed/merged anytime (I had to go to get groceries and got lost in traffic sigh) [17:09:39] merged the change, I think that Aiko's accounts are good now [17:10:40] I was distracted today since I found out one of my old Dire Straits LPs is worth $$$ and I was trying to figure out the specifics :) [17:12:46] 10Machine-Learning-Team, 10ORES, 10Discovery-Search, 10Growth-Team (Current Sprint): Include Argentina (and any other individual countries that are ready to include) in ORES topic modeling - https://phabricator.wikimedia.org/T301671 (10EBernhardson) We talked about this a bit within search platform today.... [17:49:06] 10Lift-Wing, 10artificial-intelligence, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks): Upload model binaries to storage - https://phabricator.wikimedia.org/T301413 (10ACraze) I just uploaded all of the editquality model files to storage on stat1008 (using this script: P20723). Here are ne... [18:08:41] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks): Lift Wing proof of concept - https://phabricator.wikimedia.org/T272917 (10elukey) [18:08:43] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks): Improve ml-serve's Istio logs - https://phabricator.wikimedia.org/T300707 (10elukey) 05Open→03Resolved [18:14:56] 10Lift-Wing, 10artificial-intelligence, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Revert editquality isvc architecture to predictor-only - https://phabricator.wikimedia.org/T301412 (10calbon) 05Open→03Resolved [18:14:58] 10Lift-Wing, 10artificial-intelligence, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks): Migrate editquality models - https://phabricator.wikimedia.org/T301409 (10calbon) [18:17:07] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks): Lift Wing proof of concept - https://phabricator.wikimedia.org/T272917 (10calbon) [18:17:22] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks): Factor out feature retrieve functionality to a transformer - https://phabricator.wikimedia.org/T294419 (10calbon) 05Open→03Resolved a:03calbon [18:43:22] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10elukey) >>! In T300195#7708077, @Halfak wrot... [18:44:53] * elukey afk! [19:05:13] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10Halfak) > I think that there were little err... [19:22:02] ughh there was a git lfs issue when i uploaded all the editquality model files to storage, need to do it again :( [19:22:53] can't wait to not juggle 1GB+ data inside a git repo lol [19:30:29] 10Machine-Learning-Team, 10ORES, 10Discovery-Search, 10Growth-Team (Current Sprint): Include Argentina (and any other individual countries that are ready to include) in ORES topic modeling - https://phabricator.wikimedia.org/T301671 (10Isaac) I like what @EBernhardson proposed and specifically agree that `... [19:35:32] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10elukey) >>! In T300195#7708898, @Halfak wrot... [19:36:36] 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10Halfak) I really am trying to be constructiv... [19:50:50] ok 2nd upload for editquality model binaries is complete and things look good. the list of newest storage uris is here: https://phabricator.wikimedia.org/T301413#7708487 [20:04:03] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks): Factor out feature retrieve functionality to a transformer - https://phabricator.wikimedia.org/T294419 (10ACraze) [20:04:09] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Topic model transformer - https://phabricator.wikimedia.org/T298990 (10ACraze) 05Open→03Resolved [20:04:37] 10Lift-Wing, 10artificial-intelligence, 10editquality-modeling, 10Epic, 10Machine-Learning-Team (Active Tasks): Migrate editquality models - https://phabricator.wikimedia.org/T301409 (10ACraze) 05Open→03In progress [20:08:30] ok next, im going to add the first batch of editquality isvc pod configs, going to follow elukey's lead and do 3-4 isvcs at a time [20:14:51] ah actually first we need to update the editquality model server to use the recent "predictor-only" version in the helmfile [20:15:49] 10Lift-Wing, 10artificial-intelligence, 10editquality-modeling, 10Machine-Learning-Team (Active Tasks): Add editquality isvc configurations to ml-services helmfile - https://phabricator.wikimedia.org/T301415 (10ACraze) 05Open→03In progress [20:15:51] 10Lift-Wing, 10artificial-intelligence, 10editquality-modeling, 10Epic, 10Machine-Learning-Team (Active Tasks): Migrate editquality models - https://phabricator.wikimedia.org/T301409 (10ACraze) [20:51:04] 10Machine-Learning-Team, 10ORES, 10Discovery-Search, 10Growth-Team (Current Sprint): Include Argentina (and any other individual countries that are ready to include) in ORES topic modeling - https://phabricator.wikimedia.org/T301671 (10kostajh) p:05High→03Triage (Changing the priority since this task h... [20:53:15] 10Machine-Learning-Team, 10ORES, 10Discovery-Search, 10Growth-Team (Current Sprint): Investigate what would be required to include countries in ORES articletopic modeling - https://phabricator.wikimedia.org/T301671 (10kostajh)