[06:31:07] ottomata: nice! [06:34:16] good morning folks [06:34:40] I've let Benthos run over night, and there was a problem around 4:30 UTC with Kafka, a little weird [06:34:45] also other conns breakages [06:34:48] but way less than before [06:41:01] aiko: o/ I have also found this today [06:41:02] model-server/model.py:46: DeprecationWarning: The object should be created within an async function self._http_client_session = aiohttp.ClientSession() [06:41:10] sigh didn't notice it before [07:52:06] morning! :) [07:57:38] elukey: yeah I also noticed that yesterday. I'm gonna send a patch for that in a min and pls let me know what you think [08:00:20] thanks! [08:00:32] I was wondering what was best, but if you have a solution ready +1 [08:06:54] (03PS1) 10AikoChou: editquality: move asyncio session creation out of __init__ [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849016 [08:08:50] mmm aiko does the above removes the warning? [08:09:05] because in theory the session is not created inside an async session [08:09:09] err function [08:09:19] maybe the @property is different [08:09:24] yeah I don't see the warning [08:10:11] I think @property is different [08:10:11] weird, and you tested the model-server with multiple requests etc.. ? [08:11:13] yep I tested [08:11:26] [I 221025 07:45:55 model:61] Opening a new Asyncio session. [08:11:26] [I 221025 07:45:56 web:2243] 200 POST /v1/models/enwiki-goodfaith:predict (172.17.0.1) 1617.56ms [08:11:27] [I 221025 07:46:03 web:2243] 200 POST /v1/models/enwiki-goodfaith:predict (172.17.0.1) 1310.73ms [08:11:27] [I 221025 07:46:08 web:2243] 200 POST /v1/models/enwiki-goodfaith:predict (172.17.0.1) 1439.81ms [08:11:27] [I 221025 07:46:20 model:67] Closing asyncio session [08:11:43] super [08:13:48] (03CR) 10Elukey: [C: 03+1] "Left some nits on the commit msg, feel free to proceed after those!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849016 (owner: 10AikoChou) [08:15:47] It'll create an asyncio session when receiving the first request, not during model server initialization [08:17:57] okay I'll modify the commit msg :) [08:19:14] yes yes the change makes perfect sense, I was puzzled this morning since I thought we needed to have an async @property etc.. (that would have been weird) [08:19:37] aiko: do you have time to fix the other model servers too? Otherwise I'll do it [08:22:47] * elukey brb [08:28:54] elukey: I can do it, but is it urgent? because I want to do other stuff first [08:37:01] aiko: nono on the contrary [08:37:15] aiko: thinking about it, I can do it together with the process pool things [08:48:03] (03PS2) 10AikoChou: editquality: move aiohttp session creation out of __init__ [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849016 (https://phabricator.wikimedia.org/T320374) [08:49:31] (03CR) 10Elukey: editquality: move aiohttp session creation out of __init__ [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849016 (https://phabricator.wikimedia.org/T320374) (owner: 10AikoChou) [08:50:40] Morning! [08:50:50] morning! [08:50:58] elukey: https://www.space.com/partial-solar-eclipse-tomorrow-europe-africa-asia this is today (in ~30m), don't miss it =) [08:51:33] Timing for Rome is start at 11:26, Max at 12:22, End at 13:19 [08:52:22] For me it's all ~10m earlier, so I suspect Bologna is somewhere in the middle between the two [08:54:49] ah didn't know it thanks [08:55:09] I don't have proper gear though to watch it, so I'll skip the partial one [08:55:44] piece of cardboard and a pinhole in it is enough :) [08:56:41] yeah but I am not that fond of low tech experiments [08:57:06] And yet you work with k8s! [08:58:01] :) [08:59:51] hello! any updates on the wikilabels databases? [09:00:04] Working on it today [09:00:59] (03PS3) 10AikoChou: editquality: move aiohttp session creation out of __init__ [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849016 (https://phabricator.wikimedia.org/T320374) [09:03:17] (03CR) 10AikoChou: [C: 03+2] editquality: move aiohttp session creation out of __init__ [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849016 (https://phabricator.wikimedia.org/T320374) (owner: 10AikoChou) [09:04:27] (03Merged) 10jenkins-bot: editquality: move aiohttp session creation out of __init__ [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849016 (https://phabricator.wikimedia.org/T320374) (owner: 10AikoChou) [09:05:27] (03CR) 10AikoChou: [C: 03+2] editquality: move aiohttp session creation out of __init__ (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849016 (https://phabricator.wikimedia.org/T320374) (owner: 10AikoChou) [09:14:41] taavi: o/ we'd like to create a single VM with postgres on the wikilabels project, and dump the pg_database from clouddb-wikilabels-01/02 [09:14:59] do we have perms to do that? Otherwise we'll ask your assistance if you have time [09:15:16] (I don't know what puppet role would be best too for the postgres instance, never created one) [09:16:51] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Support pre-transformed inputs for Outlink topic model - https://phabricator.wikimedia.org/T315998 (10achou) The code change for this task are complete. Going to mark this as RESOLVED. I'll open another task when we're going to test lar... [09:17:23] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Support pre-transformed inputs for Outlink topic model - https://phabricator.wikimedia.org/T315998 (10achou) [09:17:41] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Support pre-transformed inputs for Outlink topic model - https://phabricator.wikimedia.org/T315998 (10achou) 05Open→03Resolved [09:23:28] elukey: I think you should have. I'd imagine pg_dump just needs read access to the database itself, which you have [09:25:26] the current instances use role::wmcs::db::wikilabels and role::wmcs::db::wikilabels_secondary [09:26:10] taavi: ack nice. Do you mind to give me the fqdn of the vms? I don't find the project in horizon [09:26:36] elukey: I made a VM already. The data on our instance is O(100M), so I don't think we need anything big [09:27:03] the old vms? [09:27:08] elukey: I am already connected to the DB using psql, pg_dump should work fine from the 03 instance you made [09:27:45] klausman: ack super [09:27:45] `psql -U u_wikilabels -h wikilabels-database u_wikilabels` from the 03 instance, plus the password from the service config in /srv [09:28:16] I was wondering if we need to check settings on the clouddb instances as well [09:28:32] but if everything is in the puppet role we should be ok [09:29:34] The whole dump is 99M. The only bits not in there are of course the u/p config, but that can be recreated easily [09:30:18] wikilabels-database-01.wikilabels.eqiad1.wikimedia.cloud is the new instance for the DB [09:31:36] Ok, the roles I can add, but the hiera config is empty atm [09:32:41] it is probably stored in horizon [09:34:45] I don't have access to the old DB's hierdata, I suspect. [09:35:00] taavi: can you make a dump of that config somewhere in a safe place? [09:35:35] ohwait, there is also gerrit [09:37:13] Oh, I don't need it. [09:37:44] elukey: the old setup was a replicated one (hence one instance using secondary), I blindly applied that and of course it needs some pointer towards the primary. [09:38:28] I'll set up one with puppet (and without the secondary role) and see if I can get that to work [09:40:04] klausman: can we change the puppet role/profile to use only one vm? [09:40:18] yeah, pretty sure [09:40:29] I don't think much is needed, if anything [09:40:51] I see profile::wmcs::services::postgres::primary [09:40:57] that in theory should be able to run standalone [09:41:03] yep [09:46:45] sigh I see the error "upstream_reset_before_response_started{connection_termination}" even for eventgate [09:47:11] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Connect Outlink topic model to eventgate - https://phabricator.wikimedia.org/T315994 (10achou) Current status: - code changes for the model server are done - created a Benthos [[ https://phabricator.wikimedia.org/P36242 | confi... [09:52:14] klausman: just to confirm: did you need anything from me? [09:52:45] not at the moment, I think, thanks! [10:03:11] created https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/849040/ to apply the same settings to eventgate [10:03:29] going afk for lunch, ttl! [10:09:42] elukey: so the problem is that the VM has 8G of RAM but the default Postgres config asks for 7G of shmem, which the machine can't satisfy. This is configurable in psql, but not in the puppet role we're using. [10:10:59] Unfortunately, the role we use is not exclusive to WL, so we can't just change the hardcoded value [10:11:01] klausman: but we can create a VM with more ram no? [10:11:13] which role are you using? [10:11:15] 8G is the biggest our quota allows [10:11:29] profile::wmcs::services::postgres::primary [10:11:49] role::wmcs::db::wikilabels just inherits from that [10:12:04] (plus `system::role { $name: }`) [10:13:13] you can probably either make a new wikilabels specific role, or modify that role to take a hiera param for the amount of ram to use [10:13:15] elukey: and I'd argue that an 8G instance is already way bigger than what WL needs. [10:13:33] Yeah, that was my plan, similar to what other psql roles already do [10:13:43] klausman: sure, as taavi suggested we can have special hiera param for our use case [10:13:57] I'll get to work on that after lunch [10:14:15] I'd probably slightly prefer you having a separate role that's not named 'wmcs' [10:17:23] I'll see what I can do ;) [12:22:05] +1 to the change in naming, it makes sense [12:27:37] (03PS1) 10Elukey: articlequality: add process pool and improve aiohttp's client session [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849062 (https://phabricator.wikimedia.org/T320374) [12:29:32] (03PS1) 10Elukey: draftquality: add process pool and improve aiohttp's client session [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849064 (https://phabricator.wikimedia.org/T320374) [12:31:23] (03PS1) 10Elukey: topic: add process pool and improve aiohttp's client session [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849086 (https://phabricator.wikimedia.org/T320374) [13:06:06] (03PS2) 10Elukey: articlequality: add process pool and improve aiohttp's client session [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849062 (https://phabricator.wikimedia.org/T320374) [13:06:08] (03PS2) 10Elukey: draftquality: add process pool and improve aiohttp's client session [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849064 (https://phabricator.wikimedia.org/T320374) [13:06:10] (03PS2) 10Elukey: topic: add process pool and improve aiohttp's client session [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849086 (https://phabricator.wikimedia.org/T320374) [13:06:14] all tested! [13:06:57] going to switch ml-etcd1001 to DRBD for ca. 1-2 hrs, as such latencies might go up for a bit [13:07:50] ack! [13:22:49] elukey: so I had to remake the db vm, but the DNS name is still pointing to the old IP [13:22:59] Anything I need to bump to update it? [13:23:36] I can SSH to the new IP just fine (via the bastion), but obviously, changing puppet stuff etc won't work right until the name is correct [13:25:56] klausman: a bit more details :) Did you kill the instance and re-created with the same hostname, but now the A record points to the old IP? [13:26:06] exactly [13:26:19] weird.. maybe the TTL needs to expire? [13:26:26] it's been hours [13:26:26] I don't recall what it is set to [13:26:57] sure, but what is the TTL for the record? Did you check? [13:26:59] The new machine is >3,.5h old [13:27:21] 60s [13:28:54] weird.. [13:29:10] can you give me the hostname? [13:29:12] so I can check [13:29:17] wikilabels-database-01.wikilabels.eqiad1.wikimedia.cloud [13:29:47] Old IP is 172.16.3.81, new is 172.16.6.103 [13:31:12] it is probably something related to openstack, we may need to ask to the cloud folks [13:31:26] or, the quick thing could be to create a new VM called -02 [13:31:31] and delete the other one :D [13:31:40] (if you haven't already set up everything) [13:32:00] yeah, I wanted to switch to bullseye anyway [13:32:46] openstack should have its own auth dns servers, I guess that the control plane has some bugs in updating same-name VMs [13:32:51] never happened to me though [13:33:52] Does pcc work the same with cloud VMs? [13:38:07] In theory it should, but you can't mix prod with cloud (so hosts from both realms in the same run) [13:38:17] That's fine [13:39:36] (03CR) 10AikoChou: [C: 03+1] articlequality: add process pool and improve aiohttp's client session [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849062 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [13:39:59] (03CR) 10AikoChou: [C: 03+1] draftquality: add process pool and improve aiohttp's client session [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849064 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [13:40:18] (03CR) 10AikoChou: [C: 03+1] topic: add process pool and improve aiohttp's client session [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849086 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [13:40:29] thanks aiko :) [13:40:49] (03CR) 10Elukey: [C: 03+2] articlequality: add process pool and improve aiohttp's client session [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849062 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [13:40:55] (03CR) 10Elukey: [C: 03+2] draftquality: add process pool and improve aiohttp's client session [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849064 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [13:41:00] (03CR) 10Elukey: [C: 03+2] topic: add process pool and improve aiohttp's client session [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849086 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [13:41:12] elukey: you can mix them these days [13:41:23] taavi: ah nice! TIL [13:41:46] I guess I have to thank you and John :) [13:42:02] (03Merged) 10jenkins-bot: articlequality: add process pool and improve aiohttp's client session [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849062 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [13:42:30] dcaro and John I believe, I wasn't involved [13:45:31] (03Merged) 10jenkins-bot: draftquality: add process pool and improve aiohttp's client session [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849064 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [13:45:33] (03Merged) 10jenkins-bot: topic: add process pool and improve aiohttp's client session [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/849086 (https://phabricator.wikimedia.org/T320374) (owner: 10Elukey) [13:45:53] taavi: well I can always thank you for all the work that you do daily (like Terraform!) [13:46:07] :-) [13:47:38] Hrm. Now the damn thing won't let me log in [13:47:51] when did you create it? [13:48:12] Created [13:48:14] Oct. 25, 2022, 1:33 p.m. [13:48:16] Age [13:48:18] 12 minutesy [13:48:21] where the Y comes from, I dunno [13:48:44] (it's not in the webpage, but happened during pasting) [13:49:14] sigh. of course horizon has killed my devices [13:49:22] * taavi goes to fetch their 2fa device [13:49:43] debug1: Offering public key: tklausmann@wikimedia.org-cloud ED25519 SHA256:bla [13:49:44] debug1: Server accepts key: tklausmann@wikimedia.org-cloud ED25519 SHA256:bla [13:49:46] Connection closed by 172.16.3.151 port 22 [13:49:55] sooo, dunno what that means. [13:50:06] I don't recall if puppet needs to run one time before you can log in [13:50:21] or if it is ready from the get-go [13:50:34] yeah, the first puppet run needs to complete [13:51:00] klausman: you can check the logs panel of the instance, it should show if puppet ran or not [13:51:01] it also needs to upgrade all packages that have had updates after the image was created, which takes a while at the moment [13:51:20] The logs tab only rrally has cron messages [13:51:31] if the log panel shows a root shell session, it's probably complete, otherwise probably not [13:51:51] yep, root shell visible, and then juts agent stat collection etc [13:52:15] The weird thing is that it accepts my key and then just drops the SSH session [13:52:24] root@wikilabels-database-02:~# run-puppet-agent [13:52:24] Error: Could not request certificate: The certificate retrieved from the master does not match the agent's private key. Did you forget to run as root? [13:52:46] aiko: did you see https://docs.python.org/3.11/whatsnew/3.11.html ? :) [13:53:04] it's failing to run puppet for whatever reason, and because of that it can't configure the authentication stack properly [13:53:10] taavi: hrm, do I need to wipe the cert on puppet and issue a new one? [13:53:21] that should happen automatically [13:53:48] It's also weird that it already has a cert for a machine that I just created. [13:53:57] did you re-use the instance name? [13:54:10] No, I switched to 02 [13:54:26] weird [13:54:27] But I did try a "rebuild" in Horizon, so maybe that was it :-/ [13:54:57] I thought we disabled that option, iirc it currently does absolutely nothing [13:55:19] So what's my best course of action now? [13:55:30] give me a sec, I'm fixing the intsance [13:55:35] thank you! [13:56:22] try now [13:56:36] and it works! tyvm! [14:00:54] elukey: not yet, what's new? [14:01:57] aiko: there are some nice things, still checking it as well [14:02:15] aiko: like https://docs.python.org/3.11/whatsnew/3.11.html#pep-657-fine-grained-error-locations-in-tracebacks [14:26:49] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Decide external URL scheme (on API GW) for models on Lift Wing - https://phabricator.wikimedia.org/T319178 (10calbon) [14:35:23] 10Machine-Learning-Team: Spike. Try to ML models distributted in jupyter notebooks with dask - https://phabricator.wikimedia.org/T243089 (10calbon) 05Open→03Declined [14:38:48] 10ORES, 10Documentation, 10Machine-Learning-Team (Active Tasks), 10User-AKlapper: Update docs that ORES will be replaced by Lift Wing - https://phabricator.wikimedia.org/T305963 (10calbon) [14:42:47] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Decide external URL scheme (on API GW) for models on Lift Wing - https://phabricator.wikimedia.org/T319178 (10klausman) a:03klausman [14:44:57] 10Machine-Learning-Team, 10ORES, 10artificial-intelligence, 10editquality-modeling: Generate dump of scored-revisions from 2018-2020 for Wikis except English Wikipedia - https://phabricator.wikimedia.org/T280107 (10calbon) Just checking in, if this is still needed. Otherwise I will close ticket [14:45:50] 10Lift-Wing, 10Machine-Learning-Team, 10ORES, 10artificial-intelligence, 10Documentation: Model Reporting - https://phabricator.wikimedia.org/T276397 (10calbon) 05Open→03Declined [14:47:27] 10Machine-Learning-Team, 10artificial-intelligence, 10Wikilabels, 10editquality-modeling: Edit quality campaign for Hypixel SkyBlock Wiki - https://phabricator.wikimedia.org/T274697 (10calbon) 05Open→03Declined [14:47:30] 10Machine-Learning-Team, 10artificial-intelligence, 10Wikilabels, 10editquality-modeling: Edit quality campaign for Hypixel SkyBlock Wiki - https://phabricator.wikimedia.org/T274697 (10calbon) Declined due to not related to mediawiki/wikipedia [14:52:00] 10Machine-Learning-Team: Write Lift Wing Wiki Page - https://phabricator.wikimedia.org/T302897 (10calbon) [15:02:09] elukey: so re: wikilabels. I can run puppet-agent on the machine just fine, but running pcc (with my change and limited to the db machine) gives me: [ 2022-10-25T14:46:35 ] CRITICAL: Unexpected error running run_host on wikilabels-database-02.wikilabels.eqiad1.wikimedia.cloud: Unable to find fact file for: wikilabels-database-02.wikilabels.eqiad1.wikimedia.cloud under directory [15:02:11] /var/lib/catalog-differ/puppet [15:02:31] So I wondered if this is because the machine currently has 0 roles assigned in Horizon. [15:03:07] Using the newly-created role doesn't work, but of course that may be a bug in my change [15:03:28] Or am I using pcc wrong for a cloud vm? [15:04:23] yes with no roles assigned I don't think that you can get facts (they should be automatically synced nowadays) [15:04:55] you can apply the role and run puppet on the host directly, then solve the errors from there [15:05:05] not ideal but it shouldn't be a big issue [15:05:11] do you have a new role ? [15:05:36] weeell. If I apply the role in Horizon and run r-p-a, I of course get errors since the change isn't submitted [15:06:04] Yes, profile::wikilabels::services::postgres::single [15:06:18] https://gerrit.wikimedia.org/r/c/operations/puppet/+/849095 is the WIP change [15:06:33] (likely full of errors) [15:07:26] The files are mostly just copies of the wmcs role [15:07:42] the pcc node sync is happening on a nightly cron [15:08:03] aaah, so that likely hasn't happened yet [15:08:12] also pcc won't display anything unless you apply the role via horizon [15:08:29] taavi: but does it happen even without any role etc..? I guess there is a base role that installs things, like we were discussing before right [15:08:33] yeah, I tried it, but got the same error (which is not a surprise) [15:08:56] r-p-a with no role assigned worked just fine [15:09:04] elukey: for cloud VMs it does, the concept of a 'role' is quite different there [15:19:07] I am inclined to just do this tomorrow [16:01:47] going afk folks, have a nice rest of the day! [16:04:17] \o [16:17:33] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): Deploy revert-risk-model to production - https://phabricator.wikimedia.org/T321594 (10achou) [18:15:28] 10Lift-Wing, 10Machine-Learning-Team: Deploy revert-risk-model to production - https://phabricator.wikimedia.org/T321594 (10calbon) [18:16:34] 10Machine-Learning-Team, 10Data-Services, 10Wikilabels, 10Cloud-VPS (Debian Stretch Deprecation), and 2 others: Upgrade wikilabels databases to buster/bullseye - https://phabricator.wikimedia.org/T307389 (10calbon) [18:16:41] 10Machine-Learning-Team, 10ORES, 10Documentation, 10User-AKlapper: Update docs that ORES will be replaced by Lift Wing - https://phabricator.wikimedia.org/T305963 (10calbon) [18:16:57] 10Lift-Wing, 10Machine-Learning-Team, 10Documentation: Improve Lift Wing documentation - https://phabricator.wikimedia.org/T316098 (10calbon) [18:17:42] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Current Sprint), 10Patch-For-Review, 10User-notice: Deploy "add a link" to 5th round of wikis - https://phabricator.wikimedia.org/T304549 (10calbon) [18:17:44] 10Machine-Learning-Team, 10artificial-intelligence, 10Bad-Words-Detection-System, 10revscoring: Add language support for Cantonese (yue) - https://phabricator.wikimedia.org/T312776 (10calbon) [18:18:02] 10Machine-Learning-Team: Investigate high latencies registered by the ml-serve api control plane - https://phabricator.wikimedia.org/T310073 (10calbon) [18:18:28] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10CommRel-Specialists-Support (Oct-Dec-2022): Inspect "add a link" models to improve their performance - https://phabricator.wikimedia.org/T309263 (10calbon) [18:18:37] 10Machine-Learning-Team, 10Epic: Migrate Content Translation Recommendation API to Lift Wing - https://phabricator.wikimedia.org/T308164 (10calbon) [18:19:19] 10Machine-Learning-Team: Live Vandalism Detection - https://phabricator.wikimedia.org/T299436 (10calbon) [18:19:21] 10Lift-Wing, 10Machine-Learning-Team: Implement an online feature store - https://phabricator.wikimedia.org/T294434 (10calbon) [18:19:24] 10Lift-Wing, 10Machine-Learning-Team, 10artificial-intelligence: Create a tutorial for deploying a model on toolforge - https://phabricator.wikimedia.org/T281317 (10calbon) [18:19:35] 10Lift-Wing, 10Machine-Learning-Team: Explore ingress filtering for Lift Wing - https://phabricator.wikimedia.org/T300259 (10calbon) [18:19:37] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Custom enwiki-goodfaith Explainer - https://phabricator.wikimedia.org/T301378 (10calbon) [18:19:47] 10Lift-Wing, 10Machine-Learning-Team: Match model-server dockerfiles with blubber files - https://phabricator.wikimedia.org/T289127 (10calbon) [18:20:16] 10Lift-Wing, 10Machine-Learning-Team, 10ORES, 10artificial-intelligence, 10ML-Governance: Create Draft Model Deployment Guidelines - https://phabricator.wikimedia.org/T276598 (10calbon) [18:20:43] 10Lift-Wing, 10Machine-Learning-Team, 10ML-Governance: Outlinks model card - https://phabricator.wikimedia.org/T287527 (10calbon) [18:21:09] 10Lift-Wing, 10Machine-Learning-Team, 10ML-Governance: Investigate storing model metadata on Wikidata - https://phabricator.wikimedia.org/T286508 (10calbon) [18:21:15] 10Machine-Learning-Team, 10ML-Governance: Publish Draft Guidlines - https://phabricator.wikimedia.org/T276603 (10calbon) [18:21:45] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Connect Outlink topic model to eventgate - https://phabricator.wikimedia.org/T315994 (10calbon) [18:22:49] 10Lift-Wing, 10Machine-Learning-Team, 10Epic: Lift Wing proof of concept - https://phabricator.wikimedia.org/T272917 (10calbon) [18:23:12] 10Machine-Learning-Team: Deploy new fawiki articlequality model to ORES and LiftWing - https://phabricator.wikimedia.org/T319373 (10calbon) [18:23:21] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 6th round of wikis - https://phabricator.wikimedia.org/T304550 (10calbon) [18:23:23] 10Machine-Learning-Team: ORES doesn't support fawiki family template - https://phabricator.wikimedia.org/T314302 (10calbon) [18:23:26] 10Machine-Learning-Team: Retrain fawiki articlequality model - https://phabricator.wikimedia.org/T317531 (10calbon) [18:23:34] 10Lift-Wing, 10Machine-Learning-Team: Load test the Lift Wing cluster - https://phabricator.wikimedia.org/T296173 (10calbon) [18:24:12] 10Lift-Wing, 10Machine-Learning-Team: API Gateway Integration - https://phabricator.wikimedia.org/T288789 (10calbon) [18:24:16] 10Machine-Learning-Team, 10Patch-For-Review: Test ML model-servers with Benthos - https://phabricator.wikimedia.org/T320374 (10calbon) [18:24:22] 10Machine-Learning-Team: Move Wikilabels Postgres Instances to VMs - https://phabricator.wikimedia.org/T312564 (10calbon) [18:24:32] 10Lift-Wing, 10Machine-Learning-Team: Decide external URL scheme (on API GW) for models on Lift Wing - https://phabricator.wikimedia.org/T319178 (10calbon) [18:24:34] 10Machine-Learning-Team, 10Observability-Logging, 10Patch-For-Review, 10SRE Observability (FY2022/2023-Q2): Logging spam from revscoring deploys - https://phabricator.wikimedia.org/T320468 (10calbon) [18:24:42] 10Machine-Learning-Team, 10Epic: Explore what would be required to migrate the content translation recommendation model to Lift Wing - https://phabricator.wikimedia.org/T308165 (10calbon) [18:24:51] 10Machine-Learning-Team: Help Language team to make progress on open MT models to be used by Content Translation tool - https://phabricator.wikimedia.org/T302516 (10calbon) [18:26:18] 10Machine-Learning-Team, 10ORES, 10Wikidata, 10Wikidata-Query-Service: Measure maturity/quality of Wikidata items in a way it can be queried - https://phabricator.wikimedia.org/T166427 (10calbon)