[04:44:55] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): API Gateway Integration - https://phabricator.wikimedia.org/T288789 (10tstarling) Hi, I see you escalated a task to UBN priority. Do you need help? If not, please refer to [[https://www.mediawiki.org/wiki/Phabricator/Project_management#Priority_levels|mw:Ph... [06:12:04] (03CR) 10Kevin Bazira: editquality: refactor setting of the HTTP host header into its own method (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/805388 (https://phabricator.wikimedia.org/T309623) (owner: 10Kevin Bazira) [08:17:13] 10Lift-Wing, 10Machine-Learning-Team (Active Tasks): API Gateway Integration - https://phabricator.wikimedia.org/T288789 (10elukey) p:05Unbreak!→03High @tstarling thanks for the ping, I have downgraded the task to High. [08:22:39] hi folks [08:56:06] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Set up the ml-cache clusters - https://phabricator.wikimedia.org/T302232 (10elukey) I tried to run the Cassandra a instance on ml-cache1001 and I got an error while starting it, since the TLS truststore was not present on disk... [09:21:39] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Set up the ml-cache clusters - https://phabricator.wikimedia.org/T302232 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host ml-cache1002.eqiad.wmnet with OS buster [09:22:44] first cassandra instance up! on ml-cache1001 [09:22:53] going to reimage the other two eqiad nodes to buster [09:32:38] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Set up the ml-cache clusters - https://phabricator.wikimedia.org/T302232 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by elukey@cumin1001 for host ml-cache1003.eqiad.wmnet with OS buster [09:55:07] 10Lift-Wing, 10artificial-intelligence, 10Machine-Learning-Team (Active Tasks): Upload draftquality model binaries to storage - https://phabricator.wikimedia.org/T310701 (10kevinbazira) a:03kevinbazira [10:08:48] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Set up the ml-cache clusters - https://phabricator.wikimedia.org/T302232 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host ml-cache1003.eqiad.wmnet with OS buster completed: - ml-... [10:11:38] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Set up the ml-cache clusters - https://phabricator.wikimedia.org/T302232 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by elukey@cumin1001 for host ml-cache1002.eqiad.wmnet with OS buster completed: - ml-... [10:13:44] elukey: we'll have to do more Bullseye reboots. I'll take care of it [10:14:34] elukey: I'll leave the cache machines alone for now, since I don't know if you're working on them [10:15:03] klausman: sure! I am reimaging them so it shouldn't be a concern [10:15:21] ack. [10:19:43] elukey: btw, moritz mentions that -some_ the reasons for needing Buster for Cassandra are gone [10:22:18] yep yep but Eric mentioned that they are still working on it, sooo I didn't object :) [10:22:26] the eqiad cluster is up! [10:23:09] nice [10:23:34] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Set up the ml-cache clusters - https://phabricator.wikimedia.org/T302232 (10elukey) ` root@ml-cache1001:/var/log/cassandra# nodetool-a status Datacenter: eqiad ================= Status=Up/Down |/ State=Normal/Leaving/Joining/M... [10:28:34] * elukey lunch! [12:02:06] Ok, ml-serve*eqiad done, now lunch [12:05:07] (03CR) 10Klausman: [C: 03+1] editquality: refactor setting of the HTTP host header into its own method [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/805388 (https://phabricator.wikimedia.org/T309623) (owner: 10Kevin Bazira) [14:09:49] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks): Send score to eventgate when requested - https://phabricator.wikimedia.org/T301878 (10elukey) >>! In T301878#8006109, @Ottomata wrote: > I know nothing about kserve. Are the 'KServe services' that will respond with prediction [[ https://knative.de... [14:14:01] Morning all! [14:14:07] o/ [14:20:51] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks): Send score to eventgate when requested - https://phabricator.wikimedia.org/T301878 (10Ottomata) > mediawiki.revision-score. Makes a lot of sense to me! FYI in {T308017} and {T310082}, we are changing the way we model state change even... [14:33:51] chrisalbon: https://phabricator.wikimedia.org/T301878#8008932 [14:34:00] (nothing urgent, when you have time) [14:34:16] this seems exactly what we would want for lift wing right? [14:41:19] (03CR) 10Elukey: editquality: refactor setting of the HTTP host header into its own method (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/805388 (https://phabricator.wikimedia.org/T309623) (owner: 10Kevin Bazira) [14:57:14] elukey yeah it does. Do you see any potential problems making that change? [14:58:18] chrisalbon: we'd need to identify users of the current stream (like https://wikitech.wikimedia.org/wiki/Search/articletopic) and offer a migration path, but I'd say nothing big [14:59:30] As we talked about yesterday, I’d prefer we build what we think is the best way to do this and then help people migrate [15:00:35] +1 yes [15:00:53] cool, sounds good [15:29:44] 10Lift-Wing, 10Epic, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: Set up the ml-cache clusters - https://phabricator.wikimedia.org/T302232 (10elukey) Next step is to bootstrap the codfw cluster, and then we should be done. We should try to figure out if we can use Bullseye and not Buster tho... [18:16:04] (03CR) 10AikoChou: editquality: refactor setting of the HTTP host header into its own method (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/805388 (https://phabricator.wikimedia.org/T309623) (owner: 10Kevin Bazira)