[06:31:47] hello folks! [06:31:50] kevinbazira: nice! [06:49:01] (03PS1) 10Elukey: Update billiard dependency to 3.6.4.0 [research/ores/wheels] (python37) - 10https://gerrit.wikimedia.org/r/785106 (https://phabricator.wikimedia.org/T303801) [06:49:41] (03CR) 10Elukey: [C: 03+2] Update billiard dependency to 3.6.4.0 [research/ores/wheels] (python37) - 10https://gerrit.wikimedia.org/r/785106 (https://phabricator.wikimedia.org/T303801) (owner: 10Elukey) [06:49:49] (03Merged) 10jenkins-bot: Update billiard dependency to 3.6.4.0 [research/ores/wheels] (python37) - 10https://gerrit.wikimedia.org/r/785106 (https://phabricator.wikimedia.org/T303801) (owner: 10Elukey) [06:51:13] (03PS5) 10Elukey: Update scap settings for the Python 3.7 migration [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/784649 (https://phabricator.wikimedia.org/T303801) [06:54:27] (03PS6) 10Elukey: Update scap settings for the Python 3.7 migration [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/784649 (https://phabricator.wikimedia.org/T303801) [06:54:41] (03PS7) 10Elukey: Update scap settings for the Python 3.7 migration [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/784649 (https://phabricator.wikimedia.org/T303801) [06:55:58] (03PS8) 10Elukey: Update scap settings for the Python 3.7 migration [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/784649 (https://phabricator.wikimedia.org/T303801) [07:39:44] good morning! [07:39:56] o/ [07:58:39] (03PS1) 10AikoChou: outlink: handle http bad request [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/785112 (https://phabricator.wikimedia.org/T306029) [08:41:19] PASS: 124 requests sent to deployment-ores02.deployment-prep.eqiad1.wikimedia.cloud. All assertions passed. [08:41:28] this is the new VM running python 3.7 \o/ [08:41:30] wooowww [08:47:02] 10Machine-Learning-Team, 10Patch-For-Review: Upgrade ORES to Debian Buster or Bullseye - https://phabricator.wikimedia.org/T303801 (10elukey) With the last version of wheels and https://gerrit.wikimedia.org/r/c/mediawiki/services/ores/deploy/+/784649 I was able to run tests on all models with good results! So... [08:47:20] 10Machine-Learning-Team, 10Patch-For-Review: Upgrade ORES to Debian Buster - https://phabricator.wikimedia.org/T303801 (10elukey) [08:49:33] taking a little break [08:57:03] niceeeeee \o/ [09:53:56] 10Machine-Learning-Team, 10Patch-For-Review: Upgrade ORES to Debian Buster - https://phabricator.wikimedia.org/T303801 (10elukey) I just merged https://gerrit.wikimedia.org/r/785124, and now we are able to switch from celery 4 to celery 5 configs via hiera (even host-by-host if needed). [10:43:18] (03CR) 10Elukey: outlink: handle http bad request (033 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/785112 (https://phabricator.wikimedia.org/T306029) (owner: 10AikoChou) [10:43:35] * elukey lunch! [12:52:54] one small nit - scap (our deployment tool) is not compatible with Buster/Python3.7 and git-lfs, so I filed https://gerrit.wikimedia.org/r/c/mediawiki/tools/scap/+/785154 [12:53:17] that means a new release + rollout of scap if accepted :( [13:04:27] ok so despite the list of things to do, we should have a migration plan in theory [13:18:47] 10Machine-Learning-Team: Fix alternate names for ML ks8 TLS certificates - https://phabricator.wikimedia.org/T306613 (10elukey) [13:39:09] going to add the new 4 nodes to ml-serve-eqiad [13:43:24] (after that we'll have eqiad and codfw with 8 worker nodes each) [13:56:24] Morning all [15:03:21] chrisalbon: ORES seems to run fine on python 3.7 with the new wheels [15:03:32] in deployment-prep I can see all models scoring correctly [15:04:30] I think that we could also bump pyyaml, just realized it [15:06:47] ohhh reallly? [15:06:54] yep yep [15:08:30] the upgrade will also involve scap (our deployment system in python) since git-lfs is upgraded an one option used by scap is deprecated [15:13:12] (03PS1) 10Elukey: Upgrade yamlconf to 0.2.5 and PyYAML to 5.4 [research/ores/wheels] (python37) - 10https://gerrit.wikimedia.org/r/785174 (https://phabricator.wikimedia.org/T303801) [15:15:53] (03Abandoned) 10Elukey: Upgrade yamlconf to 0.2.5 and PyYAML to 5.4 [research/ores/wheels] (python37) - 10https://gerrit.wikimedia.org/r/785174 (https://phabricator.wikimedia.org/T303801) (owner: 10Elukey) [15:25:17] (03PS1) 10Elukey: Upgrade yamlconf to 0.2.5 and PyYAML to 5.4 [research/ores/wheels] (python37) - 10https://gerrit.wikimedia.org/r/785177 (https://phabricator.wikimedia.org/T303801) [15:26:25] (03CR) 10Elukey: [C: 03+2] Upgrade yamlconf to 0.2.5 and PyYAML to 5.4 [research/ores/wheels] (python37) - 10https://gerrit.wikimedia.org/r/785177 (https://phabricator.wikimedia.org/T303801) (owner: 10Elukey) [15:27:22] (03PS9) 10Elukey: Update scap settings for the Python 3.7 migration [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/784649 (https://phabricator.wikimedia.org/T303801) [15:27:48] (03PS10) 10Elukey: Update scap settings for the Python 3.7 migration [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/784649 (https://phabricator.wikimedia.org/T303801) [15:28:03] (03PS11) 10Elukey: Update scap settings for the Python 3.7 migration [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/784649 (https://phabricator.wikimedia.org/T303801) [15:28:11] (03PS12) 10Elukey: Update scap settings for the Python 3.7 migration [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/784649 (https://phabricator.wikimedia.org/T303801) [15:32:25] all good even with pyyaml and yamlconf upgraded [15:33:14] 10Machine-Learning-Team, 10Patch-For-Review: Upgrade ORES to Debian Buster - https://phabricator.wikimedia.org/T303801 (10elukey) Upgraded also yamlconf and PyYAML to 5.4 since we were not able to do it a while ago (due to Python 3.5). [15:34:03] (03PS13) 10Elukey: Update scap settings for the Python 3.7 migration [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/784649 (https://phabricator.wikimedia.org/T303801) [16:19:05] going afk folks! Enjoy the longer weekend, talk with you on Monday! [16:50:32] Bye Luca! [16:57:28] bye Luca :) [17:00:10] Also Aiko, just to make sure you know. Tomorrow is the WMF holiday for everyone [17:01:24] what really? why? [17:01:55] Earth Day [17:03:37] Ohhh! [17:07:14] that's nice thanks for letting me know [17:07:51] that's why Luca said it's a longer weekend 😄 [17:22:30] Yep. Have a fun day off! [17:59:31] Since I'm on PTO and Monday is a half-day in Zurich, I'll be keeping that PTO booked and take Monday off instead :) [17:59:53] (Safeguard does not know Earth Day is a holiday at WMF) [18:15:54] Sounds good!