[00:32:26] <accraze>	 hmm yeah so the itwiki model binary issues were definitely a git lfs smudge issue
[00:33:22] <accraze>	 accraze@stat1008:~/editquality/models$ s3cmd -c /etc/s3cmd/cfg.d/ml-team.cfg du -H s3://wmf-ml-models/damaging/itwiki/20220214171756/model.bin
[00:33:31] <accraze>	    9M       1 objects s3://wmf-ml-models/damaging/itwiki/20220214171756/model.bin
[00:33:54] <accraze>	 itwiki-damaging should be good now -- checking on itwiki-goodfaith
[00:40:06] <accraze>	 ok itwiki-goodfaith should be good now
[00:40:11] <accraze>	 accraze@stat1008:~/editquality/models$ s3cmd -c /etc/s3cmd/cfg.d/ml-team.cfg du -H s3://wmf-ml-models/goodfaith/itwiki/20220214171756/model.bin
[00:40:23] <accraze>	    8M       1 objects s3://wmf-ml-models/goodfaith/itwiki/20220214171756/model.bin
[00:47:09] <accraze>	 the pods are still in a CrashLoopBackOff state -- if they don't resolve themselves we may need to manually delete them and redeploy
[07:33:02] <elukey>	 accraze: I ran ` kubectl delete pod itwiki-goodfaith-predictor-default-kd56q-deployment-6c4f57gqhzn -n revscoring-editquality-goodfaith` in ml-serve-eqiad and now the pod is up and running
[07:33:39] <elukey>	 just issued a similar comand for codfw
[07:33:59] <elukey>	 (in this way we force the storage initializer container to re-execute, to re-download the model
[07:34:02] <elukey>	 )
[07:44:12] * elukey back to afk :)
[15:15:09] <wikibugs>	 10Lift-Wing, 10Machine-Learning-Team: Support (or not) the ORES augmented feature output in liftwing - https://phabricator.wikimedia.org/T301766 (10achou) @elukey Yes, I would like to do it! :)
[16:13:12] <accraze>	 o/
[16:13:47] <accraze>	 thanks for deleting the pods elukey! makes sense since we need to run the init:container to have the storage-initializer re-download the model
[16:17:21] <wikibugs>	 10Machine-Learning-Team, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:(Need By: TBD) rack/setup/install ml-cache100[1-3] - https://phabricator.wikimedia.org/T299435 (10Jclark-ctr) | name |rack_name |port |cableid ml-cache1001 E1 23 20220147 ml-cache1002 E2 23 20220137 ml-cache1003 F1 23 20220125 |
[16:46:53] <chrisalbon>	 Morning all!
[17:22:05] <wikibugs>	 10ORES, 10artificial-intelligence, 10articlequality-modeling, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES deployment - Winter 2022 - nlwiki articlequality/hiwiki editquality/ores observability - https://phabricator.wikimedia.org/T300195 (10Halfak) My spot checking looks good on Beta....