[02:13:49] FIRING: KubernetesDeploymentUnavailableReplicas: ... [02:13:49] Deployment reference-need-predictor-00010-deployment in revision-models at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revision-models&var-deployment=reference-need-predictor-00010-deployment - ... [02:13:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [03:18:49] RESOLVED: KubernetesDeploymentUnavailableReplicas: ... [03:18:49] Deployment reference-need-predictor-00010-deployment in revision-models at eqiad has persistently unavailable replicas - https://wikitech.wikimedia.org/wiki/Kubernetes/Troubleshooting#Troubleshooting_a_deployment - https://grafana.wikimedia.org/d/a260da06-259a-4ee4-9540-5cab01a246c8/kubernetes-deployment-details?var-site=eqiad&var-cluster=k8s-mlserve&var-namespace=revision-models&var-deployment=reference-need-predictor-00010-deployment - ... [03:18:49] https://alerts.wikimedia.org/?q=alertname%3DKubernetesDeploymentUnavailableReplicas [06:21:45] (03PS1) 10Divec: Ensure CORS headers allow all origins [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134401 [06:24:09] (03CR) 10Divec: [C:04-1] "I guess there's probably a cleaner way to do this? (I did this hack so I can run the model server locally, and test my local mediawiki ins" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134401 (owner: 10Divec) [06:29:21] (03CR) 10CI reject: [V:04-1] Ensure CORS headers allow all origins [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134401 (owner: 10Divec) [06:45:17] 10Lift-Wing, 06Machine-Learning-Team, 10EditCheck: Local peacock model server doesn't send CORS headers allowing all origins - https://phabricator.wikimedia.org/T391229 (10dchan) 03NEW [06:45:29] (03PS2) 10Divec: Ensure CORS headers allow all origins [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134401 (https://phabricator.wikimedia.org/T391229) [06:46:18] (03CR) 10CI reject: [V:04-1] Ensure CORS headers allow all origins [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134401 (https://phabricator.wikimedia.org/T391229) (owner: 10Divec) [06:49:04] (03CR) 10Divec: Ensure CORS headers allow all origins [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134401 (https://phabricator.wikimedia.org/T391229) (owner: 10Divec) [07:01:54] Good morning [07:07:08] 10Lift-Wing, 06Machine-Learning-Team, 13Patch-For-Review: LiftWing model-servers log improper JSON in stderr - https://phabricator.wikimedia.org/T389768#10716472 (10kevinbazira) 05In progress→03Resolved All model-servers now log events as JSON serialized output and don't add a newline character in th... [07:10:45] Hi folks o/ [08:47:48] 10Lift-Wing, 06Machine-Learning-Team, 10EditCheck, 13Patch-For-Review: Local peacock model server doesn't send CORS headers allowing all origins - https://phabricator.wikimedia.org/T391229#10716716 (10isarantopoulos) @dchan Does this happen only when running the model locally or also on the API GW? [09:30:33] (03PS1) 10Ozge: feat: refactors edit check with pydantic settings. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 [09:31:22] (03CR) 10CI reject: [V:04-1] feat: refactors edit check with pydantic settings. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 (owner: 10Ozge) [09:33:53] (03CR) 10Ozge: "refactors reading environment variables in edit-check with pydantic settings:" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 (owner: 10Ozge) [09:36:16] (03PS2) 10Ozge: feat: refactors edit check with pydantic settings. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 [09:37:23] (03CR) 10CI reject: [V:04-1] feat: refactors edit check with pydantic settings. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 (owner: 10Ozge) [09:47:49] Ozge: o/ you can check https://integration.wikimedia.org/ci/job/inference-services-pipeline-edit-check/95/console to see CI logs for the test build. inference-services-pipeline-edit-check #95 > console output [09:49:58] (03PS3) 10Ozge: feat: refactors edit check with pydantic settings. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 [10:04:31] 10Lift-Wing, 06Machine-Learning-Team, 10EditCheck, 13Patch-For-Review: Local peacock model server doesn't send CORS headers allowing all origins - https://phabricator.wikimedia.org/T391229#10717019 (10dchan) The error only occurs when running locally. It works correctly when using the API gateway `https://... [10:29:26] @aiko: thank you! Fixed in the latest run: https://integration.wikimedia.org/ci/job/inference-services-pipeline-edit-check/96/ [11:15:07] (03CR) 10AikoChou: "Thanks for working on this! I have two comments." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 (owner: 10Ozge) [11:24:25] (03PS4) 10Ozge: feat: refactors edit check with pydantic settings. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 [11:32:35] (03PS5) 10Ozge: feat: refactors edit check with pydantic settings. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 [11:33:24] (03CR) 10CI reject: [V:04-1] feat: refactors edit check with pydantic settings. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 (owner: 10Ozge) [11:34:14] (03CR) 10Ozge: feat: refactors edit check with pydantic settings. (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 (owner: 10Ozge) [11:52:42] Hey @aiko , how can we add environments variables (model_name) to the unit tests in Jenkins as we want to make it mandatory? [12:02:29] what if we add a default value in the settings instead of adding an env var in the unit tests? [12:04:55] otherwise there would be 2 options: [12:04:55] 1. set the env var in the blubber test image [12:04:55] 2. set the environment variable in the tests in python [12:05:31] I think I'd prefer the second one which would guarantee that tests would pass even when ran outside of docker -- e.g. in a local virtual environment [12:06:51] @isoranto good idea! I’ve set the default value for the model name as `edit_check`. Please feel free to suggest another name, if it does not apply the model naming rules. [12:08:25] (03PS6) 10Ozge: feat: refactors edit check with pydantic settings. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 [12:12:51] (03CR) 10Ilias Sarantopoulos: "Congrats on your first patch which is also a nice improvement🎉" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 (owner: 10Ozge) [12:18:30] (03PS7) 10Ozge: feat: refactors edit check with pydantic settings. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 [12:23:28] (03PS8) 10Ozge: feat: refactors edit check with pydantic settings. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 [12:24:36] (03PS9) 10Ozge: feat: refactors edit check with pydantic settings. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 [12:25:42] (03CR) 10Ozge: "thank you! added a small description." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 (owner: 10Ozge) [13:02:57] (03CR) 10Ozge: feat: refactors edit check with pydantic settings. (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 (owner: 10Ozge) [13:14:11] (03CR) 10Ozge: feat: refactors edit check with pydantic settings. (032 comments) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 (owner: 10Ozge) [13:41:35] (03PS3) 10Ilias Sarantopoulos: edit-check: ensure CORS headers allow all origins [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134401 (https://phabricator.wikimedia.org/T391229) (owner: 10Divec) [13:42:24] (03CR) 10CI reject: [V:04-1] edit-check: ensure CORS headers allow all origins [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134401 (https://phabricator.wikimedia.org/T391229) (owner: 10Divec) [13:46:13] (03PS4) 10Ilias Sarantopoulos: edit-check: ensure CORS headers allow all origins [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134401 (https://phabricator.wikimedia.org/T391229) (owner: 10Divec) [13:50:37] ozge_: if aiko is ok you can +2 and merge. [13:51:14] it is not urgent so we can wait [13:51:24] great job, it is much cleaner! [13:53:32] Thank you! Let me check with @aiko [13:57:14] o/ yoooo! looks good to me :) [13:57:24] (03CR) 10AikoChou: [C:03+1] feat: refactors edit check with pydantic settings. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 (owner: 10Ozge) [14:09:53] 🎉 [14:11:19] Hey, looks like I’m missing something to merge. I guess I can’t +2 https://usercontent.irccloud-cdn.com/file/ZztCTqNR/1744034670.JPG [14:16:07] 06Machine-Learning-Team, 06Data-Engineering, 06Research, 10Event-Platform, 13Patch-For-Review: Emit revision revert risk scores as a stream and expose in EventStreams API - https://phabricator.wikimedia.org/T326179#10717935 (10Ottomata) Okay, that sounds good to me! The `model_name` is a data field, so... [14:22:56] ozge_: can you try again? [14:24:41] I added you to the ml group on gerrit [14:30:17] (03CR) 10Ozge: [C:03+2] feat: refactors edit check with pydantic settings. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 (owner: 10Ozge) [14:31:01] (03CR) 10Ozge: [V:03+2 C:03+2] feat: refactors edit check with pydantic settings. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 (owner: 10Ozge) [14:31:23] (03Merged) 10jenkins-bot: feat: refactors edit check with pydantic settings. [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134643 (owner: 10Ozge) [14:32:42] It worked! Thank you @isaranto . merged [14:39:26] 06Machine-Learning-Team, 10LDAP-Access-Requests, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users & Kerberos identity & deployment POSIX group & ml-team-admins for Ozge Karakaya - https://phabricator.wikimedia.org/T390855#10718084 (10Jelto) 05In progress→03Resolved Özg... [14:48:28] (03PS9) 10AikoChou: edit-check: add SHAP values [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1133426 (https://phabricator.wikimedia.org/T387984) [14:54:59] (03CR) 10Majavah: [C:03+2] build: Updating eslint-config-wikimedia to 0.29.1 [extensions/ORES] - 10https://gerrit.wikimedia.org/r/1132588 (owner: 10Libraryupgrader) [15:01:41] 06Machine-Learning-Team, 06Data-Engineering, 06Research, 10Event-Platform, 13Patch-For-Review: Emit revision revert risk scores as a stream and expose in EventStreams API - https://phabricator.wikimedia.org/T326179#10718156 (10Ottomata) Once this stream is deployed and active, if we'd like to expose it p... [15:12:45] (03CR) 10AikoChou: "Thanks for testing the patch! One thing to clarify is that the tokens are not necessarily complete words—this largely depends on the token" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1133426 (https://phabricator.wikimedia.org/T387984) (owner: 10AikoChou) [15:37:48] (03PS5) 10Ilias Sarantopoulos: edit-check: ensure CORS headers allow all origins [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134401 (https://phabricator.wikimedia.org/T391229) (owner: 10Divec) [15:38:37] (03CR) 10CI reject: [V:04-1] edit-check: ensure CORS headers allow all origins [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134401 (https://phabricator.wikimedia.org/T391229) (owner: 10Divec) [15:39:35] (03PS6) 10Ilias Sarantopoulos: edit-check: ensure CORS headers allow all origins [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1134401 (https://phabricator.wikimedia.org/T391229) (owner: 10Divec) [15:42:26] going afk folks, have a nice evening/rest of day! [15:44:32] (03CR) 10AikoChou: "I ran more examples from our eval data. Here are the responses: https://phabricator.wikimedia.org/P74618" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/1133426 (https://phabricator.wikimedia.org/T387984) (owner: 10AikoChou)