[05:03:44] FIRING: LiftWingServiceErrorRate: ... [05:03:44] LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=codfw%20prometheus/k8s-mlserve&var-namespace=revertrisk&var-backend=revertrisk-language-agnostic-predictor-default.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [06:44:00] INFO:root:Unsupported lang: www. [06:44:01] ERROR:root:An error has occurred while fetching revisions: [2378529203] (www). Reason: Cannot connect to host en.wikipedia.org:443 ssl:default [None] [06:44:23] so yeah non-well-formed requests [06:44:48] ideally we should return HTTP 400 in this case though [09:03:59] FIRING: LiftWingServiceErrorRate: ... [09:03:59] LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=codfw%20prometheus/k8s-mlserve&var-namespace=revertrisk&var-backend=revertrisk-language-agnostic-predictor-default.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [13:03:59] FIRING: LiftWingServiceErrorRate: ... [13:03:59] LiftWing service has a high rate of non 2/3/400 error code responses - https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing/Alerts#LiftWingServiceErrorRate - https://grafana.wikimedia.org/d/G7yj84Vnk/istio?orgId=1&refresh=30s&var-cluster=codfw%20prometheus/k8s-mlserve&var-namespace=revertrisk&var-backend=revertrisk-language-agnostic-predictor-default.%2A - https://alerts.wikimedia.org/?q=alertname%3DLiftWingServiceErrorRate [13:25:42] wtf