[00:10:55] 10Machine-Learning-Team, 10ORES, 10PageTriage, 10ci-test-error (WMF-deployed Build Failure): CI broken for ORES/PageTriage. Insert returned unacceptable warning: Data truncated for column 'oresc_probability' at row 1 - https://phabricator.wikimedia.org/T355089 (10Novem_Linguae) Any tips for triggering this... [00:23:08] 10Machine-Learning-Team, 10ORES, 10PageTriage, 10ci-test-error (WMF-deployed Build Failure): CI broken for ORES/PageTriage. Insert returned unacceptable warning: Data truncated for column 'oresc_probability' at row 1 - https://phabricator.wikimedia.org/T355089 (10Novem_Linguae) includes/Storage/SqlScoreSto... [06:12:53] (03PS1) 10Kevin Bazira: test: refactor outlink, revertrisk, and revscoring load tests [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/990678 (https://phabricator.wikimedia.org/T354722) [06:17:03] 10Machine-Learning-Team, 10ORES, 10PageTriage, 10ci-test-error (WMF-deployed Build Failure): CI broken for ORES/PageTriage. Insert returned unacceptable warning: Data truncated for column 'oresc_probability' at row 1 - https://phabricator.wikimedia.org/T355089 (10tstarling) I would suggest rounding with `s... [07:16:27] Morning! [07:38:50] Morning Ilias o/ [07:42:03] I have been looking at T348850, where you tested locust. it seems interesting! [07:42:21] is it the Python tool we should work with or there are other tools you wanted to explore? [07:57:40] Hey,we can check if there are other tools as well. Afaik it seems the most prominent [07:58:02] I'll share my findings in this week's meeting and we can take it on from there [08:02:15] Glad you like it :D [08:02:21] I found it nice as well [08:09:40] yep, a Python tool would definitely make load testing easier for the team than Lua. [08:09:40] looking forward to your findings :) [08:09:40] in the meantime, is it ok if I explore how we can use locust to store previous results? [08:09:40] it's something you talk about in T351939 but it's not locust specific. [08:29:49] I think that would be great! [08:30:17] We need a representation that we can use to compare the new values against the old ones [08:31:20] okok let me look into it. [08:35:21] I'm trying to think what we could store. Trying to brainstorm ideas.. [08:36:13] Perhaps saving a csv with the results. Then after we run the load test we can load this csv in a table (pandas dataframe) and join with the new results and compare [08:41:39] 10Machine-Learning-Team, 10ORES, 10PageTriage, 10ci-test-error (WMF-deployed Build Failure): CI broken for ORES/PageTriage. Insert returned unacceptable warning: Data truncated for column 'oresc_probability' at row 1 - https://phabricator.wikimedia.org/T355089 (10Novem_Linguae) I can now test this in local... [08:43:58] yep, that would definitely give us the comparison we need. [08:54:43] morning :) [09:00:39] o/ aiko [09:06:02] (03PS1) 10Novem Linguae: SqlScoreStorage: round probabilities to nearest thousandth [extensions/ORES] - 10https://gerrit.wikimedia.org/r/990949 (https://phabricator.wikimedia.org/T355089) [09:07:53] 10Machine-Learning-Team, 10ORES, 10PageTriage, 10ci-test-error (WMF-deployed Build Failure): CI broken for ORES/PageTriage. Insert returned unacceptable warning: Data truncated for column 'oresc_probability' at row 1 - https://phabricator.wikimedia.org/T355089 (10Novem_Linguae) I think I figured it out :)... [09:07:57] (03CR) 10CI reject: [V: 04-1] SqlScoreStorage: round probabilities to nearest thousandth [extensions/ORES] - 10https://gerrit.wikimedia.org/r/990949 (https://phabricator.wikimedia.org/T355089) (owner: 10Novem Linguae) [09:08:49] (03PS2) 10Novem Linguae: SqlScoreStorage: round probabilities to nearest thousandth [extensions/ORES] - 10https://gerrit.wikimedia.org/r/990949 (https://phabricator.wikimedia.org/T355089) [09:10:53] (03PS3) 10Novem Linguae: SqlScoreStorage: round probabilities to nearest thousandth [extensions/ORES] - 10https://gerrit.wikimedia.org/r/990949 (https://phabricator.wikimedia.org/T355089) [09:53:33] Morning! [09:53:58] https://phabricator.wikimedia.org/F41693028 <- Experimental latency heatmap for the rec-api-ng service [09:54:19] (this is a side-product of me getting the SLI/SLO for thats service to work :D) [09:57:30] kevinbazira: ^^^ I think you might be interested :) [09:58:23] morning Tobias! [09:59:24] klausman o/ [10:00:26] super cool latency heatmap. what did you use to generate that? [10:05:38] It's just the istio latency buckets we need for the SLO anyway. And then some finagling with Grafana graph setup [10:05:55] Might be a good addition to some future LW Service dashboard [10:06:57] nice! [10:07:46] +1 on adding it to LW service dashboards in the future [10:11:41] (03CR) 10Tim Starling: [C: 03+2] SqlScoreStorage: round probabilities to nearest thousandth [extensions/ORES] - 10https://gerrit.wikimedia.org/r/990949 (https://phabricator.wikimedia.org/T355089) (owner: 10Novem Linguae) [10:14:47] (03Merged) 10jenkins-bot: SqlScoreStorage: round probabilities to nearest thousandth [extensions/ORES] - 10https://gerrit.wikimedia.org/r/990949 (https://phabricator.wikimedia.org/T355089) (owner: 10Novem Linguae) [10:16:29] they look nice Tobias :) [10:19:08] ty <3 [10:19:19] I shared another one on Slack [10:31:03] (03PS9) 10Novem Linguae: Don't use live configuration [extensions/ORES] - 10https://gerrit.wikimedia.org/r/957970 (https://phabricator.wikimedia.org/T345922) (owner: 10Jsn.sherman) [11:04:05] 10Machine-Learning-Team, 10ORES, 10PageTriage, 10MW-1.42-notes (1.42.0-wmf.15; 2024-01-23), 10ci-test-error (WMF-deployed Build Failure): CI broken for ORES/PageTriage. Insert returned unacceptable warning: Data truncated for column 'oresc_probability' at row 1 - https://phabricator.wikimedia.org/T355089 (... [11:09:29] * klausman lunch [11:48:49] * isaranto lunch [14:16:13] klausman: I think these histograms (lineplots actually) are a bit easier to read than heatmaps https://grafana.wikimedia.org/d/n3LJdTGIk/kserve-inference-services?orgId=1&var-cluster=eqiad%20prometheus%2Fk8s-mlserve&var-namespace=revscoring-editquality-goodfaith&var-component=All&var-model_name=enwiki-goodfaith [14:16:43] well, at least for me so it may be a personal preference [14:16:56] But the cool factor! ;) [14:18:28] One thing the heatmaps do show a bit cleare is "banding", i.e. if you have separate classes of requests, one at a lower latency (mostly) than the other. E.g. in the rec-api-ng one, you can see that the replies for "this article doesn't exist" are much faster (milliseconds) than replies with substance (>1s, typically) [14:18:50] So I guess it's about personal preference and what kind of information you're trying to glean [14:26:04] going to run a quick errand back in a bit [14:37:25] true! also heatmaps give some information about density/volume, info which we don't get by lines [14:41:12] (03CR) 10Ilias Sarantopoulos: "LGTM! one question: is there a reason for moving the input files to the top directory?" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/990678 (https://phabricator.wikimedia.org/T354722) (owner: 10Kevin Bazira) [14:54:55] (03CR) 10Kevin Bazira: [V: 03+2 C: 03+2] test: refactor outlink, revertrisk, and revscoring load tests (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/990678 (https://phabricator.wikimedia.org/T354722) (owner: 10Kevin Bazira) [14:59:30] 10Machine-Learning-Team, 10Patch-For-Review: Refactor wrk load tests to make them DRY - https://phabricator.wikimedia.org/T354722 (10kevinbazira) outlink, revertrisk, and revscoring isvc load tests have been refactored to use functions from the `utils.lua` shared module. [14:59:46] 10Machine-Learning-Team, 10Patch-For-Review: Refactor wrk load tests to make them DRY - https://phabricator.wikimedia.org/T354722 (10kevinbazira) [15:07:22] still in an interview! [17:22:09] * isaranto afk!