[07:24:54] <isaranto>	 God morning!
[08:33:23] <isaranto>	 * good 
[08:34:11] <isaranto>	 just saw that. My keyboard has a faulty button, should get a new one!
[08:50:08] <aiko>	 morning :D
[09:17:10] <klausman>	 Morning!
[09:17:24] <wikibugs>	 10Machine-Learning-Team, 10Moderator-Tools-Team, 10Research, 10Temporary accounts, 10Trust and Safety Product Team: RevertRisk model readiness for temporary accounts - https://phabricator.wikimedia.org/T352839 (10kostajh) >>! In T352839#9429641, @Tchanders wrote: > @diego We return `temp` from the APIs t...
[09:17:33] <klausman>	 isaranto: seems to be A Thing, given my early-January experience :D
[09:18:28] <isaranto>	 the  "G" keycap fell off  and kinda broke because I put the keyboard in my backpack
[09:18:53] <klausman>	 ah, one of those low-profile kbds?
[09:20:00] <isaranto>	 no just a standard keyboard
[09:21:58] <klausman>	 huh. Maybe add one of those plastic dustcovers?
[09:27:03] <isaranto>	 yeah or a case if there is such a thing
[09:30:13] <klausman>	 Some of the fancier kbd manufacturers also have carry cases for their keyboards, but I think those are more common for the TKL and smaller ones, since people are more likely to carry those around
[11:17:07] <wikibugs>	 10Machine-Learning-Team, 10Observability-Metrics, 10SRE Observability (FY2023/2024-Q3): Gap in metrics rendered from Thanos Rules - https://phabricator.wikimedia.org/T352756 (10fgiunchedi)
[11:42:07] * isaranto lunch
[12:49:49] <wikibugs>	 (03CR) 10AikoChou: [C: 03+1] "LGTM! Thanks for working on this :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/989863 (https://phabricator.wikimedia.org/T354722) (owner: 10Kevin Bazira)
[12:50:03] * klausman lunch
[13:24:56] <wikibugs>	 (03CR) 10Kevin Bazira: [V: 03+2 C: 03+2] "Sure sure, thanks for the review Aiko :)" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/989863 (https://phabricator.wikimedia.org/T354722) (owner: 10Kevin Bazira)
[13:55:48] <wikibugs>	 (03CR) 10Dreamy Jazz: "Needs a manual rebase." [extensions/ORES] - 10https://gerrit.wikimedia.org/r/957970 (https://phabricator.wikimedia.org/T345922) (owner: 10Jsn.sherman)
[13:56:28] <wikibugs>	 10Machine-Learning-Team: Refactor wrk load tests to make them DRY - https://phabricator.wikimedia.org/T354722 (10kevinbazira)
[13:58:20] <wikibugs>	 10Machine-Learning-Team: Refactor wrk load tests to make them DRY - https://phabricator.wikimedia.org/T354722 (10kevinbazira) langid and ores-legacy isvc load tests have been refactored to use functions from the `utils.lua` shared module.
[14:22:01] <isaranto>	 klausman: I created a patch to increase memory allowance in pods for ml-staging. https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/990699
[14:22:10] <klausman>	 checking
[14:22:41] <isaranto>	 I want to successfully deploy the plain version and then experiment on lowering the memory requirements by tweaking the model
[14:22:47] <klausman>	 sgtm
[14:23:02] <klausman>	 +2'd. will push as soon as it is merged
[14:24:24] <isaranto>	 I remember there was a dashboard for node capacity but can't find it
[14:26:04] <klausman>	 https://grafana.wikimedia.org/goto/zCro0vcIk?orgId=1 this one?
[14:27:15] <isaranto>	 oh yeah, thanks. bookmarked!
[14:27:34] <klausman>	 You can also "star" dashboards inside of Grafana
[14:27:46] <klausman>	 But I dunno if that works on the readonly instance
[14:28:24] <klausman>	 change hasd been applied.
[14:28:48] <klausman>	 Looks like we're good on memory on staging, but CPU is running tight
[14:29:58] <isaranto>	 thanks Tobias!
[14:31:47] <isaranto>	 I'll be sending another patch to increase the pod limits
[14:33:23] <klausman>	 ack
[14:34:15] <isaranto>	 sent!
[14:34:36] <isaranto>	 https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/990705
[14:35:52] <isaranto>	 also lowered cpu requests/limits to match the cpu limitranges 
[14:38:09] <klausman>	 +2'd
[14:40:22] <isaranto>	 Danke!
[15:46:55] <aiko>	 the new batch revertrisk model seems to scale very well
[15:47:06] <aiko>	 https://phabricator.wikimedia.org/P54724
[15:47:48] <aiko>	 I'll add a summary to the task
[15:55:46] <klausman>	 Nice to hear. I'm meanwhile banging my head against Prometheus recording rules that randomly drop labels :D
[15:56:58] <isaranto>	 nice Aiko!
[15:57:51] <isaranto>	 I got another OOM on model load :( . Something is off. I'll be trying to run it locally to see what happens with memory usage on load
[16:25:35] <klausman>	 My brain is mush, I'll do some email catchup and then head out early.
[16:37:50] <isaranto>	 <3
[16:38:00] <isaranto>	 today is Blue monday after all
[16:38:15] <isaranto>	 https://en.wikipedia.org/wiki/Blue_Monday_(date)
[16:56:24] <klausman>	 Also, great electronic song from the 80s. Best selling 12" maxi of all time
[17:18:37] <isaranto>	 yeees
[17:24:19] <wikibugs>	 10Machine-Learning-Team: Deploy 7b parameter models from HF - https://phabricator.wikimedia.org/T354870 (10isarantopoulos) Falcon, llama and mistral (and mixtral ) models have been incorporated in the transformers library so we don't need to use the `trust_remote_code=True`.  In the ml-staging deployment we're s...
[17:44:22] * isaranto afk!
[20:03:48] <wikibugs>	 (03PS7) 10Novem Linguae: Don't use live configuration [extensions/ORES] - 10https://gerrit.wikimedia.org/r/957970 (https://phabricator.wikimedia.org/T345922) (owner: 10Jsn.sherman)
[20:06:40] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Don't use live configuration [extensions/ORES] - 10https://gerrit.wikimedia.org/r/957970 (https://phabricator.wikimedia.org/T345922) (owner: 10Jsn.sherman)
[20:14:20] <wikibugs>	 10Machine-Learning-Team, 10ORES, 10PageTriage, 10ci-test-error: CI broken for ORES/PageTriage. Insert returned unacceptable warning: Data truncated for column 'oresc_probability' at row 1 - https://phabricator.wikimedia.org/T355089 (10Novem_Linguae)
[20:16:18] <wikibugs>	 (03PS8) 10Novem Linguae: Don't use live configuration [extensions/ORES] - 10https://gerrit.wikimedia.org/r/957970 (https://phabricator.wikimedia.org/T345922) (owner: 10Jsn.sherman)
[20:17:07] <wikibugs>	 (03CR) 10Novem Linguae: "In addition to Jason's changes, I've uploaded a patchset that makes additional changes in the PopulatedSqlModelLookupTest.php file. This f" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/957970 (https://phabricator.wikimedia.org/T345922) (owner: 10Jsn.sherman)
[20:17:37] <wikibugs>	 (03CR) 10Novem Linguae: "I've filed https://phabricator.wikimedia.org/T355089 for the broken CI, which I believe to be unrelated since it is also happening for Pag" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/957970 (https://phabricator.wikimedia.org/T345922) (owner: 10Jsn.sherman)
[20:18:26] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Don't use live configuration [extensions/ORES] - 10https://gerrit.wikimedia.org/r/957970 (https://phabricator.wikimedia.org/T345922) (owner: 10Jsn.sherman)
[20:18:57] <wikibugs>	 (03CR) 10Novem Linguae: "I've filed https://phabricator.wikimedia.org/T355035, which is a suggestion for adding $this->loadExtensionDefaultConfigVars() to MediaWik" [extensions/ORES] - 10https://gerrit.wikimedia.org/r/957970 (https://phabricator.wikimedia.org/T345922) (owner: 10Jsn.sherman)
[22:32:44] <wikibugs>	 10Machine-Learning-Team, 10ORES, 10PageTriage, 10ci-test-error (WMF-deployed Build Failure): CI broken for ORES/PageTriage. Insert returned unacceptable warning: Data truncated for column 'oresc_probability' at row 1 - https://phabricator.wikimedia.org/T355089 (10Novem_Linguae)
[23:17:33] <wikibugs>	 10Machine-Learning-Team, 10ORES, 10PageTriage, 10ci-test-error (WMF-deployed Build Failure): CI broken for ORES/PageTriage. Insert returned unacceptable warning: Data truncated for column 'oresc_probability' at row 1 - https://phabricator.wikimedia.org/T355089 (10matmarex) Error message is new, from {551ec...
[23:37:35] <wikibugs>	 10Machine-Learning-Team, 10ORES, 10PageTriage, 10ci-test-error (WMF-deployed Build Failure): CI broken for ORES/PageTriage. Insert returned unacceptable warning: Data truncated for column 'oresc_probability' at row 1 - https://phabricator.wikimedia.org/T355089 (10Novem_Linguae) A search for decimals longer...