[09:01:55] <aiko>	 morning :)
[09:02:07] <klausman>	 \o
[09:08:38] <elukey>	 hello :)
[09:18:26] <elukey>	 aiko: o/ one qs about https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/849627
[09:18:30] <elukey>	 what is the idea for staging?
[09:19:29] <elukey>	 because you are swapping the nsfw isvc with the revert-risk one afaics
[09:30:36] <aiko>	 elukey: the idea is I only want to deploy revert-risk to staging, because nsfw isvc had been deployed to prod, so it doesn't need to be in staging.
[09:31:05] <aiko>	 elukey: and we don't want too many pods in staging
[09:34:47] <elukey>	 aiko: yeah but we'd need a representative of each model server in staging to test
[09:35:03] <elukey>	 so I'd say to remove the staging override, and then we are good to go
[09:35:26] <elukey>	 these are small model server deployments so we can afford it
[09:35:32] <elukey>	 does it sound good?
[09:40:12] <aiko>	 elukey: ok considering we need a representative of each model server for test purpose, that sounds good
[09:40:28] <aiko>	 elukey: I'll update the patch :)
[09:48:43] <aiko>	 updated :)
[09:52:16] <elukey>	 merged :)
[10:15:56] <elukey>	 https://aws.amazon.com/de/blogs/containers/using-prometheus-to-avoid-disasters-with-kubernetes-cpu-limits/ is really nice
[10:17:38] <elukey>	 going to lunch in a bit!
[10:19:51] <wikibugs>	 10Machine-Learning-Team, 10ContentTranslation, 10SRE, 10Wikimedia Enterprise: Run NLLB-200 model in a new instance - https://phabricator.wikimedia.org/T321781 (10Pginer-WMF)
[10:20:32] <wikibugs>	 10Machine-Learning-Team, 10ContentTranslation, 10SRE, 10Wikimedia Enterprise: Run NLLB-200 model in a new instance - https://phabricator.wikimedia.org/T321781 (10Pginer-WMF)
[12:09:50] <wikibugs>	 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 6th round of wikis - https://phabricator.wikimedia.org/T304550 (10kevinbazira) The conclusion on the backtesting results is that most of the languages look fine besides:  - crhwiki and cuwiki whose precision is below...
[12:25:54] <klausman>	 elukey: when you're back, I'd love some help with that Postgres Puppet change
[13:00:50] <klausman>	 (meeting now, til 15:00)
[13:12:07] <elukey>	 I am back :)
[13:45:31] <klausman>	 above time is wrong, meeting ends at 16:00 :)
[13:46:57] <elukey>	 I added some comments to the code review :)
[13:47:11] <klausman>	 merci!
[14:02:34] <klausman>	 elukey: the weird thing is that I still get a PCC error that says that the role can't be found (https://puppet-compiler.wmflabs.org/pcc-worker1001/37807/wikilabels-database-02.wikilabels.eqiad1.wikimedia.cloud/prod.wikilabels-database-02.wikilabels.eqiad1.wikimedia.cloud.err)
[14:04:57] <elukey>	 (meeting)
[14:06:08] <klausman>	 (ack)
[14:12:28] <chrisalbon>	 Morning all!
[14:23:37] <klausman>	 \o hey crhis
[14:23:42] <klausman>	 chris*
[14:29:48] <elukey>	 klausman: checking pcc now
[14:29:51] <elukey>	 hello Chris
[14:35:39] <elukey>	 klausman: I think it is a weird error due to the fact that the role is not merged yet, I wouldn't worry about it
[14:37:36] <klausman>	 Alright.
[14:38:41] <elukey>	 aiko: wow I got a weird error for the process pool while testing
[14:38:41] <elukey>	 concurrent.futures.process.BrokenProcessPool: A child process terminated abruptly, the process pool is not usable anymore
[14:42:23] <elukey>	 perfs are still not good with the process pool
[14:42:28] <elukey>	 will need to work on it more
[14:54:17] <aiko>	 elukey: really! does that error happen frequently?
[15:37:18] <elukey>	 aiko: sporadically afaics, but probably the exeption needs to be handled
[15:43:03] <klausman>	 elukey: you think the postgres patch is ready to merge?
[15:44:31] <elukey>	 +1ed
[15:44:32] <klausman>	 I take that as a yes :)
[15:45:14] <elukey>	 the code with the process pool seems to be slow when traffic hits the model server sigh :(
[15:45:50] <elukey>	 I will have to run more comparisons
[15:57:40] <klausman>	 elukey: one last puppet fix :)
[16:16:41] <elukey>	 ship it!
[16:16:50] <elukey>	 going afk!
[16:51:49] <klausman>	 \o