[02:31:40] 10Machine-Learning-Team: Move secret keys to constants in WikiGPT - https://phabricator.wikimedia.org/T329135 (10kevinbazira) [08:28:47] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Data-Persistence, and 9 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) [08:29:01] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Data-Persistence, and 9 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) We'll depool eqiad I would assume? cc @Joe @akosiaris We'd still need to switchover m1 master (we do have m1 databases but I guess w... [08:36:29] 10Machine-Learning-Team, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10MoritzMuehlenhoff) [08:38:43] 10Machine-Learning-Team, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10MoritzMuehlenhoff) [08:41:25] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Data-Persistence, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) [09:01:42] hello folks! [09:03:56] 10Machine-Learning-Team, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10Marostegui) [09:04:36] 10Machine-Learning-Team, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 9 others: codfw row B switches upgrade - https://phabricator.wikimedia.org/T327991 (10Marostegui) [09:15:51] 10Machine-Learning-Team: Review ORES traffic to better understand Lift Wing's requirements - https://phabricator.wikimedia.org/T325763 (10elukey) I checked in the ORES dashboard (https://grafana-rw.wikimedia.org/d/HIRrxQ6mk/ores) and on Thanos (https://thanos.wikimedia.org), I don't see metrics related to specif... [09:29:32] heey! [09:30:01] can someone add kevin to the machine learning group on gitlab? https://gitlab.wikimedia.org/groups/repos/machine-learning/-/group_members [09:30:28] I don't see an invite button - I guess Luca and Tobias u can do it as owners [09:31:10] isaranto: o/ I promoted you as owner, can you check if you can add people? [09:32:20] yep, now I can, thanks! [09:33:38] kevinbazira: I added you! [09:34:41] isaranto: thanks! [09:41:55] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Data-Persistence, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10akosiaris) eqiad will still be depooled for this one. The current timeline for repooling eqiad in on March 8th, 1 day after the proposed timelin... [09:45:38] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Data-Persistence, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) [09:49:43] after some digging I created https://gerrit.wikimedia.org/r/887732 to have per-model metrics in the ORES dashboard [09:53:30] 10Machine-Learning-Team, 10Patch-For-Review: Review ORES traffic to better understand Lift Wing's requirements - https://phabricator.wikimedia.org/T325763 (10elukey) a:03elukey [09:59:54] nice work Luca! I'm not really familiar with puppet code so I might miss something in the review. [10:01:56] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Data-Persistence, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) [10:08:29] 10Machine-Learning-Team, 10Data-Engineering, 10Event-Platform Value Stream: Add a new outlink topic stream for EventGate main - https://phabricator.wikimedia.org/T328899 (10elukey) @Ottomata we decided, as a team, to offer the support for simple Streams via Change-prop leaving the choice of the source event... [10:09:32] isaranto: it is a weird config, but the idea is to create new metrics (so we don't impact the current ones). We'll see from observability if the cardinality is ok [10:09:58] but having more insights on the models would definitely help [10:26:52] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Data-Persistence, and 10 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10Marostegui) [10:32:53] 10Machine-Learning-Team: [WikiGPT] Improve search results of WikiGPT - https://phabricator.wikimedia.org/T329016 (10isarantopoulos) [10:34:08] hey I renamed above task from "add more articles to wikiGPT" to "improve search results"so that we can keep track of the improvements done with screenshots etc. feel free to chip in with ideas or implementations [10:40:19] 10Machine-Learning-Team: [WikiGPT] Improve search results of WikiGPT - https://phabricator.wikimedia.org/T329016 (10isarantopoulos) I figured out how to not have irrelevant links and text in the answer. Initially I got this when asking "Where can I buy ski goggles in Jordan?" {F36802980} after tweaking the promp... [10:42:00] can anybody recommend any place (slack/IRC etc) where I can ask questions about CI/CD for toolforge? [10:43:05] I'd say #wikimedia-cloud [10:43:24] (only interested in CD). what I tried from the documentation with the webhook didn't work , but also ideally there would be a gitlab integration with toolforge to do a push operation instead of a pull [10:43:29] thanks elukey:! [10:48:05] isaranto: or maybe #wikimedia-releng [10:48:23] will do, thanks Guillaume! [10:58:11] TIL gitlab cli can fill in the MR title and description automatically `glab mr create --fill` [10:59:28] I'll be sharing my TILs, please don't pay attention if anything seems to basic for u 😄 [11:01:10] what does glab generate there? [11:04:08] yeah i didn't mention that part. it auto generates title and description from the commit message [11:17:46] neat [11:25:50] * isaranto afk lunch [11:28:26] klausman: o/ [11:28:29] ok if I merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/887732/ ? [11:28:46] should only affect the statsd exporter [11:29:03] yes! [11:29:24] For a moment, I thought I had missed reviewing it :) [11:32:33] ores_model_revision_scored_total{model="damaging",wiki="eswiki"} 2 [11:32:34] ores_model_revision_scored_total{model="damaging",wiki="fiwiki"} 5 [11:32:36] \o/ [11:32:40] wheee! [11:33:08] I'll let puppet to run on all nodes, I've only done 1001 [11:33:56] I'll laugh my ass off if we find that 75% of models get <1qps [11:33:59] +1k metrics for each node, it is a bit but surely useful [11:34:04] yeah [11:34:11] Plus, it's not going to be forever. [11:34:11] this is my theory [11:40:07] going to watch https://w.wiki/6KCB [11:40:20] so far only few nodes got the update, we'll see how it goes [12:01:39] Hello all. FYI we're about to go ahead and try to move some GPUS from Hadoop to the DSE Kubernetes cluster today: T318696 🤞 [12:06:52] nice! Good lunch :) [12:07:04] Better graph for scores: https://w.wiki/6KCb [12:08:31] wow, the short urls only differ in case of the last character!? [13:01:01] great work elukey: ! [13:02:12] * klausman late lunch [13:43:10] 10Machine-Learning-Team, 10Data-Engineering, 10Event-Platform Value Stream: Add a new outlink topic stream for EventGate main - https://phabricator.wikimedia.org/T328899 (10Ottomata) > live with any reliability promises: TBD Wait! Actually, what I said here is not true! I believe we will have `mediawiki.p... [15:00:32] Sry will be 2' late [15:41:06] 10Machine-Learning-Team: Investigate procuring and installing two GPUs on Lift Wing - https://phabricator.wikimedia.org/T327923 (10BTullis) I can report that two have the first two GPUs on the Lift Wing / DSE cluster. We moved two cards from nodes in the Hadoop cluster to dse-k8s-worker1001, so we now have one... [16:10:03] going afk for the evening folks o/ [16:11:08] bye Ilias o/ [16:19:01] * elukey errand for a bit, will do a final quick check later if people need me! [16:45:08] elukey: I'm "soft heading out". Ping me if the sre.k8s.upgrade-cluster change (886317) needs anything from y side. [16:45:15] my* [17:47:39] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Data-Persistence, and 9 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10BTullis) [18:00:22] 10Machine-Learning-Team, 10DBA, 10Data-Engineering, 10Data-Persistence, and 9 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10BTullis) [18:19:39] klausman: ack sorry just got back! I still have to fix 3/4 comments (involving some refactoring), will try to finish + test it tomorrow morning! [18:23:21] Roger. Now off to the well deserved Feierabend with you ;)