[06:09:18] (03CR) 10Elukey: [C: 03+2] revert-risk: upgrade to KServe 0.11.1 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/964559 (https://phabricator.wikimedia.org/T347550) (owner: 10Elukey) [06:09:52] o/ [06:09:59] Guten tag [06:14:46] (03Merged) 10jenkins-bot: revert-risk: upgrade to KServe 0.11.1 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/964559 (https://phabricator.wikimedia.org/T347550) (owner: 10Elukey) [06:24:59] morning :) [07:00:05] Afk be back in 30-40' [07:08:15] (03CR) 10Elukey: "One nit about naming - in the repo the top level dirs are almost all called something-model (except readability, for some reason). Should " [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/963367 (https://phabricator.wikimedia.org/T347404) (owner: 10Ilias Sarantopoulos) [07:15:24] 10Machine-Learning-Team: Configure envoy settings to enable rec-api-ng container to access endpoints external to k8s/LiftWing - https://phabricator.wikimedia.org/T348607 (10kevinbazira) [07:19:47] kevinbazira: o/ [07:20:02] elukey: o/ [07:20:22] one suggestion - it is fine to create a task for everything, but if you do so I'd suggest to add a little summary in the parent task to clarify the next steps [07:20:35] (like in TXXXXX I am going to fix this, etc..) [07:20:40] yes, that was the next step [07:20:44] otherwise who reads the other tasks doesn't know [07:20:45] okok [07:21:10] https://phabricator.wikimedia.org/T347015 is also missing a summary [07:21:11] etc.. [07:21:25] (it is just a nit, nothing really problematic, it is mostly for people reading) [07:21:49] so did you guys figure out what is the endpoint for wikidata? [07:22:01] if not we can check together [07:24:59] kevinbazira: --^ [07:25:06] I stepped back and worked on wrapping my head around using envoy listeners instead of specifying hosts in the mesh. [07:25:07] I've share the steps we plan to take in T348607 [07:27:57] kevinbazira: yes yes the plan is sound, I am talking about the discovery endpoints to add [07:28:12] we can discuss the ones that you need [07:28:44] ah you already figured it out, I see from the comments in the patch [07:28:46] nice work kevinbazira :) [07:29:05] :) [07:36:00] kevinbazira: new listener for port 6500 deployed to staging [07:37:22] thanks, we are going to proceed with configuring external endpoints in the rec-api to use localhost:6500 and set the respective headers for the apis [07:40:37] super [07:59:18] 10Machine-Learning-Team: Investigate recommendation-api-ng internal endpoint failure - https://phabricator.wikimedia.org/T347475 (10kevinbazira) Following T347475#9235065, it was clear that the rec-api-ng instance on LiftWing doesn't work like the rec-api container outside LiftWing, despite having similar resour... [08:23:46] (03PS8) 10Ilias Sarantopoulos: revscoring: allow local runs [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/963367 (https://phabricator.wikimedia.org/T347404) [08:25:20] (03CR) 10Ilias Sarantopoulos: revscoring: allow local runs (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/963367 (https://phabricator.wikimedia.org/T347404) (owner: 10Ilias Sarantopoulos) [08:26:09] elukey: I changed the package name to `revscoring_model`. already tested it again, works fine [08:26:41] isaranto: ack thanks, do we need to use the _ or is the - an option? [08:26:49] I don't recall what's needed [08:29:33] if we use a hyphen (-) then in order to import we'll need to use importlib e.g. [08:29:33] ``` [08:29:33] import importlib [08:29:33] mod = importlib.import_module("path.to.my-module") [08:29:33] ``` [08:30:07] I took the above from a stackoverflow thread https://stackoverflow.com/questions/7583652/python-module-with-a-dash-or-hyphen-in-its-name but this is what I also know [08:30:19] isaranto: are we going to extend the same behavior to all model servers? If so it is fine if we'll have all top dirs with underscores [08:30:25] it is just to have a minimal consistency [08:31:05] I would suggest yes. We can track all these changes in https://phabricator.wikimedia.org/T347404 [08:31:13] then we can proceed [08:31:25] my suggestion would be to gradually change them all and then put them under src/ dir or sth similar [08:31:26] (03CR) 10Elukey: [C: 03+1] revscoring: allow local runs [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/963367 (https://phabricator.wikimedia.org/T347404) (owner: 10Ilias Sarantopoulos) [08:31:51] thanks for the patience :) [08:31:57] so top level would only be src, test, ,pipeline etc . Otherwise the repo will start to be unreadable [08:32:21] np, there is no hurry! [08:38:54] (03CR) 10Ilias Sarantopoulos: [C: 03+2] revscoring: allow local runs [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/963367 (https://phabricator.wikimedia.org/T347404) (owner: 10Ilias Sarantopoulos) [08:39:03] (03CR) 10Ilias Sarantopoulos: [C: 03+2] revscoring: allow local runs (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/963367 (https://phabricator.wikimedia.org/T347404) (owner: 10Ilias Sarantopoulos) [08:46:01] (03CR) 10Nikerabbit: [V: 03+2] Localisation updates from https://translatewiki.net. [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/963684 (owner: 10L10n-bot) [08:55:49] (03PS9) 10Ilias Sarantopoulos: revscoring: allow local runs [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/963367 (https://phabricator.wikimedia.org/T347404) [09:54:11] isaranto: I think there is a way to add cors headers to envoy, but it is more complicated than what I had in mind originally [09:54:20] going to work on it, envoy's conf is not straightforward [09:54:26] but I think there is a way [09:54:46] I was looking at the api gateway configs yesterday I didnt find anything [09:54:57] if there is anything I can look into or do let me know [09:57:34] yep! [09:57:57] I still need to wrap my head around how envoy works [09:58:25] need to run an errand, bbl! [10:20:58] 10Machine-Learning-Team: Upgrade Revert Risk Language-agnostic docker images to KServe 0.11 - https://phabricator.wikimedia.org/T347550 (10achou) When testing the model in ml-staging, the following error was encountered, resulting in the pod being in a CrashLoopBackOff state: ` Message: Traceback (most recen... [10:29:40] (03PS1) 10AikoChou: revert-risk: add python3-setuptools to revertrisk-la blubber file [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965094 (https://phabricator.wikimedia.org/T347550) [10:31:16] (03PS2) 10AikoChou: revert-risk: add python3-setuptools to revertrisk-la blubber file [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965094 (https://phabricator.wikimedia.org/T347550) [10:52:16] (03CR) 10Ilias Sarantopoulos: [C: 03+1] revert-risk: add python3-setuptools to revertrisk-la blubber file [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965094 (https://phabricator.wikimedia.org/T347550) (owner: 10AikoChou) [10:52:47] aiko: I need to verify that there is no issue caused to the model servers by the last change (that everything is imported correctly etc) [10:53:13] could you try that for revertrisk? by rebasing your patch I mean [10:53:56] I'm only curious about this change https://gerrit.wikimedia.org/r/plugins/gitiles/machinelearning/liftwing/inference-services/+/refs/heads/main/python/process_utils.py#7 [10:55:18] * isaranto lunch [11:21:28] 10Machine-Learning-Team: Decommission ORES configurations and servers - https://phabricator.wikimedia.org/T347278 (10Volans) @klausman the DNS step is marked as done, but I see the ORES SVC records still existing in Netbox ( https://netbox.wikimedia.org/ipam/ip-addresses/?q=ores ) is that a leftover or pending s... [12:18:00] (03CR) 10Klausman: [C: 03+1] revert-risk: add python3-setuptools to revertrisk-la blubber file [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965094 (https://phabricator.wikimedia.org/T347550) (owner: 10AikoChou) [12:22:13] 10Machine-Learning-Team, 10Research: Deploy multilingual readability model to LiftWing - https://phabricator.wikimedia.org/T334182 (10leila) [12:28:58] isaranto: I rebased the patch and it looks no issue. revertrisk doesn't use any functions in process_utils.py though [12:29:43] ack. I saw that the files are copied in blubber that's why I mentioned it. thanks for checking! [12:31:27] (03CR) 10AikoChou: [C: 03+2] "Thanks for the review!" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965094 (https://phabricator.wikimedia.org/T347550) (owner: 10AikoChou) [12:32:12] (03Merged) 10jenkins-bot: revert-risk: add python3-setuptools to revertrisk-la blubber file [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965094 (https://phabricator.wikimedia.org/T347550) (owner: 10AikoChou) [12:38:24] klausman: I tried to see what we'd need to do to "free" the ORES redis instances, and I think https://gerrit.wikimedia.org/r/c/operations/puppet/+/965124 should be enough [12:38:35] we then flush the data and mark their usage in wikitech as unassigned [12:38:40] lemme know if it makes sense [12:38:46] (I'll check with serviceops too) [12:39:24] looking... [12:39:59] LGTM [12:41:26] (03PS1) 10Ilias Sarantopoulos: revscoring: fix mp [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965125 (https://phabricator.wikimedia.org/T348265) [12:43:52] (03PS2) 10Ilias Sarantopoulos: revscoring: fix mp [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965125 (https://phabricator.wikimedia.org/T348265) [12:44:28] (03PS2) 10Ilias Sarantopoulos: revscoring: bump scipy version [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/964954 [12:47:09] (03PS4) 10Ilias Sarantopoulos: revscoring: customize kserve logs [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/964568 (https://phabricator.wikimedia.org/T333804) [12:47:58] Morning all [12:48:11] morning :) [12:48:21] morning! [12:48:54] phew, I fixed the patches I had open for inf-services [12:49:01] rebase hell [12:53:14] 10artificial-intelligence: Use AI to automatically generate edit summaries - https://phabricator.wikimedia.org/T334598 (10Bugreporter2) [13:00:09] isaranto: ah wait the errors on https://github.com/WikiEducationFoundation/WikiEduDashboard/pull/5501 don't say that we don't have cors [13:00:28] but that content-type is not listed in the allowed headers [13:03:37] Headers in the request? [13:03:44] yes [13:04:00] we already have a base cors config in the api-gateway [13:04:04] so I'll just add the use case [13:04:05] ah, yeah, makes sense [13:08:06] https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/965153 [13:09:10] (03CR) 10Elukey: [C: 03+1] revscoring: fix mp [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965125 (https://phabricator.wikimedia.org/T348265) (owner: 10Ilias Sarantopoulos) [13:09:31] (03CR) 10Elukey: [C: 03+1] revscoring: bump scipy version [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/964954 (owner: 10Ilias Sarantopoulos) [13:11:34] ouch, sry I didnt pay close attention [13:12:11] np! I think the fix is easier than I thought [13:12:24] I'll wait for Hugh or Kamila to confirm but we should be good to go [13:14:20] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: kserve CORS error - https://phabricator.wikimedia.org/T348511 (10elukey) The API Gateway already supports CORS, but we don't allow the `Content-type` header in the base config. I filed a change to support it, once reviewed and deployed the issue shoul... [13:35:09] something weird that I don't get [13:35:22] helm fails to deploy only in ml-serve-eqiad for drafttopic [13:35:31] Error: query: failed to query with labels: proto: Unknown: illegal tag 0 (wire type 0) [13:39:47] ah interesting [13:39:47] kubectl get secrets -n revscoring-drafttopic [13:39:48] Error from server: proto: Unknown: illegal tag 0 (wire type 0) [13:39:53] wow never seen this [13:41:11] weird, it doesn't happen always [13:41:24] mmm smells like a lb issue with the kube api [13:42:54] ok it sems ml-serve-ctrl1001 [13:42:57] restarting kube-api [13:44:30] E1011 13:44:19.717981 2665385 status.go:71] apiserver received an error that is not an metav1.Status: &errors.errorString{s:"proto: Unknown: illegal tag 0 (wire type 0)"}: proto: Unknown: illegal tag 0 (wire type 0) [13:46:28] better after the restarts, but not solved [13:50:06] (03CR) 10Nikerabbit: [V: 03+2] Localisation updates from https://translatewiki.net. [research/recommendation-api] - 10https://gerrit.wikimedia.org/r/964461 (owner: 10L10n-bot) [13:50:14] hmmm [13:52:06] 10Lift-Wing, 10Machine-Learning-Team, 10I18n, 10NewFunctionality-Worktype, 10Patch-For-Review: Create a language detection service in LiftWing - https://phabricator.wikimedia.org/T340507 (10isarantopoulos) @santhosh Thanks for creating the model card! Is there a client/system that will use this at the mo... [13:58:38] elukey: I saw some interface errors on AM today, but I don't recall whether it was that machine [13:58:50] ah, no, that was serve2005 [14:35:51] What about creating an llm namespace on LW? There aren't gonna be many pods in there and it can hold all the efforts [14:36:00] I was thinking to deploy langid to that namespace [14:37:16] 10Machine-Learning-Team: Add sha512 checksum files to all the ML's models in the public dir - https://phabricator.wikimedia.org/T347838 (10elukey) We decided during the team meeting to add prompt in the `upload_model` script to copy the model to `/srv/published`. In this way we don't forget :) Next steps: - I... [14:37:22] isaranto: sure makes sense [14:37:52] klausman: could you assist me in this? [14:38:10] pointing me to a patch would be enough to start with! [14:38:14] sure! [14:38:32] or doc in wikitech. I am searching for stuff that you did for revertrisk [14:39:04] digging as we speak :) [14:41:12] https://phabricator.wikimedia.org/T333124#8850464 this change and the three following are what I did for revertrisk [14:42:11] correction: only two changes, 919842 and 919843 [14:43:13] Change 920208 then moved RR from experimental to its own namespace [14:43:37] super! thanks [14:44:03] There's also some stuff for the private repo, but we can do that afterwards [14:46:35] let's do private first :) [14:46:49] (including the labs/private fake creds) [14:47:13] I was just looking at that and I am confused. Did that repo structure change recently? [14:47:52] what do you mean? [14:48:00] I can find zero mention of revertrisk or revert-risk in the private repo at current HEAD [14:48:33] did we ever add them? :) [14:48:39] yes. [14:48:50] I can find my commit that did, but the file it edits is gone [14:49:14] https://gerrit.wikimedia.org/r/c/labs/private/+/919845 [14:49:30] But at HEAD, there is no hieradata/common/profile/kubernetes.yaml anymore [14:50:13] Ooooh, this is due to T325268 [14:50:28] > Migrate use of infrastructure_users tokens to client certificates [14:51:10] ah right! [14:52:01] nevermind then [14:55:14] 10Lift-Wing, 10Machine-Learning-Team: Create llm namespace on Lift Wing - https://phabricator.wikimedia.org/T348661 (10isarantopoulos) [14:56:57] 10Machine-Learning-Team, 10Goal: Goal: Users can query a large language model using the API Gateway and receive a response in a reasonable amount of time. - https://phabricator.wikimedia.org/T348154 (10isarantopoulos) [14:56:59] 10Lift-Wing, 10Machine-Learning-Team: Create llm namespace on Lift Wing - https://phabricator.wikimedia.org/T348661 (10isarantopoulos) [14:57:58] klausman: totally forgot yes, we now have client certs automatically requested [14:58:08] so in theory we shouldn't need any private stuff anymore [14:58:28] Yeah, that's not a big step removed, but it still matters [14:58:44] I'd say it is a big step removed :) [14:58:57] it is basically half of the procedure [14:58:59] ok fine :) [15:01:48] 10Machine-Learning-Team, 10Patch-For-Review: Visualize KServe latency metrics in a dashboard - https://phabricator.wikimedia.org/T348456 (10elukey) Missing steps: - Drop unused labels from the new metrics - https://gerrit.wikimedia.org/r/965178 - Try to fix the deployment of revscoring-drafttopic on ml-serve-... [15:08:22] klausman: I opened the patches for puppet and dep-charts. could you take care of the private repo? I don't really know what we're doing over there :D [15:08:39] private repo is not needed anymore, as Luca and I just discovered [15:08:45] aa ok [15:11:03] ping me when you merge puppet changes so I can poke the puppetmaster [15:13:10] (03CR) 10Ilias Sarantopoulos: "recheck" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965125 (https://phabricator.wikimedia.org/T348265) (owner: 10Ilias Sarantopoulos) [15:13:25] (03CR) 10Ilias Sarantopoulos: "recheck" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/964954 (owner: 10Ilias Sarantopoulos) [15:15:04] klausman: you'll have to merge the puppet patch. I don't have +2 permissions 🙏 [15:15:28] ah, right, so I can do the poking synchronized with it [15:15:43] elukey: do you want to review the two changes Ilias made or do you trust my review? [15:15:55] I am not going to answer this question :) [15:16:18] heh [15:16:20] (please don't ask again, srsly :) [15:16:27] Noted [15:17:44] 10Machine-Learning-Team: Upgrade the readability model server to KServe 0.11.1 - https://phabricator.wikimedia.org/T348664 (10elukey) [15:19:07] isaranto: puppet change is merged, as is the d-c change. Doing deployment in a sec [15:20:19] applying on staging... [15:20:31] (03PS2) 10Elukey: readability: Upgrade to KServe 0.11.1 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/964561 [15:20:48] geat, u rock! [15:20:52] *great [15:21:21] # kubectl get namespaces|grep llm [15:21:24] llm Active 47s [15:21:40] aaand this is the langid patch https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/965189 [15:22:38] serve-codfw deployed as well [15:25:32] isaranto: can you wait before deploying langid? I am testing/sending a patch for kserve 0.11.1 [15:26:00] sure I won't deploy, just creating the patches [15:27:27] (03PS1) 10Elukey: langid: Upgrade to KServe 0.11.1 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965190 [15:27:30] there :) [15:27:36] update kserve -> deploy on namespace -> run load test -> expose to APIGW [15:27:57] 10Machine-Learning-Team, 10SRE, 10decommission-hardware, 10ops-eqiad: decommission ores{1001..1009}.eqiad.wmnet - https://phabricator.wikimedia.org/T348144 (10Jclark-ctr) 05Open→03Resolved [15:28:18] (03CR) 10Ilias Sarantopoulos: [C: 03+1] langid: Upgrade to KServe 0.11.1 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965190 (owner: 10Elukey) [15:28:23] 10Machine-Learning-Team, 10SRE, 10decommission-hardware, 10ops-eqiad: decommission ores{1001..1009}.eqiad.wmnet - https://phabricator.wikimedia.org/T348144 (10Jclark-ctr) a:03Jclark-ctr [15:28:44] I just wanted to create all the paches at once :) I created the APIGW one as well [15:29:35] yes yes :) [15:32:07] (03CR) 10CI reject: [V: 04-1] langid: Upgrade to KServe 0.11.1 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965190 (owner: 10Elukey) [15:32:22] 10Lift-Wing, 10Machine-Learning-Team, 10I18n, 10NewFunctionality-Worktype, 10Patch-For-Review: Create a language detection service in LiftWing - https://phabricator.wikimedia.org/T340507 (10isarantopoulos) We've started the procedure to expose the service. These are the following steps: - Update kse... [15:34:04] lol, weird CI message [15:36:16] elukey: I'd also like to fix the model server so it can run locally before we ship it [15:36:46] (03CR) 10Elukey: "recheck" [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965190 (owner: 10Elukey) [15:37:03] (03CR) 10Ilias Sarantopoulos: [V: 03+2 C: 03+2] revscoring: fix mp [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965125 (https://phabricator.wikimedia.org/T348265) (owner: 10Ilias Sarantopoulos) [15:37:35] 10Machine-Learning-Team: Upgrade the readability model server to KServe 0.11.1 - https://phabricator.wikimedia.org/T348664 (10elukey) Research bumped the numpy deps to a version compatible with KServe 0.11.1 in https://gitlab.wikimedia.org/trokhymovych/readability-liftwing/-/merge_requests/2 [15:37:36] (03CR) 10Ilias Sarantopoulos: [C: 03+2] revscoring: bump scipy version [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/964954 (owner: 10Ilias Sarantopoulos) [15:38:47] (03CR) 10Ilias Sarantopoulos: [C: 03+1] readability: Upgrade to KServe 0.11.1 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/964561 (owner: 10Elukey) [15:40:10] (03PS3) 10Ilias Sarantopoulos: revscoring: bump scipy version [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/964954 [15:40:20] (03CR) 10Ilias Sarantopoulos: [V: 03+2] revscoring: bump scipy version [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/964954 (owner: 10Ilias Sarantopoulos) [15:40:44] well many changes, tomorrow is test and deploy day :) [15:41:27] and most of them are not accounted in our goals :) [15:41:40] (this is what I was saying, too much unscheduled but necessary work) [15:41:50] anyway, going afk for a run, have a nice rest of the day folks! [15:42:37] o/ enjoy your run! [15:43:08] and report your running pace, total time VO2max etc on a phab task :P [15:43:09] (03PS1) 10Elukey: llm: Upgrade to KServe 0.11.1 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965192 [15:48:26] (03PS2) 10Ilias Sarantopoulos: langid: Upgrade to KServe 0.11.1 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965190 (owner: 10Elukey) [15:48:42] (03PS2) 10Ilias Sarantopoulos: llm: Upgrade to KServe 0.11.1 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965192 (owner: 10Elukey) [15:53:14] (03CR) 10CI reject: [V: 04-1] langid: Upgrade to KServe 0.11.1 [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/965190 (owner: 10Elukey) [16:19:20] heading out now, seeyall tomorrow [16:21:35] I'm heading out as well! Will continue to try to fix langid tomorrow [16:43:45] alright night all! [16:46:14] 10Machine-Learning-Team, 10Add-Link, 10Chinese-Sites, 10Growth-Team (Sprint 0 (ending Oct 16, 2023)), 10User-notice: Deploy "add a link" to 14th round of wikis - https://phabricator.wikimedia.org/T308139 (10Sgs) 05In progress→03Open [16:46:53] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Sprint 0 (ending Oct 16, 2023)), 10User-notice: Deploy "add a link" to 15th round of wikis - https://phabricator.wikimedia.org/T308141 (10Sgs) [16:47:01] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team (Sprint 0 (ending Oct 16, 2023)), 10User-notice: Deploy "add a link" to 15th round of wikis - https://phabricator.wikimedia.org/T308141 (10Sgs) 05Open→03In progress