[05:05:46] Good morning folks! [05:23:49] (03PS4) 10Ilias Sarantopoulos: ores-legacy: rename project to match the agreed name [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/923621 [06:53:59] kalimera! [06:54:25] isaranto: the new bloom pod is crash looping, the knative queue proxy fails to ping 127.0.0.1:8080 [06:56:17] (03CR) 10Elukey: [C: 03+1] ores-legacy: rename project to match the agreed name [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/923621 (owner: 10Ilias Sarantopoulos) [06:56:38] * elukey commutes to the office [07:04:20] o/ ack! [07:28:43] back! [07:29:12] so I manually bumped the limit ranges, I need to check why helmfile didn't do it (there is surely a admin_ng template to fix) [07:36:11] (03CR) 10Ilias Sarantopoulos: [C: 03+2] ores-legacy: rename project to match the agreed name [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/923621 (owner: 10Ilias Sarantopoulos) [07:37:29] elukey: do u want me to do anything on the redis/cache front? [07:38:33] isaranto: still working on the patch, no idea if Wikimedia Enterprise will need it, they seem to be more inclined to use Rever Risk, but haven't got a definitive answer.. will complete the read only part in these days so you can see if it makes sense or not to proceed [07:40:20] ok, thnx. are there instructions on how to access the ores redis cache? all I found is this https://wikitech.wikimedia.org/wiki/ORES/Deployment#Restarting_Redis [07:41:52] isaranto: not that I know, we have different rdbXXXX nodes set in the ores config (for example, score cache, celery queue, etc..) [07:41:58] they are set by puppet [07:42:07] ack [07:42:44] seems like the bloom deployment gets OOM errors. we'll have to increase memory.. [07:42:48] (03Merged) 10jenkins-bot: ores-legacy: rename project to match the agreed name [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/923621 (owner: 10Ilias Sarantopoulos) [07:43:11] where did you see it? [07:50:30] ok so I found the template issue, and of course it was a Luca's PEBCAK https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/923924/ [07:54:28] ``` [07:54:28] Last State: Terminated [07:54:28] Reason: OOMKilled [07:54:28] Exit Code: 137 [07:54:28] Started: Mon, 29 May 2023 07:29:00 +0000 [07:54:29] Finished: Mon, 29 May 2023 07:29:29 +0000 [07:54:29] ``` [07:54:59] is is from kubectl describe etc.. ? [07:55:04] didn't see it, nice [07:55:18] ok so lemme deploy the right limit ranges config, maybe it is my fault [07:55:23] if not, we'll increase [07:55:31] team's PEBCAK [07:55:34] <3 [07:56:13] I dont see it now, now we get CrashLoopBackOff again [07:56:14] cool [07:58:56] ok so now the pod doesn't come up, I recall that I've set higher ranges manually [07:59:04] if I run `get events -n experimental` I see [07:59:15] FailedCreate replicaset/bloom-3b-predictor-default-00001-deployment-74bcf8b7c4 (combined from similar events): Error creating: pods "bloom-3b-predictor-default-00001-deployment-74bcf8b7c4-xv9sm" is forbidden: [maximum cpu usage per Pod is 9, but limit is 11, maximum memory usage per Pod is 8Gi, but limit is 16263413760] [08:00:58] (03PS6) 10Ilias Sarantopoulos: ores-legacy: run tests in ci [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/923599 [08:02:19] (03CR) 10CI reject: [V: 04-1] ores-legacy: run tests in ci [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/923599 (owner: 10Ilias Sarantopoulos) [08:03:25] I need to decrypt the issue [08:14:48] so in theory (IIUC) we set the pod's limit to 11, but 9 is set in limitranges [08:15:36] but it is weird since I don't find 11 [08:16:11] ahhhh wait [08:16:19] the pod is not only the kserve container [08:16:28] we need to take into account the rest [08:18:41] and memory requested seem to be 16G [08:18:46] so I'll bump it to 20G [08:19:49] yeah also, I see ilias' PEBCAK regarding limits and requests compared to limitranges [08:20:22] max pod memory: "8Gi" [08:20:44] while max memory for container is set to ```memory: "14Gi"``` [08:20:52] 🤦‍♂️ [08:24:15] ok modified the limit ranges, in theory the k8s controller should re-create it in a bit [08:25:53] created https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/924047 [08:25:56] let's see if it works [08:26:30] ok the pod is coming up [08:29:03] yeah same error, the kserve container seems not responding [08:30:15] isaranto: at this point we can bump memory for the kserve container, what do you think? [08:31:02] requested 20G [08:31:58] yes, everything seems memory related [08:33:54] I don't know what I was thinking when I set the limitranges [08:34:07] nah it is fine I also review it :) [08:34:07] or perhaps I wasnt thinking ;) [08:34:25] the knative queue proxy says [08:34:26] aggressive probe error (failed 202 times): dial tcp 127.0.0.1:8080: connect: connection refused [08:35:54] I checked on the ml-staging host with nsenter and I see [08:36:03] elukey@ml-staging2002:~$ sudo nsenter -t 2030897 -n curl localhost:8080 -i [08:36:03] HTTP/1.1 200 OK [08:36:03] date: Mon, 29 May 2023 08:35:39 GMT [08:36:03] server: uvicorn [08:36:03] content-length: 18 [08:36:06] content-type: application/json [08:36:08] {"status":"alive"} [08:36:11] that is strange [08:36:45] mmmm [08:37:08] now it is running [08:37:15] ah wow [08:37:41] maybe it just takes time to bootstrap [08:37:42] networking could be misbehaving due to memory issues (?) [08:38:09] nono my suspiction is that the model takes several seconds to start answering to probes correctly [08:39:22] a yes, I saw you deployed the 20G version already [08:39:41] filed https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/924049 [08:39:59] thanks for the great support Luca! [08:40:21] <3 [08:40:33] well thanks for transforming us into llmops, I'll blame you in the future :D [08:40:53] ... [08:41:03] jokes aside, I like a lot to update limit ranges settings only for specific namespaces [08:41:06] it feels cleaner [08:42:12] * elukey takes a break [08:46:55] 10Machine-Learning-Team, 10ORES, 10artificial-intelligence, 10ML-Governance, 10Documentation: Use licenses template on English Wikipedia ORES model cards - https://phabricator.wikimedia.org/T337668 (10kevinbazira) [08:54:43] ok, the new model works "fine" - "fine" because inference times vary from 10s-100 according to the size of the requested output [08:55:45] I'll update the tasks so that we define a set of tests to run as benchmarks on such models [08:55:51] super [09:12:36] 10Machine-Learning-Team, 10Add-Link, 10Growth-Team, 10User-notice: Deploy "add a link" to 18th round of wikis - https://phabricator.wikimedia.org/T308144 (10elukey) a:05kevinbazira→03kostajh [09:13:53] 10Machine-Learning-Team: create checklist before adding models to api gateway/prod - https://phabricator.wikimedia.org/T332711 (10elukey) 05Open→03Resolved a:03elukey [09:13:55] 10Machine-Learning-Team, 10Epic: Lift Wing improvements to get out of MVP state - https://phabricator.wikimedia.org/T333453 (10elukey) [09:14:48] 10Machine-Learning-Team: The nsfw model hangs in predict() after moving to Kserve 0.10 - https://phabricator.wikimedia.org/T331416 (10elukey) @Htriedman any news? Let us know if you need any support etc.. :) [09:18:21] 10Machine-Learning-Team, 10ORES: Generate a top stub pageview using ORES - https://phabricator.wikimedia.org/T138560 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES since it is being deprecated, p... [09:19:29] 10Machine-Learning-Team, 10ORES, 10Wikidata, 10Wikidata-Query-Service: Measure maturity/quality of Wikidata items in a way it can be queried - https://phabricator.wikimedia.org/T166427 (10elukey) [09:19:52] 10Machine-Learning-Team, 10ORES: Request raw features/datasources from ORES - https://phabricator.wikimedia.org/T133585 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES since it is being deprecated... [09:20:09] 10Machine-Learning-Team, 10ORES, 10Wikidata, 10Wikidata-Query-Service: Include ORES predictions in RDF export - https://phabricator.wikimedia.org/T200716 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks relate... [09:20:13] 10Machine-Learning-Team, 10ORES: How far is ORES in training? - https://phabricator.wikimedia.org/T207548 (10elukey) [09:20:17] 10Machine-Learning-Team, 10ORES, 10Growth-Team, 10Growth-Team-Filtering, 10PageTriage: Consolidate articlequality and itemquality models into a "model family" - https://phabricator.wikimedia.org/T206037 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machi... [09:20:29] 10Machine-Learning-Team, 10artificial-intelligence: Politeness as signal/model - https://phabricator.wikimedia.org/T152710 (10elukey) [09:20:33] 10Machine-Learning-Team, 10ORES, 10artificial-intelligence: Address memory usage issues for deploying PCFG-based features - https://phabricator.wikimedia.org/T157041 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old ta... [09:20:37] 10Machine-Learning-Team, 10artificial-intelligence, 10draftquality-modeling, 10editquality-modeling, and 2 others: [Epic] Implement PCFG features for editquality and draftquality - https://phabricator.wikimedia.org/T144636 (10elukey) [09:20:41] 10Machine-Learning-Team: Host open source LLM (bloom, etc.) on Lift Wing - https://phabricator.wikimedia.org/T333861 (10isarantopoulos) Successfully deployed bloom-3b model. It can be queried like this: ` curl https://inference-staging.svc.codfw.wmnet:30443/v1/models/bloom-3b:predict -X POST -i -H "Host: bloom-... [09:20:45] 10Machine-Learning-Team, 10artificial-intelligence, 10Research ideas: [Epic] Paid editing (COI) detection model - https://phabricator.wikimedia.org/T120170 (10elukey) [09:20:49] 10Machine-Learning-Team, 10ORES: Model information UI (graphs and statistics) - https://phabricator.wikimedia.org/T140364 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES since it is being deprecat... [09:20:53] 10Machine-Learning-Team, 10ORES: Deploy Tensorflow models in production - https://phabricator.wikimedia.org/T161383 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES since it is being deprecated, pl... [09:20:57] 10Machine-Learning-Team, 10ORES, 10Epic: Using TensorFlow in ORES - https://phabricator.wikimedia.org/T161375 (10elukey) [09:21:01] 10Machine-Learning-Team, 10ORES: ORES UI could accept more human-readable inputs such as pasted article links - https://phabricator.wikimedia.org/T170168 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to... [09:21:05] 10Machine-Learning-Team, 10ORES: User-facing tool to explain scoring algorithms - https://phabricator.wikimedia.org/T204560 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES since it is being deprec... [09:21:09] 10Machine-Learning-Team, 10ORES: Templatize articlequality makefile - https://phabricator.wikimedia.org/T212533 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES since it is being deprecated, please... [09:21:43] 10Machine-Learning-Team, 10ORES, 10RESTBase, 10RESTBase-API: Set up revscoring entry points in RESTBase - https://phabricator.wikimedia.org/T107196 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to O... [09:21:59] 10Machine-Learning-Team, 10ORES: [Spec] Deployment tests for ORES - https://phabricator.wikimedia.org/T140412 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES since it is being deprecated, please r... [09:22:14] 10Machine-Learning-Team, 10ORES, 10Mobile-Content-Service, 10Product-Infrastructure-Team-Backlog-Deprecated, and 2 others: Add ORES articlequality data to summaries? - https://phabricator.wikimedia.org/T157132 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/... [09:22:47] 10Machine-Learning-Team, 10ORES: Merge ORES precaching with ORESFetchScoreJob - https://phabricator.wikimedia.org/T201868 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES since it is being deprecat... [09:23:05] 10Machine-Learning-Team: Monitoring for top IPs and User-Agents hitting the ORES service - https://phabricator.wikimedia.org/T181542 (10elukey) 05Open→03Resolved a:03elukey There is a dashboard in logstash that the machine learning team created :) [09:23:21] 10Machine-Learning-Team, 10ORES: ORES service should return a readable error when MW API read times out - https://phabricator.wikimedia.org/T179117 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES... [09:23:28] 10ORES, 10Machine-Learning-Team (Active Tasks), 10Patch-For-Review: ORES service erroring, in a way that throws exceptions in Extension:ORES - https://phabricator.wikimedia.org/T179107 (10elukey) [09:23:34] 10Machine-Learning-Team, 10ORES, 10Technical-Debt: ores-wmflabs-deploy and ores production deploy repos should share a common ancestor - https://phabricator.wikimedia.org/T171014 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am c... [09:24:55] 10Machine-Learning-Team, 10ORES: ORES returns 200 for timed out scores - https://phabricator.wikimedia.org/T198819 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES since it is being deprecated, ple... [09:25:05] 10Machine-Learning-Team, 10ORES, 10Scap: scap deploy --service-restart doesn't affect ORES celery - https://phabricator.wikimedia.org/T182912 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES sinc... [09:25:25] 10Machine-Learning-Team, 10ORES: Spike in ORES errors every 24hr - https://phabricator.wikimedia.org/T191864 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES since it is being deprecated, please re... [09:25:35] 10Machine-Learning-Team, 10ORES, 10SRE, 10Scap: Use external dsh group to list pooled ORES nodes - https://phabricator.wikimedia.org/T179501 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES sin... [09:25:47] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10MediaWiki-Vagrant: Can't enable ores role in vagrant - https://phabricator.wikimedia.org/T177555 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related t... [09:25:56] 10Machine-Learning-Team, 10Privacy: Use filesystem group permissions to protect deleted article content on ores wmflabs boxes - https://phabricator.wikimedia.org/T169123 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old... [09:26:05] 10Machine-Learning-Team, 10ORES, 10Performance Issue: Profile ORES code memory use, optimize pre-fork allocation - https://phabricator.wikimedia.org/T182350 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks relat... [09:26:20] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10Performance Issue: Clean up ORES thresholds cache: pre-emptively check before expiry - https://phabricator.wikimedia.org/T182256 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing... [09:26:22] 10Machine-Learning-Team, 10MediaWiki-extensions-ORES, 10Wikimedia-Incident: Rate limit thresholds requests when the service is down - https://phabricator.wikimedia.org/T181567 (10elukey) [09:28:31] 10Machine-Learning-Team, 10ORES: Include monitoring graphs (5 minute wait) in ORES deploy script - https://phabricator.wikimedia.org/T159614 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES since i... [09:28:39] 10Machine-Learning-Team, 10ORES: Refactor ORES to become scikit-learn agnostic (to support TensorFlow) - https://phabricator.wikimedia.org/T161378 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES s... [09:28:41] 10Machine-Learning-Team, 10ORES, 10Epic: Using TensorFlow in ORES - https://phabricator.wikimedia.org/T161375 (10elukey) [09:29:17] 10Machine-Learning-Team, 10ORES: Investigate a Hyperswitch frontend for ORES - https://phabricator.wikimedia.org/T214874 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES since it is being deprecate... [09:29:20] 10Machine-Learning-Team: [Discuss] ORES model development and deployment processes - https://phabricator.wikimedia.org/T216246 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES since it is being depre... [09:29:27] 10Machine-Learning-Team, 10ORES, 10Edit-Review-Improvements, 10Growth-Team, 10Growth-Team-Filtering: Let ERI order based on ORES scores - https://phabricator.wikimedia.org/T163040 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I... [09:29:37] 10Machine-Learning-Team, 10ORES: Code generation should assert configuration basic sanity - https://phabricator.wikimedia.org/T192118 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES since it is be... [09:29:39] 10Machine-Learning-Team, 10ORES: Sort classes when printing model_info - https://phabricator.wikimedia.org/T194221 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closing old tasks related to ORES since it is being deprecated, ple... [09:29:51] 10Machine-Learning-Team, 10ORES, 10Analytics-Radar, 10Data-Engineering-Icebox: Backfill ORES Hadoop scores with historical data - https://phabricator.wikimedia.org/T209737 (10elukey) 05Open→03Declined We are moving to Lift Wing: https://wikitech.wikimedia.org/wiki/Machine_Learning/LiftWing I am closin... [09:29:55] 10Machine-Learning-Team, 10ORES, 10artificial-intelligence, 10Analytics-Radar, and 6 others: [Epic] Make ORES scores for wikidata available as a dump - https://phabricator.wikimedia.org/T209611 (10elukey) [09:30:10] ok enough cleanups for today :D [09:34:02] wow [09:46:13] added the first high level deprecation timeline for ORES https://wikitech.wikimedia.org/wiki/ORES [09:50:25] and added a warning banner in https://www.mediawiki.org/wiki/ORES [09:55:45] 10Machine-Learning-Team, 10ORES, 10Documentation, 10User-AKlapper: Update docs that ORES will be replaced by Lift Wing - https://phabricator.wikimedia.org/T305963 (10elukey) Added a very high level timeline in https://wikitech.wikimedia.org/wiki/ORES, and added a warning in https://www.mediawiki.org/wiki/O... [09:55:57] 10Machine-Learning-Team, 10ORES, 10Documentation, 10User-AKlapper: Update docs that ORES will be replaced by Lift Wing - https://phabricator.wikimedia.org/T305963 (10elukey) a:03elukey [09:59:54] Nice! [10:15:05] 10Lift-Wing, 10Machine-Learning-Team: Move Revert-risk language agnostic model from staging to production - https://phabricator.wikimedia.org/T332998 (10elukey) @achou do we have a model card for this? [10:23:15] 10Lift-Wing, 10Machine-Learning-Team: Move Revert-risk language agnostic model from staging to production - https://phabricator.wikimedia.org/T332998 (10elukey) Created https://api.wikimedia.org/wiki/API_reference/Service/Lift_Wing/Get_reverted_risk_language_agnostic_prediction, we need to add: 1) The return... [10:25:30] lunch, ttl! [10:58:39] * isaranto lunch o clock [12:04:58] 10Lift-Wing, 10Machine-Learning-Team, 10Patch-For-Review: Move Revert-risk multilingual model from staging to production - https://phabricator.wikimedia.org/T333124 (10elukey) Created https://api.wikimedia.org/wiki/API_reference/Service/Lift_Wing/Get_reverted_risk_multilingual_prediction, we need to add: T... [12:14:20] (03PS1) 10Elukey: Add .gitignore [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/924076 [12:31:56] (03CR) 10Kevin Bazira: [C: 03+1] Add .gitignore [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/924076 (owner: 10Elukey) [13:34:06] (03CR) 10Ilias Sarantopoulos: [C: 03+1] "I usually prefer not to have a .gitignore checked in vcs (git)." [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/924076 (owner: 10Elukey) [13:54:05] elukey: I put +1 but it was more of a neutral one :) [13:55:53] sure sure we can discuss, I think that findind a reasonable .gitignore common for all shouldn't be a big deal [13:55:58] so people can update it etc.. [13:56:10] is there a use case that it could impact your workflow? (to understand) [14:08:12] Yes usually I have files and directories from stuff that I try (usually models and data) for which I don't check in VCS within a branch. [14:08:26] But go ahead I'll figure out another workaround [14:09:31] * isaranto afk for 30' [14:09:54] 10Machine-Learning-Team, 10ORES, 10artificial-intelligence, 10ML-Governance, 10Documentation: Use licenses template on English Wikipedia ORES model cards - https://phabricator.wikimedia.org/T337668 (10kevinbazira) The licenses transclusion template has been added to 5/6 English Wikipedia ORES model cards... [14:11:07] 10Machine-Learning-Team, 10ORES, 10artificial-intelligence, 10ML-Governance, 10Documentation: Use licenses template on English Wikipedia ORES model cards - https://phabricator.wikimedia.org/T337668 (10kevinbazira) [14:12:17] isaranto: nah let's work on something common, I think it is ok to add your things to the common .gitignore [14:12:32] worst that happens is that they don't do anything in other's envs [15:14:18] whatever suits the rest of the team [15:14:59] I'll probably find a workaround, as I change gitignore much too often. e.g. this is my .giignore for inferenceservices -> https://phabricator.wikimedia.org/P48621 [15:15:43] a third option is adding a user wide gitingore for every git repo on the system [15:16:37] going afk for today folks, let's decide tomorrow :) [15:16:37] o/ [15:33:21] good evening! o/ [16:10:10] (03CR) 10Ilias Sarantopoulos: [C: 03+1] Add .gitignore (031 comment) [machinelearning/liftwing/inference-services] - 10https://gerrit.wikimedia.org/r/924076 (owner: 10Elukey) [16:11:07] ok, yeah turns out last time I took a gitignore out of the github proposed ones , so pretty standard https://github.com/github/gitignore/blob/main/Python.gitignore [16:17:00] going afk o/