[02:27:42] (SystemdUnitFailed) resolved: cleanup_tmpdumps.service Failed on dumpsdata1006:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:18:41] hello folks! [06:19:23] I noticed that we have some SLO dashboards like https://grafana.wikimedia.org/d/slo-WDQS/wdqs-slo-s?orgId=1 [06:19:54] now they have a fixed time window, a quarter shifted earlier by a month [06:20:12] (so we change the time window once a quarter for all dashboards etc.) [07:25:34] 10Data-Platform-SRE, 10Elasticsearch, 10Discovery-Search (Current work), 10Patch-For-Review: Change partitioning scheme for elasticsearch from RAID to JBOD - https://phabricator.wikimedia.org/T231010 (10dcausse) I think we should test what elastic is doing with one of its datapath broken, the ideal failure... [09:12:38] Starting build #27 for job wikimedia-event-utilities-maven-release-docker [09:15:58] Project wikimedia-event-utilities-maven-release-docker build #27: 09SUCCESS in 3 min 19 sec: https://integration.wikimedia.org/ci/job/wikimedia-event-utilities-maven-release-docker/27/ [09:25:20] 10Data-Platform-SRE, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install wdqs102[0-4] - https://phabricator.wikimedia.org/T342749 (10VRiley-WMF) @RKemper @bking Hey there! I was wondering if you please verify the racking proposal. It is listed to have the racking locations... wdqs1006 (row A) to be repl... [09:32:00] 10Data-Platform-SRE, 10SRE, 10SRE-Access-Requests, 10Patch-For-Review: Requesting Creation of a new POSIX group and system user for the Analytics WMDE team. - https://phabricator.wikimedia.org/T345726 (10Stevemunene) Hi @odimitrijevic , Requesting approval for adding the `analytics-wmde` user to analtyics-... [09:32:18] elukey: I know that rkemper has been working on this recently under T324811 - I think the latest work might be this one. https://grafana.wikimedia.org/dashboard/snapshot/EeCR6g9VketzwysFo4V4XMG9Q4Sb0viR?orgId=1 [09:32:19] T324811: Create WDQS Lag SLO dashboard with Grizzly && documentation - https://phabricator.wikimedia.org/T324811 [09:35:04] ack! The one generated by Grizzly should be https://grafana.wikimedia.org/d/slo-WDQS/wdqs-slo-s?orgId=1, really weird that we have two [09:35:11] maybe ryankemper can clarify later on [09:36:11] Yes, I don't know if we should deprecate the older one or not. I can see about updating the grizzly definition to shift the time window, if that helps in the meantime. [09:37:49] The non Grizzly dashboard predates the Grizzly one (I'm not even sure we had Grizzly standardized at that time). We should probably drop the non Grizzly dashboard now. [09:38:45] Also maybe the new dashboard should be in the 'SLOs' folder in Grafana? I haven't done much work with grizzly yet, but I'd imagine that locating it within the menus is something that it does. [09:39:45] Oh yes I see. The one elukey lined to is completely non-grizzy, so it wouldn't be a patch, just an edit and save. Correct? [09:39:55] The new one is in the SLO folder as far as I can see [09:40:48] I'm still not entirely confident that we are measuring what I think we should be measuring, or how to interpret the data :/ [09:40:49] Oh, maybe I'm confusing new with old then. [09:41:11] Prometheus is confusing to me, and the way we present SLOs is also confusing to me :/ [09:43:43] Then there is this dashboard https://grafana.wikimedia.org/d/yCBd7Tdnk/wdqs-wcqs-lag-slo?forceLogin&orgId=1 which is the only one linked to from: https://wikitech.wikimedia.org/wiki/SLO/WDQS [09:45:15] btullis: would you have time to review all this with ryankemper? No emergency at all, but a new set of eyes might be helpful! [09:45:31] Yep, will do. [09:48:11] thanks! [09:49:54] 10Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service: Move whitelist.txt from WDQS deploy repo into puppet and rename it to "allow list" - https://phabricator.wikimedia.org/T343856 (10Gehel) [09:49:57] 10Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service, 10Discovery-Search (Current work), 10Patch-For-Review: Rename usages of whitelist to allowlist in query service rdf repo - https://phabricator.wikimedia.org/T344284 (10Gehel) 05Open→03Resolved [09:56:39] gehel: what confuses you about the way we present SLOs? [09:56:55] (trying to get what you mean) [09:57:18] so far I've added several dashboards and it seems more-or-less a straightforward process [09:58:16] btullis: the dashboard that I pointed out is the one generated via the grizzly config in the repo (and the one in the slo folder etc..) [09:58:29] it doesn't show data for the budget so it may need some tuning [09:58:48] (there is a grafana-grizzly repo where we store the configs etc.. in gerrit) [10:09:27] elukey: Yep, got it now. Thanks. I will check with rkempe.r to see what we need to update and where we can add clarity, remove duplication, add links etc. [10:16:43] 10Data-Platform-SRE, 10Dumps-Generation, 10cloud-services-team: clouddumps100[12] puppet alert: "Puppet performing a change on every puppet run" - https://phabricator.wikimedia.org/T346165 (10taavi) That was fixed, but now there seem to be some permissions that are getting overwritten: ` Notice: /Stage[main]... [10:26:39] btullis: thanks! [10:28:14] 10Data-Engineering, 10Data-Platform-SRE, 10Data Engineering and Event Platform Team, 10Data Pipelines, 10Patch-For-Review: Enable libmamba by default for conda environment solving - https://phabricator.wikimedia.org/T337258 (10BTullis) I've tried several more builds locally and I feel that I'm getting cl... [10:42:26] 10Data-Platform-SRE, 10SRE, 10ops-eqiad: Q1:rack/setup/install wdqs101[789] - https://phabricator.wikimedia.org/T342660 (10VRiley-WMF) wdqs1017 D 2. U 38. CableID 230304500154. Port 24 wdqs1018 E 2. U 40. CableID 230304500260. Port 32 wdqs1019 F 2. U 39. CableID 230304500198 Port 32 [10:47:40] 10Data-Engineering, 10Data-Platform-SRE, 10Data Engineering and Event Platform Team, 10Data Pipelines, 10Patch-For-Review: Enable libmamba by default for conda environment solving - https://phabricator.wikimedia.org/T337258 (10BTullis) When I activate the build environment `/opt/miniconda` I get the foll... [11:00:55] 10Data-Engineering, 10Data-Platform-SRE, 10Data Engineering and Event Platform Team, 10Data Pipelines, 10Patch-For-Review: Enable libmamba by default for conda environment solving - https://phabricator.wikimedia.org/T337258 (10BTullis) I was going to try a build with the order of the channels reversed in... [11:07:01] 10Data-Platform-SRE, 10Patch-For-Review: Bring druid10[09-11] into service - https://phabricator.wikimedia.org/T336042 (10Stevemunene) [11:12:53] 10Data-Engineering, 10Data Engineering and Event Platform Team (Sprint 2), 10Event-Platform, 10Patch-For-Review: [BUG] eventutilites-python: fix type checking CI job - https://phabricator.wikimedia.org/T346085 (10CodeReviewBot) gmodena merged https://gitlab.wikimedia.org/repos/data-engineering/eventutiliti... [11:14:20] 10Data-Platform-SRE: Users are redirected to the wrong screen on logout and from certain urls. - https://phabricator.wikimedia.org/T347149 (10Stevemunene) [11:14:50] 10Data-Engineering, 10Data Engineering and Event Platform Team (Sprint 2), 10Event-Platform: Document the onboarding journey on Event Platfrom - https://phabricator.wikimedia.org/T345193 (10CodeReviewBot) gmodena merged https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_request... [11:15:03] 10Data-Engineering, 10Data Engineering and Event Platform Team (Sprint 2), 10Event-Platform: eventutilities-python: cookicutter template example should be updated - https://phabricator.wikimedia.org/T345390 (10CodeReviewBot) gmodena merged https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-py... [11:17:53] 10Data-Platform-SRE, 10CAS-SSO, 10Infrastructure-Foundations: Switch DataHub authentication to OIDC - https://phabricator.wikimedia.org/T305874 (10Stevemunene) 05Open→03Resolved Closing this task as resolves as we did meet our acceptance criteria and tracking the login/logout user experience on T347149... [11:17:57] 10Data-Platform-SRE: Datahub user records are not being created after login - https://phabricator.wikimedia.org/T327884 (10Stevemunene) [11:18:13] 10Data-Engineering, 10Data-Platform-SRE, 10Epic: Data Catalog MVP - https://phabricator.wikimedia.org/T299910 (10Stevemunene) [11:21:46] 10Data-Engineering, 10Data-Platform-SRE, 10Data Engineering and Event Platform Team, 10Data Pipelines, 10Patch-For-Review: Enable libmamba by default for conda environment solving - https://phabricator.wikimedia.org/T337258 (10BTullis) Remove conda-forge from the `conda-environment.lock.yml` manually. `... [11:33:45] 10Data-Platform-SRE: Datahub user records are not being created after login - https://phabricator.wikimedia.org/T327884 (10Stevemunene) 05Open→03Resolved After the switch to OIDC we can confirm that new users, can login and their records are created. [11:39:23] 10Data-Engineering, 10Data-Platform-SRE, 10Data Engineering and Event Platform Team, 10Data Pipelines, 10Patch-For-Review: Enable libmamba by default for conda environment solving - https://phabricator.wikimedia.org/T337258 (10BTullis) Created the environment. ` (base) root@01237b0b7310:/srv/conda-analyt... [11:43:02] Is `kafka-test-eqiad` safe to rolling-restart? I'm guessing it is, but I might as well ask beforehand [11:48:11] I *think* so :-) We have little bits of datahub's staging environment hooked up to it, but not active ingestion. There is also a staging deployment of the mw-page-content-change-enrich pipeline hooked up to it, but that's probably ok. You might want to check with gmodena - just in case. [11:48:16] https://codesearch.wmcloud.org/search/?q=kafka-test&files=&excludeFiles=&repos=#operations/deployment-charts [11:48:59] 10Data-Engineering, 10Data-Platform-SRE, 10Data Engineering and Event Platform Team, 10Data Pipelines, 10Patch-For-Review: Enable libmamba by default for conda environment solving - https://phabricator.wikimedia.org/T337258 (10BTullis) The output from `conda info` in this newly activated environment is a... [11:49:59] btullis brouberol it's ok to restart. mw-page-content-change-enrich is hooked to it for testing purposes, but off the critical path. No disruption whatsoever if the broker goes down [11:50:12] and thanks for checking - much appreciated! [11:51:00] gmodena thanks! I have an improved kafka rolling restart cookbook that I'll be testing on it [11:54:45] 10Data-Engineering, 10Data-Platform-SRE, 10Data Engineering and Event Platform Team, 10Data Pipelines, 10Patch-For-Review: Enable libmamba by default for conda environment solving - https://phabricator.wikimedia.org/T337258 (10BTullis) Woah there! Earliest recorded occurrence of the libarchive error, run... [11:55:26] brouberol btullis speaking of kafka: would it be an option for me to have access (read only) to broker hosts? I would like to review some configs (partitioning), and having to roll my own kafka CLIs on stat hosts is a pain [11:59:46] gmodena: I am reminded of this ticket, which I see got closed, but I'm not sure that is was a satisfactory result. T344989 [11:59:47] T344989: Unable to use kafka-topic.sh - Topic authorization failed - https://phabricator.wikimedia.org/T344989 [12:00:54] gmodena: Relevant slack thread as well. https://wikimedia.slack.com/archives/C02291Z9YQY/p1692019188700089 [12:01:30] btullis ack. I'd like to sue the wrapper mentioned in that phab, but I think my ssh pub key is not deployed to kafka hosts? [12:02:01] btullis or I was hitting the wrong broker [12:02:40] That's right, it seems that the way the ACLs are set up at the moment, you can only get information about the topics if you are on the brokers themselves, which you're currently unable to do. [12:02:48] kafka-jumbo1001.eqiad.wmnet does not recognize the I'm offering, and ask for password auth [12:03:57] So I think we should probably try to restate the problem that you and Peter were both facing, which is needing to /get/ information on topics (without using a long-winded workaround). [12:04:32] btullis that's exactly it, just stated better than I did :) [12:04:47] btullis would you like the to open a phab for it? [12:05:40] Yes, will do. I think I'm tempted to reopen the one above and try to restate the problem. [12:10:16] btullis err... I forgot a *me* in the previous question. I can help with opening a phab if helpful, or +1 to re-opening and restating the problem [12:11:02] Aha :-) I'm already on it. Will subscribe you and you can modify as you see fit. [12:16:09] gmodena: By the way, does datahub give you the information you need? e.g. https://datahub.wikimedia.org/dataset/urn:li:dataset:(urn:li:dataPlatform:kafka,eqiad.mediawiki_page_content_change_enrichment.error,PROD)/Properties?is_lineage_mode=false [12:17:29] It has the number of partitions and replication factor per topic. It might get other properties if they are present on a per-topic basis. [12:23:36] btullis datahub is a good start. thanks for the pointer. [12:23:40] re new kafka rolling restart cookbook. I've run it successfully on kafka-test-eqiad. It has restarted the 5 brokers in 25 min (note: we sleep 5 min between each restart). The previous version would sleep 900+300s = 20 min between each restart, and didn't 100% guarantee that the previous broker was back to full sync before moving on to the next. [12:23:40] Progress! [12:24:35] btullis sometimes i need more granual info, like offsets and logsize. Nothing too disruptive for my workflow, because I can gather the info through other sources. But it'd be nice to have a single tool for that [12:24:52] brouberol nice! [12:24:54] brouberol: Great! Did you add the 5 minutes' sleep on advice, or was this your own recommendation? [12:25:10] gmodena: Thanks. I'll add this info to the ticket. [12:26:05] A bit of both. The previous version had 2 sleeps: wait 900s for the broker to catch up and 300s between each restart, to leave breathing room for producers/consumers. I mainly kept the second one [12:27:10] also, it aligned with how I had designed kafka rolling restart tooling in the past: we usually prefer to make them a bit artificially slow. Note that we can tweak this sleep time with --grace-sleep as I've converted the cookbook to the SREbatch base class, so we get that for free [12:39:48] 10Data-Engineering, 10Data-Platform-SRE, 10Event-Platform, 10Patch-For-Review: Use the replication status of assigned partitions as a gate between kafka broker rolling-restarts - https://phabricator.wikimedia.org/T346741 (10brouberol) Running `test-cookbooks sre.kafka.roll-restart-reboot-brokers --alias ka... [12:42:01] 10Data-Platform-SRE, 10Discovery-Search (Current work): Unable to use kafka-topic.sh - Topic authorization failed - https://phabricator.wikimedia.org/T344989 (10BTullis) 05Resolved→03Open I'm going to be bold and reopen this ticket, because I think that the intention of the original request has been slight... [12:42:53] 10Data-Platform-SRE, 10Discovery-Search (Current work): Facilitate users to query kafka topic metadata - https://phabricator.wikimedia.org/T344989 (10BTullis) a:05elukey→03None [12:47:36] brouberol: nice work! Note that the sleep values were all highly configurable, not fixed, so it was not scrictly neeed to wait all that time. Your version is definitely better though, and more secure, so thanks for working on it :) [12:53:42] (SystemdUnitFailed) firing: hadoop-yarn-nodemanager.service Failed on an-worker1153:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:54:50] PROBLEM - Check systemd state on an-worker1153 is CRITICAL: CRITICAL - degraded: The following units failed: hadoop-yarn-nodemanager.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [12:54:57] PROBLEM - Hadoop NodeManager on an-worker1153 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [12:56:19] elukey: thank you! [12:57:16] (03CR) 10Joal: "Actually this patch was for oozie - Will abandon it." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/681682 (https://phabricator.wikimedia.org/T280649) (owner: 10Joal) [12:57:38] (03Abandoned) 10Joal: Cleanup cassandra double loading [analytics/refinery] - 10https://gerrit.wikimedia.org/r/681682 (https://phabricator.wikimedia.org/T280649) (owner: 10Joal) [13:01:17] (03PS1) 10Joal: Remove cassandra from the update-jars script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/960053 [13:02:49] hey mforns - I added you as reviewer of --^ This should be deployed with the change on refinery-source (adding it to the deployment etherpad) [13:20:09] RECOVERY - Check systemd state on an-worker1153 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [13:20:15] RECOVERY - Hadoop NodeManager on an-worker1153 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.yarn.server.nodemanager.NodeManager https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23Yarn_Nodemanager_process [13:23:42] (SystemdUnitFailed) resolved: hadoop-yarn-nodemanager.service Failed on an-worker1153:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:43:04] 10Data-Platform-SRE, 10Data-Services, 10cloud-services-team: move cloudelastic behind cloudlb - https://phabricator.wikimedia.org/T346946 (10taavi) These are [[ https://wikitech.wikimedia.org/wiki/Help:CirrusSearch_elasticsearch_replicas | mirrors of the MediaWiki search indexes ]] for WMCS clients, and I be... [13:53:59] 10Data-Platform-SRE, 10Data-Services, 10cloud-services-team: move cloudelastic behind cloudlb - https://phabricator.wikimedia.org/T346946 (10taavi) [14:08:08] 10Data-Platform-SRE, 10Data-Services, 10cloud-services-team: move cloudelastic behind cloudlb - https://phabricator.wikimedia.org/T346946 (10bking) Thanks @taavi! Adding Search Platform SWEs and SREs so we can discuss this proposal. As far as Data Platform SRE, we are split between Data Engineering and Sea... [14:13:37] 10Data-Engineering, 10Product-Analytics, 10Wmfdata-Python: Improve df_to_remarkup formatting for wmfdata-python - https://phabricator.wikimedia.org/T341589 (10AndrewTavis_WMDE) Hey there, @nshahquinn-wmf! One week on after starting the new computer setup and I've been using this as a means to test it all out... [14:17:11] 10Data-Engineering, 10Data-Platform-SRE, 10Data Engineering and Event Platform Team, 10Data Pipelines, 10Patch-For-Review: Enable libmamba by default for conda environment solving - https://phabricator.wikimedia.org/T337258 (10BTullis) At last, I have a working version without the libarchive error. ` btu... [14:52:08] 10Data-Engineering, 10Data-Platform-SRE, 10Data Engineering and Event Platform Team, 10Discovery-Search (Current work), and 2 others: Rollout Elasticsearch extra plugins package and restart cluster to apply - https://phabricator.wikimedia.org/T344366 (10bking) 05Open→03Resolved [14:52:11] 10Data-Engineering, 10Data Engineering and Event Platform Team, 10Discovery-Search (Current work), 10Event-Platform, 10MW-1.41-notes (1.41.0-wmf.16; 2023-07-04): Add support for redirects in CirrusSearch - https://phabricator.wikimedia.org/T325315 (10bking) [14:53:19] 10Data-Engineering, 10Structured-Data-Backlog: NEW BUG REPORT fiwiki’s section-level image suggestions aren’t generated in production - https://phabricator.wikimedia.org/T343844 (10mfossati) 05Open→03Resolved a:03mfossati - Section topics: ` In [1]: st = spark.read.parquet('/user/analytics-platform-eng/s... [14:59:16] 10Data-Engineering, 10Structured-Data-Backlog: NEW BUG REPORT fiwiki’s section-level image suggestions aren’t generated in production - https://phabricator.wikimedia.org/T343844 (10mfossati) 05Resolved→03Open @xcollazo @mforns , actually it's essential to understand the expected behavior of `DagProperties`... [15:09:28] 10Data-Engineering, 10Data Engineering and Event Platform Team, 10MediaWiki-libs-HTTP, 10Beta-Cluster-reproducible, and 6 others: PHP Warning: curl_multi_remove_handle(): supplied resource is not a valid cURL Multi Handle resource - https://phabricator.wikimedia.org/T288624 (10Lucas_Werkmeister_WMDE) Hm, a... [15:23:35] 10Data-Engineering, 10Structured-Data-Backlog: NEW BUG REPORT fiwiki’s section-level image suggestions aren’t generated in production - https://phabricator.wikimedia.org/T343844 (10Urbanecm_WMF) [15:23:57] 10Data-Engineering, 10Data Engineering and Event Platform Team, 10stewardbots, 10Event-Platform: Frequent `429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange` errors in SULWatcher - https://phabricator.wikimedia.org/T329327 (10Ottomata) [15:24:45] 10Data-Engineering, 10Data Engineering and Event Platform Team, 10EventStreams, 10Pywikibot, 10Event-Platform: Error 429: too many requests for stream.wikimedia.org - https://phabricator.wikimedia.org/T308931 (10Ottomata) [15:25:02] 10Data-Engineering, 10Data Engineering and Event Platform Team, 10EventStreams, 10stewardbots, 10Event-Platform: Frequent `429 Client Error: Too Many Requests for url: https://stream.wikimedia.org/v2/stream/recentchange` errors in SULWatcher - https://phabricator.wikimedia.org/T329327 (10Ottomata) [15:41:38] 10Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products: Windows 11 missing in analytics ? - https://phabricator.wikimedia.org/T346890 (10mforns) Hi all! Adding some thoughts: **Continuous drop of Windows** I think the continuous drop of the Windows operating system (os family) is legitimate, and it... [15:56:39] 10Data-Platform-SRE, 10Wikidata, 10Wikidata-Query-Service, 10serviceops-radar, and 2 others: Migrate the wdqs streaming updater flink jobs to flink-k8s-operator deployment model - https://phabricator.wikimedia.org/T326409 (10bking) [15:57:26] 10Data-Engineering, 10Data-Platform-SRE, 10Discovery-Search (Current work), 10Event-Platform, 10Patch-For-Review: Test common operations in the flink operator/k8s/Flink ZK environment - https://phabricator.wikimedia.org/T342149 (10bking) [16:22:33] (03PS1) 10Sharvaniharan: New Event schema for mobile apps [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/960100 [16:23:39] (03CR) 10CI reject: [V: 04-1] New Event schema for mobile apps [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/960100 (owner: 10Sharvaniharan) [16:23:56] (03CR) 10Mforns: [C: 03+2] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/960053 (owner: 10Joal) [16:24:04] (03CR) 10Mforns: [V: 03+2 C: 03+2] Remove cassandra from the update-jars script [analytics/refinery] - 10https://gerrit.wikimedia.org/r/960053 (owner: 10Joal) [16:26:14] (03PS2) 10Sharvaniharan: New Event schema for mobile apps [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/960100 [16:36:10] 10Data-Engineering, 10Data-Platform-SRE, 10Data Engineering and Event Platform Team, 10Data Pipelines, 10Patch-For-Review: Enable libmamba by default for conda environment solving - https://phabricator.wikimedia.org/T337258 (10BTullis) Great! I've now got the conda environment cloning to work as well. It... [16:44:18] 10Data-Engineering, 10Movement-Insights, 10Product-Analytics, 10Research-Freezer: Investigate relation of UA deprecation to increase in automated traffic and reduction in unique devices - https://phabricator.wikimedia.org/T336715 (10mforns) +1 `spider` [16:45:05] 10Data-Platform-SRE: [DataHub] Users are redirected to the wrong screen on logout and from certain urls. - https://phabricator.wikimedia.org/T347149 (10BTullis) [16:57:57] 10Data-Engineering, 10Data-Platform-SRE, 10Discovery-Search (Current work), 10Event-Platform: Test common operations in the flink operator/k8s/Flink ZK environment - https://phabricator.wikimedia.org/T342149 (10bking) [16:58:30] 10Data-Engineering, 10Data-Platform-SRE, 10Data Engineering and Event Platform Team, 10Data Pipelines, 10Patch-For-Review: Enable libmamba by default for conda environment solving - https://phabricator.wikimedia.org/T337258 (10BTullis) JupyterHub spawns a new environment correctly and reports the correct... [17:00:57] 10Data-Engineering, 10Data-Platform-SRE, 10Discovery-Search (Current work), 10Event-Platform: Test common operations in the flink operator/k8s/Flink ZK environment - https://phabricator.wikimedia.org/T342149 (10bking) [17:10:16] 10Data-Engineering: [EventGate] Failures when getting stream config from MediaWiki API - https://phabricator.wikimedia.org/T286793 (10mforns) a:05mforns→03None [17:19:27] 10Data-Engineering, 10Data-Platform-SRE, 10Data Engineering and Event Platform Team, 10Data Pipelines, 10Patch-For-Review: Enable libmamba by default for conda environment solving - https://phabricator.wikimedia.org/T337258 (10BTullis) I've got a new merge request here for a 0.0.23 release: https://gitla... [17:27:34] 10Data-Engineering, 10Data-Engineering-Wikistats: Add punjabi language in Wikistats - https://phabricator.wikimedia.org/T344572 (10Kuldeepburjbhalaike) Should it be closed now? [17:53:38] 10Data-Platform-SRE, 10Patch-For-Review: Write new partman recipe for cloudelastic - https://phabricator.wikimedia.org/T342463 (10bking) [18:41:38] (03PS3) 10Sharvaniharan: New Event schema for mobile apps [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/960100 [18:45:35] (03PS4) 10Sharvaniharan: New Event schema for mobile apps [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/960100 [18:52:20] (03CR) 10Sharvaniharan: "This is a schema created for recording generic events like clicks, drop-downs, scrolls, checkboxes, impressions etc." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/960100 (owner: 10Sharvaniharan) [18:58:14] (03PS5) 10Sharvaniharan: New Event schema for mobile apps [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/960100 [18:59:01] (03CR) 10CI reject: [V: 04-1] New Event schema for mobile apps [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/960100 (owner: 10Sharvaniharan) [20:05:09] 10Data-Platform-SRE, 10Elasticsearch, 10Discovery-Search (Current work), 10Patch-For-Review: Change partitioning scheme for elasticsearch from RAID to JBOD - https://phabricator.wikimedia.org/T231010 (10bking) Based on team feedback, we are going to go with option 2. [21:07:23] 10Data-Platform-SRE, 10SRE, 10SRE-Access-Requests, 10Patch-For-Review: Requesting Creation of a new POSIX group and system user for the Analytics WMDE team. - https://phabricator.wikimedia.org/T345726 (10odimitrijevic) approved [21:43:35] (03PS1) 10Kimberly Sarabia: Adds new web fragment [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/960141 (https://phabricator.wikimedia.org/T346106) [21:44:13] (03CR) 10CI reject: [V: 04-1] Adds new web fragment [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/960141 (https://phabricator.wikimedia.org/T346106) (owner: 10Kimberly Sarabia) [21:52:35] (03PS2) 10Kimberly Sarabia: Adds new web fragment [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/960141 (https://phabricator.wikimedia.org/T346106) [21:53:06] (03CR) 10CI reject: [V: 04-1] Adds new web fragment [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/960141 (https://phabricator.wikimedia.org/T346106) (owner: 10Kimberly Sarabia)