[00:01:31] 10Quarry: Remove redis - https://phabricator.wikimedia.org/T360584#9647966 (10Frostly) Maybe consider https://www.dragonflydb.io/replace-redis or https://docs.keydb.dev/ [00:17:06] (03PS6) 10Santiago Faci: Adding a new contextual attribute: performer.active_browsing_session_token [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1008541 (https://phabricator.wikimedia.org/T358758) [00:17:06] (KafkaReplicationFactorTooLow) firing: (429) Kafka topic PlatformEvent_v1 replication factor is too low on jumbo-eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Increase_a_topic's_replication_factor - https://alerts.wikimedia.org/?q=alertname%3DKafkaReplicationFactorTooLow [00:18:19] (03CR) 10Santiago Faci: "Thanks Sam! I have fixed all the issues." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1008541 (https://phabricator.wikimedia.org/T358758) (owner: 10Santiago Faci) [00:22:09] (KafkaReplicationFactorTooLow) resolved: (429) Kafka topic PlatformEvent_v1 replication factor is too low on jumbo-eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Increase_a_topic's_replication_factor - https://alerts.wikimedia.org/?q=alertname%3DKafkaReplicationFactorTooLow [00:24:32] (03CR) 10Shay Nowick: [C:03+1] "LGTM" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1013142 (https://phabricator.wikimedia.org/T358299) (owner: 10Cooltey) [00:52:33] (03CR) 10Aqu: "It appears to replace the Gobblin job "webrequest_test.pull" (at first). So, I would add the test suffix to the job name: webrequest_front" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/983926 (https://phabricator.wikimedia.org/T314956) (owner: 10Ottomata) [00:56:56] (03CR) 10Aqu: "Answering myself: you mean `webrequest_source`, which is now dropped." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/983926 (https://phabricator.wikimedia.org/T314956) (owner: 10Ottomata) [00:59:44] (03PS2) 10Aqu: Add webrequest_frontend raw schema. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1012656 (https://phabricator.wikimedia.org/T314956) (owner: 10Gmodena) [00:59:45] (03CR) 10Aqu: "LGTM" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1012656 (https://phabricator.wikimedia.org/T314956) (owner: 10Gmodena) [02:00:23] 10Data-Engineering (Sprint 9): Airflow mapped tasks UI & metrics - https://phabricator.wikimedia.org/T357430#9648099 (10Ahoelzl) a:03Antoine_Quhen [02:03:18] 10Data-Engineering (Sprint 9): [Maintenance] Migrate ReportUpdater browser queries to Airflow - https://phabricator.wikimedia.org/T354552#9648101 (10Ahoelzl) a:05Snwachukwu→03JAllemandou [07:51:35] 06Data-Engineering: an-worker1168 in a weird statue, possiblye due to I/O errors - https://phabricator.wikimedia.org/T360594 (10akosiaris) 03NEW [07:54:06] 06Data-Engineering: an-worker1168 in a weird statue, possiblye due to I/O errors - https://phabricator.wikimedia.org/T360594#9648228 (10akosiaris) [08:02:05] 06Data-Engineering: an-worker1168 in a weird statue, possiblye due to I/O errors - https://phabricator.wikimedia.org/T360594#9648231 (10akosiaris) Related alerts in alerts.wikimedia.org have been silenced from 30 days (chosen arbitrarily) with a comment pointing to this task. [09:09:03] (GobblinKafkaRecordsExtractedNotEqualRecordsExpected) firing: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [09:09:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.mediawiki.cirrussearch.page_rerender.v1&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [09:14:54] 06Data-Engineering, 06serviceops: kafka-main certificates expiring on 2024-04-04 - https://phabricator.wikimedia.org/T360598 (10akosiaris) 03NEW [09:16:38] 06Data-Engineering, 06serviceops: kafka-main certificates expiring on 2024-04-04 - https://phabricator.wikimedia.org/T360598#9648430 (10akosiaris) p:05Triage→03High Adding @brouberol as they probably have way more experience than serviceops on refreshing kafka certificates than anyone in #serviceops [09:17:39] 06Data-Engineering, 06serviceops: kafka-main certificates expiring on 2024-04-04 - https://phabricator.wikimedia.org/T360598#9648445 (10brouberol) Let me have a look at how these certificates are generated. I'm thinking we should renew them and trigger a rolling-restart of the cluster. [09:20:48] (03PS7) 10Phuedx: Adding a new contextual attribute: performer.active_browsing_session_token [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1008541 (https://phabricator.wikimedia.org/T358758) (owner: 10Santiago Faci) [09:21:29] Ping btullis or brouberol for a gerrit repo change - https://phabricator.wikimedia.org/T357697 [09:21:48] (03CR) 10Phuedx: "PS7 wraps the description of performer.active_browsing_session_token to 75 characters." [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1008541 (https://phabricator.wikimedia.org/T358758) (owner: 10Santiago Faci) [09:22:11] (03CR) 10Phuedx: [C:03+2] Adding a new contextual attribute: performer.active_browsing_session_token [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1008541 (https://phabricator.wikimedia.org/T358758) (owner: 10Santiago Faci) [09:22:58] joal: looking [09:23:53] I don't know how to do this from top of mind, and I have 7 minutes before a 1/1, I'm punting this for later [09:24:07] ack, thanks brouberol [09:24:35] (03Merged) 10jenkins-bot: Adding a new contextual attribute: performer.active_browsing_session_token [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/1008541 (https://phabricator.wikimedia.org/T358758) (owner: 10Santiago Faci) [09:24:58] 06Data-Engineering, 10Metrics Platform Backlog, 10Data Products (Data Products Sprint 10): [SPIKE] Draft of Mediawiki extension proposal for Metrics Platform Instrumentation (& Experimentation) - https://phabricator.wikimedia.org/T355599#9648506 (10phuedx) Per yesterday's Data Products Standup meeting, @Virg... [09:25:36] 06Data-Engineering, 06serviceops: kafka-main certificates expiring on 2024-04-04 - https://phabricator.wikimedia.org/T360598#9648509 (10brouberol) ` brouberol@kafka-main2001:~$ echo y | openssl s_client -connect $(hostname -f):9093 | openssl x509 -issuer -nout x509: Unrecognized flag nout x509: Use -help for... [09:31:56] 06Data-Engineering, 06serviceops: kafka-main certificates expiring on 2024-04-04 - https://phabricator.wikimedia.org/T360598#9648523 (10akosiaris) >>! In T360598#9648509, @brouberol wrote: > ` > brouberol@kafka-main2001:~$ echo y | openssl s_client -connect $(hostname -f):9093 | openssl x509 -issuer -nout > x... [09:33:29] 06Data-Engineering, 06serviceops: kafka-main certificates expiring on 2024-04-04 - https://phabricator.wikimedia.org/T360598#9648526 (10akosiaris) So, since I 've never done this before (that I remember of), double check me on this please. Is it just enough to issue ` sudo cookbook sre.kafka.roll-restart-reb... [09:37:14] 06Data-Engineering, 06serviceops: kafka-main certificates expiring on 2024-04-04 - https://phabricator.wikimedia.org/T360598#9648538 (10MoritzMuehlenhoff) Luca migrated kafka/main to the PKI in https://phabricator.wikimedia.org/T319372 and he left a comment to that regard on the task: > What is it going to ch... [09:51:15] 06Data-Engineering, 06serviceops: kafka-main certificates expiring on 2024-04-04 - https://phabricator.wikimedia.org/T360598#9648584 (10brouberol) >>! In T360598#9648526, @akosiaris wrote: > So, since I 've never done this before (that I remember of), double check me on this please. Is it just enough to issue... [10:09:04] (GobblinKafkaRecordsExtractedNotEqualRecordsExpected) firing: (2) Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [10:09:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.mediawiki.cirrussearch.page_rerender.v1&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [10:25:02] joal: re gerrit archiving. For the first repo to archive (analytics/wmde/WD), do I only need to archive the root repo or any repo under it as well? (analytics/wmde/WD/*) [10:25:45] Ah - I don't know :S [10:26:40] brouberol: let me ping someone in WMDE [10:27:18] tx [10:33:34] brouberol: let's archive all of the subprojects please [10:33:48] on it! [10:34:11] thanks a lot! [10:38:55] all done :) [10:42:20] awesome - May I let you comment on the task brouberol? [10:42:44] already done! [10:42:56] \o/ [10:53:01] 10Quarry, 10ChangeProp, 06Commons, 10GitLab, and 7 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9648786 (10taavi) [11:09:04] (GobblinKafkaRecordsExtractedNotEqualRecordsExpected) firing: (2) Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [11:09:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.mediawiki.cirrussearch.page_rerender.v1&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [11:12:54] 06Data-Engineering: an-worker1168 in a weird statue, possibly due to I/O errors - https://phabricator.wikimedia.org/T360594#9648860 (10akosiaris) [11:48:05] (KafkaReplicationFactorTooLow) firing: (4) Kafka topic eqiad.change-prop.retry.mediawiki.job.clearUserWatchlist replication factor is too low on main-eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Increase_a_topic's_replication_factor - https://alerts.wikimedia.org/?q=alertname%3DKafkaReplicationFactorTooLow [11:53:06] (KafkaReplicationFactorTooLow) resolved: (87) Kafka topic change-prop.retry.mediawiki.revision_create replication factor is too low on main-eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Increase_a_topic's_replication_factor - https://alerts.wikimedia.org/?q=alertname%3DKafkaReplicationFactorTooLow [12:08:22] (KafkaReplicationFactorTooLow) firing: (617) Kafka topic change-prop.retry.mediawiki.page_restore replication factor is too low on main-eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Increase_a_topic's_replication_factor - https://alerts.wikimedia.org/?q=alertname%3DKafkaReplicationFactorTooLow [12:09:04] (GobblinKafkaRecordsExtractedNotEqualRecordsExpected) resolved: Gobblin job event_default ingested an unexpected number of records for a Kafka topic partition. ... [12:09:04] - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Gobblin - https://grafana.wikimedia.org/d/pAQaJwEnk/gobblin?orgId=1&var-gobblin_job_name=event_default&var-kafka_topic=eqiad.mediawiki.cirrussearch.page_rerender.v1&viewPanel=4 - https://alerts.wikimedia.org/?q=alertname%3DGobblinKafkaRecordsExtractedNotEqualRecordsExpected [12:13:22] (KafkaReplicationFactorTooLow) resolved: (530) Kafka topic change-prop.retry.mediawiki.page_restore replication factor is too low on main-eqiad - https://wikitech.wikimedia.org/wiki/Kafka/Administration#Increase_a_topic's_replication_factor - https://alerts.wikimedia.org/?q=alertname%3DKafkaReplicationFactorTooLow [13:10:54] 10Data-Engineering (Sprint 9), 10ChangeProp, 10observability, 10service-runner, 10Event-Platform: Upgrade prom-client in NodeJS service-runner and enable collectDefaultMetrics - https://phabricator.wikimedia.org/T350180#9648912 (10gmodena) > See https://github.com/wikimedia/service-runner/commit/b9c98eab... [13:16:00] 10Quarry, 10ChangeProp, 06Commons, 10GitLab, and 7 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9648979 (10Reedy) https://github.com/Snapchat/KeyDB already existed as a fork. https://github.com/Snapchat/KeyDB/issues/798 was filed... [13:20:49] 06Data-Engineering, 06Data-Platform-SRE, 06serviceops: kafka-main certificates expiring on 2024-04-04 - https://phabricator.wikimedia.org/T360598#9649020 (10lbowmaker) [13:24:55] 06Data-Engineering, 06Data-Platform-SRE, 06serviceops: 14kafka-main certificates expiring on 2024-04-04 - 14https://phabricator.wikimedia.org/T360598#9649069 (10akosiaris) 05Open→03Resolved a:03akosiaris 14Alerts gone, I 'll resolve this. As a note to anyone seeing this in the future, it's `kafka... [13:26:23] (03PS3) 10Joal: Update ProduceCanaryEvents job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1011354 (https://phabricator.wikimedia.org/T341229) [13:28:19] (03PS4) 10Joal: Update ProduceCanaryEvents job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1011354 (https://phabricator.wikimedia.org/T341229) [13:28:47] Starting build #138 for job analytics-refinery-maven-release-docker [13:35:58] (03CR) 10Joal: [C:03+2] "Merging for deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1011354 (https://phabricator.wikimedia.org/T341229) (owner: 10Joal) [13:39:57] (03Merged) 10jenkins-bot: Update ProduceCanaryEvents job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1011354 (https://phabricator.wikimedia.org/T341229) (owner: 10Joal) [13:40:13] 06Data-Engineering, 10Data-Platform-SRE (2024.03.04 - 2024.03.24), 13Patch-For-Review: aqs endpoint health alerting about mismatched check - https://phabricator.wikimedia.org/T360522#9649230 (10brouberol) We have disabled the AQS service on all hosts. The Cassandra cluster was left completely untouched. [13:40:25] 06Data-Engineering, 10Data-Platform-SRE (2024.03.04 - 2024.03.24), 13Patch-For-Review: 14aqs endpoint health alerting about mismatched check - 14https://phabricator.wikimedia.org/T360522#9649231 (10brouberol) 05Open→03Resolved [13:46:06] (03PS1) 10Joal: Bump changelog for v0.2.33 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1013307 [13:47:13] (03CR) 10Joal: [V:03+2 C:03+2] "Merging for release" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1013307 (owner: 10Joal) [13:47:58] Project analytics-refinery-maven-release-docker build #138: 09SUCCESS in 19 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release-docker/138/ [14:15:06] 10Data-Engineering (Sprint 9): We should provide DQ integration with Python - https://phabricator.wikimedia.org/T353940#9649652 (10gmodena) >>! In T353940#9615651, @xcollazo wrote: >> lets maybe pair on it? > > I'd love to hack on this at the offsite!! I cleaned up a bit some code I started working on last wee... [14:32:26] Starting build #99 for job analytics-refinery-update-jars-docker [14:32:52] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.2.33 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1012665 [14:32:52] Project analytics-refinery-update-jars-docker build #99: 09SUCCESS in 26 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars-docker/99/ [14:40:53] (03CR) 10Joal: [V:03+2 C:03+2] "Merge for future deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1012665 (owner: 10Maven-release-user) [14:53:23] 06Data-Engineering, 10Observability-Logging, 06Traffic: Remove extra fields currently sent to Kafka - https://phabricator.wikimedia.org/T360642 (10Fabfur) 03NEW [14:59:49] 10Quarry: Scrape prometheus metrics from Quarry - https://phabricator.wikimedia.org/T360220#9649858 (10phuedx) >>! In T360220#9634600, @taavi wrote: > They are supposed to be available on https://prometheus.wmcloud.org ([[ https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Monitoring | docs ]]) but seems... [15:15:20] 06Data-Engineering, 06Structured-Data-Backlog: Make HTML Dumps available in hadoop - https://phabricator.wikimedia.org/T305688#9649896 (10fkaelin) Pasting this reply from a slack [[ https://wikimedia.slack.com/archives/C01SJPB5Q10/p1711025738479019 | thread ]] for context - Agree with Isaac that no changes to... [15:20:43] 10Quarry, 10ChangeProp, 06Commons, 10GitLab, and 8 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9649904 (10brennen) For GitLab: I //think// we currently run the bundled Redis in their Omnibus package. In that case, the easiest th... [16:00:07] 06Data-Engineering, 06Data-Platform-SRE, 06serviceops: 14kafka-main certificates expiring on 2024-04-04 - 14https://phabricator.wikimedia.org/T360598#9650211 (10herron) 14FWIW I just went through a similar triage and broker restart process in T358870 It wasn't super obvious at first that all was needed... [16:17:05] 06Data-Engineering, 06Data-Platform-SRE, 06Discovery-Search, 06Java-Scala-Standardization, and 3 others: Adapt gitlab pipelines for the new wmf-jvm-parent-pom - https://phabricator.wikimedia.org/T358841#9650379 (10Gehel) After discussion, we will deploy the parent pom to Maven Central, so we will not need... [16:18:22] 06Data-Engineering, 06Data-Platform-SRE, 06Discovery-Search, 06Java-Scala-Standardization, and 3 others: Adapt gitlab pipelines for the new wmf-jvm-parent-pom - https://phabricator.wikimedia.org/T358841#9650388 (10CodeReviewBot) gehel opened https://gitlab.wikimedia.org/repos/ci-tools/wmf-jvm-parent-pom/-/... [16:21:26] brouberol: o/ [16:21:38] sorry if it was already asked, but I see a ton of warnings/alerts for aqs nodes [16:22:05] pybal/etc.. [16:22:35] IIUC aqs1.0 was stopped, any plan for the VIP etc..? [16:30:29] 10Data-Engineering (Sprint 9): [Maintenance] Migrate wmcs to Airflow - https://phabricator.wikimedia.org/T357938#9650468 (10lbowmaker) [16:31:15] 10Data-Engineering (Sprint 9): [Maintenance] Migrate wmcs to Airflow - https://phabricator.wikimedia.org/T357938#9650472 (10lbowmaker) [16:38:30] elukey: what's alerting atm? My understanding is that aqs was set to `page: false` in the service catalog [16:40:00] brouberol: check alerts.wikimedia.org :) [16:40:16] pybal for example is currently failing [16:42:25] I'm going to silence these alerts for now, and I'll work on resolving them tomorrow. I'm 20 min away from stepping out for today. If that helps, I can disable puppet on a single AQS host and restart the service [16:44:32] I've setup a silence of 7d [17:34:55] 10Data-Engineering (Sprint 9), 13Patch-For-Review: [Data Quality] Implement basic data quality metrics for MW history - https://phabricator.wikimedia.org/T354692#9650962 (10CodeReviewBot) ebysans opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/629 Add Dag to Compute Me... [18:22:41] (03PS1) 10Joal: Remove hql/cx folder [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1013384 [18:23:41] (03CR) 10Aleksandar Mastilovic: [C:03+1] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1013384 (owner: 10Joal) [18:24:09] (03CR) 10Joal: [V:03+2 C:03+2] "Merging for later deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1013384 (owner: 10Joal) [19:23:37] 06Data-Engineering, 06Data-Platform-SRE, 06Discovery-Search, 06Java-Scala-Standardization, and 3 others: Adapt gitlab pipelines for the new wmf-jvm-parent-pom - https://phabricator.wikimedia.org/T358841#9651589 (10CodeReviewBot) gehel merged https://gitlab.wikimedia.org/repos/ci-tools/wmf-jvm-parent-pom/-/... [19:50:16] 10Quarry: Remove redis - https://phabricator.wikimedia.org/T360584#9651747 (10Legoktm) >>! In T360584#9647966, @Frostly wrote: > Maybe consider https://www.dragonflydb.io/replace-redis It's not under an OSI-approved license, see https://github.com/dragonflydb/dragonfly/blob/main/LICENSE.md [19:50:22] 10Quarry: Remove redis from Quarry - https://phabricator.wikimedia.org/T360584#9651748 (10Legoktm) [20:02:53] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-coord1003:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1003:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [20:12:21] 06Data-Engineering, 06Structured-Data-Backlog: Bump memory to enable large artifacts sync on HDFS - https://phabricator.wikimedia.org/T348958#9651862 (10Ottomata) Oh, or: https://github.com/fsspec/filesystem_spec/blob/master/fsspec/implementations/http.py#L527-L528 > Supports only reading, with read-ahead... [20:28:07] 10Quarry, 10ChangeProp, 06Commons, 10GitLab, and 9 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9651946 (10Krinkle) [20:35:51] 10Quarry, 10ChangeProp, 10GitLab, 06Infrastructure-Foundations, and 8 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9651962 (10Peachey88) [21:14:14] 10Quarry, 10ChangeProp, 10GitLab, 06Infrastructure-Foundations, and 8 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9652082 (10Krinkle) In MediaWiki (as deployed at WMF), there exists 1 use of Redis, which is during file uploads v... [21:27:53] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-coord1003:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1003:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [22:02:49] 10Quarry, 10ChangeProp, 10GitLab, 06Infrastructure-Foundations, and 8 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9652274 (10Ladsgroup) >>! In T360596#9652082, @Krinkle wrote: > In MediaWiki (as deployed at WMF), there exists 1... [23:05:44] 06Data-Engineering, 06Data Products, 06Data-Platform, 06Movement-Insights: Wikistats "Active Editors by Country" does not follow definition for active editors - https://phabricator.wikimedia.org/T360073#9652449 (10kzimmerman) Hi @VirginiaPoundstone, my responses are below: > - Does this relate to a curre... [23:25:51] 10Quarry: Remove redis from Quarry - https://phabricator.wikimedia.org/T360584#9652467 (10Frostly) :( [23:32:28] 10Quarry: Remove redis from Quarry - https://phabricator.wikimedia.org/T360584#9652473 (10bd808) [23:32:35] 10Quarry, 10ChangeProp, 10GitLab, 06Infrastructure-Foundations, and 8 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9652472 (10bd808)