[06:52:34] 10Data-Engineering, 10Data-Engineering-Kanban: Refactor refinery-drop-mediawiki-snapshots so that it no longer uses a _SUCCESS file - https://phabricator.wikimedia.org/T303988 (10JAllemandou) 05Open→03Resolved [06:52:36] 10Data-Engineering: Review refinery scripts so that they no longer depend on _SUCCESS files - https://phabricator.wikimedia.org/T306611 (10JAllemandou) [06:52:38] 10Data-Engineering: Error with refinery-drop-mediawiki-snapshots: table specs not matching partitions for wmf/wikidata/entity and wmf/wikidata/item_page_link - https://phabricator.wikimedia.org/T305591 (10JAllemandou) [06:52:46] 10Data-Engineering, 10Data-Engineering-Kanban: Add the commons-entity dataset to the refinery-drop-mediawiki-snapshots script - https://phabricator.wikimedia.org/T303993 (10JAllemandou) 05Open→03Resolved [07:36:12] (VarnishkafkaNoMessages) firing: ... [07:36:12] varnishkafka for instance cp5006:9132 is not logging cache_upload requests from webrequest - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqsin%20prometheus/ops&var-source=webrequest&var-cp_cluster=cache_upload&var-instance=cp5006:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [08:30:14] 10Data-Engineering, 10Data-Engineering-Kanban: Create conda-base-env with last pyspark - https://phabricator.wikimedia.org/T309227 (10Antoine_Quhen) The .deb file is created with those repos: * https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils/-/merge_requests/28 * https://gitlab.wikimedia.org... [08:38:01] btullis, Ottomata: Hi, you can find the references to the code creating the deb file for Spark3 here https://phabricator.wikimedia.org/T309227 [08:38:01] The resulting .deb is here currently: https://gitlab.wikimedia.org/repos/data-engineering/conda-base-env/-/packages/132 [08:38:01] I've tried it with a docker debian image. [08:38:02] It works very similarly to https://gerrit.wikimedia.org/r/admin/repos/operations/debs/anaconda-wmf . [08:38:03] Let me know if you need explanations. [08:38:47] That's great aqu! spark3 everywhere soon then? [08:39:09] OK, thanks aqu - I will look into it now. We'd like this on the test cluster first, followed by production? [08:40:43] Yes, first, everywhere on the test cluster for validation. [08:40:54] :) [08:57:38] 10Data-Engineering, 10Event-Platform, 10Observability-Alerting, 10Patch-For-Review: Apparent latency warning in 90th centile of eventgate-logging-external - https://phabricator.wikimedia.org/T294911 (10BTullis) 05Open→03Resolved a:03BTullis I'm going to mark this ticket as resolved now, but there is... [09:29:15] aqu: I am confused a little by this line: https://gitlab.wikimedia.org/repos/data-engineering/conda-base-env/-/blob/main/README.md#L42 [09:30:22] The manual build instructions refer to an artifact already created. I was looking to repeat the build locally to get my head around it, but without depending on anything else. Have I misunderstood something? [10:25:38] btullis: you could also build the conda environment, and not downloding it as a tgz file. [10:26:32] The script is here: https://gitlab.wikimedia.org/repos/data-engineering/workflow_utils/-/blob/main/gitlab_ci_templates/lib/conda_dist.yml#L40-51 [10:27:18] Lets talk about it maybe. in 20minutes ? [10:27:35] aqu: I think that approach is probably going to be more acceptable, given that this is a coda-base-env it should be self-sufficient. Yeah, sure. Ping me when you're ready. [10:54:11] I've just asked a question of the Infrastructure Foundations team about whether they're happy for us to add this package directly to the apt repo, or whether we need to rebuild it. [10:54:58] btullis: batcave? [11:07:42] (VarnishkafkaNoMessages) resolved: ... [11:07:42] varnishkafka for instance cp5006:9132 is not logging cache_upload requests from webrequest - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=eqsin%20prometheus/ops&var-source=webrequest&var-cp_cluster=cache_upload&var-instance=cp5006:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [12:32:42] 10Data-Engineering, 10Event-Platform, 10Observability-Alerting, 10Patch-For-Review: Apparent latency warning in 90th centile of eventgate-logging-external - https://phabricator.wikimedia.org/T294911 (10Ottomata) Strange! `/v1/_test/events` is used for the k8s readinessProbe. I could see this endpoint tak... [12:35:08] 10Data-Engineering, 10Event-Platform, 10Observability-Alerting, 10Patch-For-Review: Apparent latency warning in 90th centile of eventgate-logging-external - https://phabricator.wikimedia.org/T294911 (10Ottomata) The [[ https://github.com/wikimedia/eventgate/blob/master/routes/events.js#L154-L163 | code tha... [12:39:04] 10Analytics, 10SRE: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10BTullis) @hashar Is this latency with Archiva still apparent? I guess it probably is, since you had to increase the timeouts again in February of this year. [12:43:00] 10Analytics, 10Data-Engineering, 10SRE, 10Traffic-Icebox: varnishkafka / ATSkafka should support setting the kafka message timestamp - https://phabricator.wikimedia.org/T277553 (10BTullis) Adding the #data-engineering tag so that this ticket does not get dropped when we deprecate #analytics. [12:44:57] 10Analytics, 10Data-Engineering, 10Patch-For-Review: Decide whether to migrate from Presto to Trino - https://phabricator.wikimedia.org/T266640 (10BTullis) Adding the #data-engineering tag so that this ticket does not get dropped when we deprecate #analytics. [12:50:46] 10Analytics: Add cache to MaxMindDB setup - https://phabricator.wikimedia.org/T265516 (10BTullis) Have we already enabled caching? https://github.com/wikimedia/analytics-refinery-source/blob/58af9b25c0c1cc8503879ae55450b4ef8838e55a/refinery-core/src/main/java/org/wikimedia/analytics/refinery/core/maxmind/Abstra... [13:00:36] 10Analytics, 10SRE, 10Traffic-Icebox: We are not capturing IPs of original requests for proxied requests from operamini and googleweblight. x-forwarded-for is null and client-ip is the same as IP on Webrequest data - https://phabricator.wikimedia.org/T232795 (10BTullis) Should we decline this ticket? [13:02:31] 10Analytics: Add cache to MaxMindDB setup - https://phabricator.wikimedia.org/T265516 (10JAllemandou) 05Open→03Resolved a:03JAllemandou I think we can close this indeed. Thanks @BTullis . [13:05:16] 10Analytics: add a more friendly message to ladp authentication box for pivot - https://phabricator.wikimedia.org/T163797 (10BTullis) 05Open→03Resolved a:03BTullis We can close this because we have replaced Pivot with Turnilo and that is now using CAS-SSO. [13:06:42] 10Analytics: Blog post about druid - https://phabricator.wikimedia.org/T157978 (10BTullis) 05Open→03Declined Declining this task as it is old and was deprioritized over four years ago. [13:10:48] 10Analytics: Could it be that the geo IP matching is not accurate for Africa? - https://phabricator.wikimedia.org/T90240 (10BTullis) I propose resolving this task, since we have not had any more reports of inaccuracies since 2015 and we have also upgraded to newer versions of the MaxMind databases for geolocatio... [13:14:46] 10Analytics: Create table in hive with a continent lookup for countries - https://phabricator.wikimedia.org/T127995 (10BTullis) I propose that we decline this ticket as it is 8 years old and does not reference a specific use case. Any objections? [13:15:19] !log roll-restarting the hadoop masters to pick up new JRE [13:15:21] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:16:59] 10Data-Engineering, 10Data-Engineering-Kanban, 10Generated Data Platform: [Shared Event Platform] - Research Flink Changelog semantics to inform POC MW schema design - https://phabricator.wikimedia.org/T310082 (10Ottomata) Note to self: document a process for making official proposals to event platform schem... [13:22:00] 10Data-Engineering, 10SRE, 10Traffic-Icebox: We are not capturing IPs of original requests for proxied requests from operamini and googleweblight. x-forwarded-for is null and client-ip is the same as IP on Webrequest data - https://phabricator.wikimedia.org/T232795 (10JAllemandou) >>! In T232795#7991992, @BT... [13:26:10] mforns: as expected, I have plenty review for you :) [13:28:01] mforns: I wonder about naming convertions if we wish to add the time-frequency to dag-id or not ... I have done it for the latest ones, let me know what you think [13:28:02] 👍 [13:28:04] aqu --^ [13:28:49] re. time frequency: might be necessary if there are several DAGs that treat the same dataset family in different time-frequencies [13:29:18] I would use them in this case to differentiate the jobs, i.e.: [13:29:34] process_dataset_daily vs process_dataset_monthly [13:29:48] right - I added them almost everywhere [13:29:50] otherwise, I don't think it adds much to have the time-frequency [13:29:56] let's see if it's worth or not [13:30:05] since Airflow shows the schedule_interval in the UI [13:30:09] yup [13:38:35] 10Data-Engineering, 10Data-Engineering-Kanban: Create conda-base-env with last pyspark - https://phabricator.wikimedia.org/T309227 (10BTullis) I have checked with the #infrastructure-foundations team that they are happy for us to use the .deb file created by GitLab-CI. [[https://wm-bot.wmflabs.org/libera_logs/... [13:47:08] btullis: o/ I have updated https://phabricator.wikimedia.org/T310169 with some info about IP ranges [13:48:27] mforns: did you get to work on that script or shall I continue tweaking and testing? [13:48:48] (the run_dev_instance work we started) [13:49:55] elukey: Thanks <3 that looks extremely useful. [13:51:13] FYI the sre.hadoop.role-restart-masters cookbook failed again for the analytics cluster. It didn't fail back from the standby to the master. Investigating now. [13:51:46] --^ that was for anyone - not intended for elu.key :-) [14:00:07] milimetric: yes! I worked on it, and I saw a bug, solved it, but I'm struggling to make the script edit the airflow.cnf file properly. [14:00:27] milimetric: if you want we can pair again. [14:00:41] mforns: sure, let's get it done :) [14:01:00] omw cave [14:01:01] ok, to the batcave! [14:01:06] :P [14:15:36] !log manually failing back HDFS namenode from an-master1002 to an-master1001 [14:15:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:32:21] 10Analytics, 10Data-Engineering, 10Event-Platform, 10Patch-For-Review: Refine drops $schema field values - https://phabricator.wikimedia.org/T255818 (10Ottomata) I just re-read some context on this ticket, and I wanted to write it down so it doesn't get lost. This change is implemented, but not enabled.... [14:42:38] (03PS3) 10Luke Bowmaker: Add Schema for Enriched MW Streams [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/799351 (https://phabricator.wikimedia.org/T308017) [14:56:09] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by pt1979@cumin1001 for host an-worker1142.eqiad.wmnet with OS buster [15:18:48] 10Data-Engineering: HDFS Namenode failover failure - https://phabricator.wikimedia.org/T310293 (10BTullis) [15:19:23] 10Data-Engineering, 10Data-Engineering-Kanban: HDFS Namenode failover failure - https://phabricator.wikimedia.org/T310293 (10BTullis) p:05Triage→03High [15:41:25] 10Data-Engineering, 10Data-Engineering-Kanban: HDFS Namenode failover failure - https://phabricator.wikimedia.org/T310293 (10BTullis) I think that this is similar in nature to the incident reported here: {T283733} One of the steps taken to resolve that incident ass an increase to the value of `dfs.namenode.se... [15:52:25] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by pt1979@cumin1001 for host an-worker1142.eqiad.wmnet with OS buster execute... [15:53:12] PROBLEM - Hadoop Namenode - Primary on an-master1001 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args org.apache.hadoop.hdfs.server.namenode.NameNode https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23HDFS_Namenode_process [15:54:12] RECOVERY - Hadoop Namenode - Primary on an-master1001 is OK: PROCS OK: 1 process with command name java, args org.apache.hadoop.hdfs.server.namenode.NameNode https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts%23HDFS_Namenode_process [15:55:06] --^ I'm working on this issue now. https://phabricator.wikimedia.org/T310293 [15:57:15] 10Data-Engineering, 10Data-Engineering-Kanban: HDFS Namenode failover failure - https://phabricator.wikimedia.org/T310293 (10BTullis) I tried starting the namenode on an-master1001 with 120 service handler threads instead of 100. It seemed to make a small difference, but it ultimately failed in the same way.... [15:59:06] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: Airflow DagProcessor not refreshing all dags - https://phabricator.wikimedia.org/T310297 (10Antoine_Quhen) [16:16:53] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10Papaul) I tested the pxe boot on an-worker1142 and server was not getting anything from dhcp server after debug , I found out that the server... [16:18:20] (03PS1) 10DCausse: Deprecate chronology_id [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/804352 (https://phabricator.wikimedia.org/T241410) [16:34:19] ottomata: https://gerrit.wikimedia.org/r/c/operations/puppet/+/803973 :] [17:02:24] 10Data-Engineering, 10Data-Engineering-Kanban: HDFS Namenode failover failure - https://phabricator.wikimedia.org/T310293 (10BTullis) It has worked with a heap of 72 GB, howver it had also had about an hour to settle before I tried the failover. ` btullis@an-master1001:~$ sudo -u hdfs /usr/bin/hdfs haadmin -fa... [17:04:14] 10Analytics, 10SRE: Downloading from Archiva.wikimedia.org seems slower than Maven Central - https://phabricator.wikimedia.org/T273086 (10hashar) @BTullis yes archiva is still rather slow. From the verbose curl commands above T273086#6783722, there is a large delay (1+ seconds) before the transfer start and t... [17:04:34] mforns, aqu: this one could take priority if you have a minute: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/82 [17:14:47] 10Data-Engineering, 10Data-Engineering-Kanban: HDFS Namenode failover failure - https://phabricator.wikimedia.org/T310293 (10BTullis) Interestingly, all that restarting of the namenode process and repeatedly loading the fsimage made a noticeable difference to the temperature of the server. {F35222963} [[https:... [17:17:25] !log Rerun refine for failed datasets [17:17:27] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:21:45] 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform, 10Generated Data Platform: Add better support for using Event Platform streams with the Flink DataStream API - https://phabricator.wikimedia.org/T310302 (10Ottomata) [17:21:47] 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform, 10Generated Data Platform: Add better support for using Event Platform streams with the Flink DataStream API - https://phabricator.wikimedia.org/T310302 (10Ottomata) [17:22:15] 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform, 10Generated Data Platform: Add better support for using Event Platform streams with the Flink DataStream API - https://phabricator.wikimedia.org/T310302 (10Ottomata) a:05gmodena→03Ottomata [17:22:40] 10Data-Engineering, 10Data-Engineering-Kanban, 10Event-Platform, 10Generated Data Platform: Add better support for using Event Platform streams with the Flink DataStream API - https://phabricator.wikimedia.org/T310302 (10Ottomata) [17:22:46] 10Quarry, 10GitLab (Project Migration): Move quarry to gitlab - https://phabricator.wikimedia.org/T308978 (10rook) Seems like waiting on this until some of the CI bits of gitlab are better established is recommended. ` Ahmon Dancy Rook: Sorry for the delay. We're still working on codifying best pract... [17:32:55] 10Data-Engineering, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Implement wikistats 2 endpoints - https://phabricator.wikimedia.org/T288301 (10BPirkle) @JAllemandou , we're finally getting ready to do actual work on this endpoint and could use some advice, specifically on best practices for testi... [19:03:47] 10Data-Engineering, 10Data-Engineering-Kanban: Drop GettingStarted* data - https://phabricator.wikimedia.org/T307774 (10Milimetric) 05Open→03Resolved p:05Triage→03Medium [19:03:49] 10Data-Engineering, 10Data-Engineering-Kanban: Drop UploadWizard* data - https://phabricator.wikimedia.org/T305556 (10Milimetric) 05Open→03Resolved p:05Triage→03Medium [19:05:55] (03CR) 10Milimetric: [C: 03+2] Fix typo in GitHub link (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/770059 (owner: 10EpicPupper) [19:06:21] (03CR) 10Milimetric: [C: 03+2] "sorry this took a while to review, it won't go out immediately, we have to do a build, but it'll be reflected eventually." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/770059 (owner: 10EpicPupper) [19:31:17] (03PS1) 10Milimetric: Improve efficiency 2x by not looking at upload [analytics/refinery] - 10https://gerrit.wikimedia.org/r/804429 [19:31:27] (03CR) 10Milimetric: [C: 03+2] Improve efficiency 2x by not looking at upload [analytics/refinery] - 10https://gerrit.wikimedia.org/r/804429 (owner: 10Milimetric) [19:31:34] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Improve efficiency 2x by not looking at upload [analytics/refinery] - 10https://gerrit.wikimedia.org/r/804429 (owner: 10Milimetric) [19:42:43] (03PS1) 10Milimetric: Review TODOs [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/804432 [19:42:45] (03PS1) 10Milimetric: Remove unused vega timeseries component [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/804433 [19:42:47] (03PS1) 10Milimetric: [WIP] Starting crossfilter layout work [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/804434 [20:01:16] (03CR) 10Milimetric: "I'm starting work on this, and pushing very often so you can see exactly what I'm doing if you want. If you just want to review the final" [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/804434 (owner: 10Milimetric) [20:26:39] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1142.eqiad.wmnet with OS buster [20:35:22] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1143.eqiad.wmnet with OS buster [20:36:11] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1144.eqiad.wmnet with OS buster [20:37:50] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1145.eqiad.wmnet with OS buster [20:38:38] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1146.eqiad.wmnet with OS buster [20:40:19] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1142.eqiad.wmnet with OS buster exec... [20:45:10] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1142.eqiad.wmnet with OS buster [20:46:43] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1144.eqiad.wmnet with OS buster exec... [20:46:55] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1146.eqiad.wmnet with OS buster exec... [20:47:05] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1145.eqiad.wmnet with OS buster exec... [20:52:21] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1143.eqiad.wmnet with OS buster exec... [20:55:06] 10Data-Engineering, 10Data-Engineering-Kanban: Create conda-base-env with last pyspark - https://phabricator.wikimedia.org/T309227 (10Ottomata) Wow okay! @Antoine_Quhen some Qs and Nits on the packaging: 1. package name. Maybe something other than conda-base-env would be more appropriate here. Perhaps cond... [21:12:47] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1142.eqiad.wmnet with OS buster comp... [21:30:12] 10Analytics, 10Cloud-Services, 10Developer-Advocacy: Data missing on the hierarchical view on the wmcs-edits tool - https://phabricator.wikimedia.org/T310317 (10srishakatux) [21:30:55] 10Analytics, 10Cloud-Services, 10Developer-Advocacy: Data missing on the hierarchical view on the wmcs-edits tool - https://phabricator.wikimedia.org/T310317 (10srishakatux) @Milimetric Would you or someone from your team may have the time to take a look at this issue? [21:52:29] (03PS2) 10Milimetric: [WIP] Starting crossfilter layout work [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/804434 [22:15:51] 10Data-Engineering, 10Data-Engineering-Kanban, 10Cloud-Services, 10Developer-Advocacy: Data missing on the hierarchical view on the wmcs-edits tool - https://phabricator.wikimedia.org/T310317 (10Milimetric) Yeah, it looks like the queries have been failing and the [[ https://analytics.wikimedia.org/publish... [22:16:05] 10Data-Engineering, 10Data-Engineering-Kanban, 10Cloud-Services, 10Developer-Advocacy: Data missing on the hierarchical view on the wmcs-edits tool - https://phabricator.wikimedia.org/T310317 (10Milimetric) a:03Milimetric [22:47:12] 10Data-Engineering, 10CheckUser, 10MW-1.38-notes (1.38.0-wmf.26; 2022-03-14), 10MW-1.39-notes (1.39.0-wmf.15; 2022-06-06), and 3 others: Update CheckUser for actor and comment table - https://phabricator.wikimedia.org/T233004 (10Zabe) >>! In T233004#7981926, @dom_walden wrote: >>>! In T233004#7975137, @Zab...