[01:13:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp2031 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp2031%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:18:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp2031 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp2031%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:28:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp2031 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp2031%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [01:33:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp2031 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp2031%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [05:26:41] (03CR) 10Gergő Tisza: [C: 03+1] image-suggestions-feedback: Bump to version 2.0.0 (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/809150 (https://phabricator.wikimedia.org/T302925) (owner: 10Kosta Harlan) [10:39:13] 10Data-Engineering-Planning, 10Epic, 10Patch-For-Review, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): Decide on installation details for new ceph cluster - https://phabricator.wikimedia.org/T326945 (10BTullis) There is another option in terms of the configuration mecahanism that I hadn't prev... [11:02:18] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 07): Tests for mediawiki-stream-enrichment-python flink job via eventutilities-python - https://phabricator.wikimedia.org/T326565 (10gmodena) a:03gmodena [11:11:07] 10Data-Engineering, 10Event-Platform Value Stream: [NEEDS GROOMING] eventutilities-python should bundle java deps. - https://phabricator.wikimedia.org/T327251 (10gmodena) [11:40:18] (03CR) 10Milimetric: Refactor and Expand External referer classification (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/864772 (https://phabricator.wikimedia.org/T309769) (owner: 10Snwachukwu) [11:43:07] PROBLEM - Presto Server on an-presto1015 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args com.facebook.presto.server.PrestoServer https://wikitech.wikimedia.org/wiki/Analytics/Systems/Presto/Administration%23Presto_server_down [11:43:17] PROBLEM - Presto Server on an-presto1011 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args com.facebook.presto.server.PrestoServer https://wikitech.wikimedia.org/wiki/Analytics/Systems/Presto/Administration%23Presto_server_down [11:43:22] PROBLEM - Check systemd state on an-presto1011 is CRITICAL: CRITICAL - degraded: The following units failed: presto-server.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [11:43:41] PROBLEM - Presto Server on an-presto1007 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args com.facebook.presto.server.PrestoServer https://wikitech.wikimedia.org/wiki/Analytics/Systems/Presto/Administration%23Presto_server_down [11:43:43] PROBLEM - Check systemd state on an-presto1014 is CRITICAL: CRITICAL - degraded: The following units failed: presto-server.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [11:43:47] PROBLEM - Presto Server on an-presto1008 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args com.facebook.presto.server.PrestoServer https://wikitech.wikimedia.org/wiki/Analytics/Systems/Presto/Administration%23Presto_server_down [11:43:51] PROBLEM - Check systemd state on an-presto1010 is CRITICAL: CRITICAL - degraded: The following units failed: presto-server.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [11:43:55] PROBLEM - Presto Server on an-presto1012 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args com.facebook.presto.server.PrestoServer https://wikitech.wikimedia.org/wiki/Analytics/Systems/Presto/Administration%23Presto_server_down [11:43:55] PROBLEM - Check systemd state on an-presto1006 is CRITICAL: CRITICAL - degraded: The following units failed: presto-server.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [11:44:01] PROBLEM - Presto Server on an-presto1009 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args com.facebook.presto.server.PrestoServer https://wikitech.wikimedia.org/wiki/Analytics/Systems/Presto/Administration%23Presto_server_down [11:44:03] PROBLEM - Presto Server on an-presto1010 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args com.facebook.presto.server.PrestoServer https://wikitech.wikimedia.org/wiki/Analytics/Systems/Presto/Administration%23Presto_server_down [11:44:05] PROBLEM - Check systemd state on an-presto1008 is CRITICAL: CRITICAL - degraded: The following units failed: presto-server.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [11:44:15] PROBLEM - Check systemd state on an-presto1012 is CRITICAL: CRITICAL - degraded: The following units failed: presto-server.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [11:44:15] PROBLEM - Check systemd state on an-presto1009 is CRITICAL: CRITICAL - degraded: The following units failed: presto-server.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [11:44:27] PROBLEM - Check systemd state on an-presto1013 is CRITICAL: CRITICAL - degraded: The following units failed: presto-server.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [11:44:31] ^-- The downtime on these new presto servers expired. [11:45:57] PROBLEM - Presto Server on an-presto1013 is CRITICAL: PROCS CRITICAL: 0 processes with command name java, args com.facebook.presto.server.PrestoServer https://wikitech.wikimedia.org/wiki/Analytics/Systems/Presto/Administration%23Presto_server_down [11:47:24] 10Data-Engineering-Planning: NEW BUG REPORT Presto cluster instabililty with more than 5 worker nodes - https://phabricator.wikimedia.org/T325809 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=4f65baff-bd05-4e2f-8578-eae4b972dc3b) set by btullis@cumin1001 for 30 days, 0:00:00 on 10 host(s) a... [12:15:31] 10Data-Engineering-Planning, 10Shared-Data-Infrastructure, 10Epic: Install Ceph Cluster for Data Engineering - https://phabricator.wikimedia.org/T324660 (10EChetty) [12:22:10] 10Data-Engineering-Planning, 10Epic, 10Patch-For-Review, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): Decide on installation details for new ceph cluster - https://phabricator.wikimedia.org/T326945 (10BTullis) Here's the initial comparison of functionality. I added the WMCS cookbooks for ceph... [13:56:34] (03CR) 10Ottomata: image-suggestions-feedback: Bump to version 2.0.0 (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/809150 (https://phabricator.wikimedia.org/T302925) (owner: 10Kosta Harlan) [14:16:55] 10Data-Engineering-Planning, 10Epic, 10Patch-For-Review, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): Decide on installation details for new ceph cluster - https://phabricator.wikimedia.org/T326945 (10MatthewVernon) I agree that taking advantage of existing Free Software (and then contributin... [14:44:23] 10Data-Engineering, 10CheckUser, 10MW-1.38-notes (1.38.0-wmf.26; 2022-03-14), 10MW-1.39-notes (1.39.0-wmf.23; 2022-08-01), and 4 others: Update CheckUser for actor and comment table - https://phabricator.wikimedia.org/T233004 (10Dreamy_Jazz) [14:56:04] 10Data-Engineering-Planning, 10Epic, 10Patch-For-Review, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): Decide on installation details for new ceph cluster - https://phabricator.wikimedia.org/T326945 (10BTullis) >...settings done by CLI rather than ceph.conf (e.g. ceph config set mon auth_allow... [15:35:14] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:rack/setup/install an-worker11[49-56] - https://phabricator.wikimedia.org/T327295 (10RobH) [15:35:41] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:rack/setup/install an-worker11[49-56] - https://phabricator.wikimedia.org/T327295 (10RobH) [15:41:55] 10Data-Engineering, 10Data-Services, 10Privacy Engineering, 10cloud-services-team (Kanban): Raw IPs of logged-out users disclosed in wiki-replicas - https://phabricator.wikimedia.org/T284948 (10Jdforrester-WMF) [16:05:44] 10Data-Engineering-Planning, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): Add an-presto10[06-15] to the presto cluster - https://phabricator.wikimedia.org/T323783 (10BTullis) Shall we move this back to in-progress @Stevemunene ? Have we got any theories as to why the cluster was less stable with... [16:06:06] 10Data-Engineering-Planning, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): Add an-presto10[06-15] to the presto cluster - https://phabricator.wikimedia.org/T323783 (10BTullis) p:05Triage→03Medium [16:08:51] (03CR) 10Eevans: image-suggestions-feedback: Bump to version 2.0.0 (033 comments) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/809150 (https://phabricator.wikimedia.org/T302925) (owner: 10Kosta Harlan) [16:09:11] 10Data-Engineering-Planning, 10serviceops, 10Discovery-Search (Current work), 10Event-Platform Value Stream (Sprint 07), 10Patch-For-Review: Flink on Kubernetes Helm charts - https://phabricator.wikimedia.org/T324576 (10Ottomata) Nope, [[ https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/88... [16:14:42] 10Data-Engineering-Planning, 10serviceops, 10Discovery-Search (Current work), 10Event-Platform Value Stream (Sprint 07), 10Patch-For-Review: Flink on Kubernetes Helm charts - https://phabricator.wikimedia.org/T324576 (10Ottomata) @JMeybohm hm, is the extra NetworkPolicy we made the flink-operator chart i... [16:18:35] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 07), 10Patch-For-Review: Flink application and flink-kubernetes-operator production docker images - https://phabricator.wikimedia.org/T316519 (10dduvall) >>! In T316519#8532670, @Ottomata wrote: > Should we change this? Should we set the run... [16:18:39] 10Data-Engineering, 10Equity-Landscape: Programs input metric (not until 2022 data update) - https://phabricator.wikimedia.org/T309277 (10JAnstee_WMF) 05In progress→03Stalled p:05Triage→03Low a:05ntsako→03KCVelaga_WMF [16:18:42] 10Data-Engineering, 10Equity-Landscape: Extract + Transformation Raw Data into Input Metrics - https://phabricator.wikimedia.org/T306625 (10JAnstee_WMF) [16:22:48] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 07), 10Patch-For-Review: Flink application and flink-kubernetes-operator production docker images - https://phabricator.wikimedia.org/T316519 (10Ottomata) > If that's not the case, I would just go with the default behavior which is the most r... [16:25:33] 10Data-Engineering: Requesting Kerberos identity for Hxi-ctr - https://phabricator.wikimedia.org/T325857 (10BTullis) [16:32:39] 10Data-Engineering: Requesting Kerberos identity for Hxi-ctr - https://phabricator.wikimedia.org/T325857 (10BTullis) Hi @HXi-WMF - I'm sorry to hear that you haven't been able to use Jupyter yet, let me see if I can help you. You mentioned this: >I put my shell username incorrectly in the original ticket — it i... [16:33:00] 10Data-Engineering: Requesting Kerberos identity for Hxi-ctr - https://phabricator.wikimedia.org/T325857 (10BTullis) [16:33:56] btullis: for T325857, the shell name in admin.yaml doesn't match the uid in ldap (T325004#8525568), which is bound to cause lots of problems [16:33:57] T325857: Requesting Kerberos identity for Hxi-ctr - https://phabricator.wikimedia.org/T325857 [16:35:04] 10Data-Engineering-Planning, 10Machine-Learning-Team, 10Research: Proposal: deprecate the mediawiki.revision-score stream in favour of more streams like mediawiki-revision-score- - https://phabricator.wikimedia.org/T317768 (10elukey) Hi @diego! I opened T327302 to investigate a way to provide streams... [16:38:25] (03CR) 10Kosta Harlan: image-suggestions-feedback: Bump to version 2.0.0 (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/809150 (https://phabricator.wikimedia.org/T302925) (owner: 10Kosta Harlan) [16:52:01] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q1:rack/setup/install druid10[09-11] - https://phabricator.wikimedia.org/T314335 (10BTullis) Hi @Papaul - apologies for the delay in getting back to you. Please could we use the `partman/raid10-8dev.cfg` recipe? I... [16:52:20] 10Data-Engineering-Planning, 10DC-Ops, 10SRE, 10Shared-Data-Infrastructure, 10ops-eqiad: Q1:rack/setup/install druid10[09-11] - https://phabricator.wikimedia.org/T314335 (10BTullis) [16:52:24] (03CR) 10Eevans: image-suggestions-feedback: Bump to version 2.0.0 (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/809150 (https://phabricator.wikimedia.org/T302925) (owner: 10Kosta Harlan) [16:53:01] (03CR) 10Ottomata: image-suggestions-feedback: Bump to version 2.0.0 (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/809150 (https://phabricator.wikimedia.org/T302925) (owner: 10Kosta Harlan) [17:12:00] (03PS5) 10Btullis: Upgrade superset to verstion 1.5.3 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/865609 (https://phabricator.wikimedia.org/T323458) [17:20:42] Hey btullis ottomata we enabled a click tracking schema (Schema:DesktopWebUIActionsTracking) on desktop today at 0.05%. We're not seeing much of an influx in data though so I'd like to bump it higher later today (to maybe 1%). Right now we're seeing 300 events per second. I assume no problem with doing this? [17:27:34] hi milimetric - if you're nearby I could use a few minutes of your time please :) [17:37:56] Jdlrobson: if you are at 300 per second at 0.05%, bumping to 1% is x 20 to aroudn 6000 per second? that's quite a lot, no? [17:42:29] 10Data-Engineering-Planning, 10Patch-For-Review, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): NEW FEATURE REQUEST: Upgrade superset to 1.5.3 - https://phabricator.wikimedia.org/T323458 (10BTullis) I've tried another deploy using version 1.5.3 but it looks like `scap` no longer works. I believe... [17:56:53] ottomata: sorry for confusing things, no [17:57:14] the300 per second includesother projects where we are sampling at 20% [17:57:29] It was around 300 per second when enwiki was 0 [17:57:40] so the bump so far is minor at 0.5 [17:57:51] unless there is something wrong with our intake pipeline / sampling code. [17:58:19] 10Data-Engineering-Planning, 10Patch-For-Review, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): NEW FEATURE REQUEST: Upgrade superset to 1.5.3 - https://phabricator.wikimedia.org/T323458 (10BTullis) That appears to have fixed the scap deploy, but version 1.5.3 of superset didn't work. The journal... [17:58:44] Jdlrobson: i'm not totally following, but my rule of thumb is as long as you aren't really going more than 1000 / second, we don't need to think too hard about it. [17:58:55] good to be aware though. thank you [17:59:04] hi joal, am I too late? [17:59:16] Hey milimetric - all good :) [17:59:23] 10Data-Engineering-Planning, 10serviceops, 10Discovery-Search (Current work), 10Event-Platform Value Stream (Sprint 07), 10Patch-For-Review: Flink on Kubernetes Helm charts - https://phabricator.wikimedia.org/T324576 (10Ottomata) FYI, I'm reverting the KUBERNETES_SERVICE_HOST change in https://gerrit.wik... [17:59:50] milimetric: batcave? [17:59:52] omw [18:05:10] Jdlrobson: Sorry, I'm not quite following the numbers either. You mention 0.05% in your first message, then later you said 0.5. I'm going by the same rule-of-thumb as ottomata, with 1000 events/sec being a reasonable ceiling, but I'm not quite sure which values you intend to tweak to get higher resolution. [18:07:02] Sorry i'm mixing decimals with percentages [18:07:52] so we log certain click events on our desktop site for various projects (JApanese and French being biggest) with a 20% sampling rate [18:08:06] We were seeing around 300 events per second relating to this schema [18:08:18] today we wanted to turn it on on English Wikipedia, so being cautious we started with 0.5% [18:08:21] (03PS6) 10Btullis: Upgrade superset to verstion 1.5.3 [analytics/superset/deploy] - 10https://gerrit.wikimedia.org/r/865609 (https://phabricator.wikimedia.org/T323458) [18:08:35] Jdlrobson: All good so far, thanks. [18:08:35] however to our surprise we saw very little change in intake of events [18:09:07] I'm not sure if this is due to caching or just because we underestimated how little people click things in English Wikipedia. [18:09:20] I'd like to double this at 1pm PST today and wanted to check in whether that was okay [18:09:39] Ideally I'd like to be seeing around 500 events per second. [18:09:55] but I suspect 1% won't get there, so I may need to increase it further [18:10:12] I wanted to check if there are any concerns your side, and what is a healthy events per second [18:10:28] For mobile we already have click tracking and we see around 1k events a second there, with English Wikipedia having a 1% sample rate. [18:10:38] It's highly likely that more clicks happen on mobile however due to the limited screen estate [18:12:52] OK gotcha, so that's a doubling from 0.5% to 0.1% for `DesktopWebUIActionsTracking` on enwiki? That sounds OK to me. Have you observed *any* events coming from your initial deploy at 0.5% today? [18:17:29] 10Data-Engineering-Planning, 10Patch-For-Review, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): NEW FEATURE REQUEST: Upgrade superset to 1.5.3 - https://phabricator.wikimedia.org/T323458 (10BTullis) OK, that version with a pinned cryptography package is now working. {F36305223,width=60%} @Mayakp... [18:29:08] 10Quarry, 10Cloud-Services, 10cloud-services-team: Consider moving Quarry to be an installation of Redash - https://phabricator.wikimedia.org/T169452 (10fnegri) [18:41:28] 10Analytics, 10Dumps-Generation, 10cloud-services-team: analytics-dumps-fetch-unique_devices.service failing on dumps servers - https://phabricator.wikimedia.org/T318849 (10fnegri) [18:49:26] 10Data-Engineering, 10Data-Services, 10cloud-services-team: Some wikibase tables not available in commonswiki_p - https://phabricator.wikimedia.org/T298452 (10fnegri) [18:50:42] 10Data-Engineering, 10Data-Services, 10Privacy Engineering, 10cloud-services-team: Increased visibility in wiki-replicas for volunteers fighting vandals - https://phabricator.wikimedia.org/T284944 (10fnegri) [18:50:51] 10Data-Engineering, 10Data-Services, 10Privacy Engineering, 10cloud-services-team: Raw IPs of logged-out users disclosed in wiki-replicas - https://phabricator.wikimedia.org/T284948 (10fnegri) [19:01:54] 10Quarry, 10cloud-services-team: Should quarry use our standard secrets management - https://phabricator.wikimedia.org/T290184 (10fnegri) [19:03:07] 10Quarry, 10cloud-services-team: Switch to using prefix puppet instead of direct-on-instance puppet - https://phabricator.wikimedia.org/T289531 (10fnegri) [19:10:35] 10Quarry, 10cloud-services-team: Quarry should detect a dead worker and report something better than "running" forever - https://phabricator.wikimedia.org/T278583 (10fnegri) [19:15:36] 10Analytics-Radar, 10Data-Services, 10cloud-services-team: Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511 (10fnegri) [19:21:29] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 07): Gitlab CI pipeline for Python applications should bundle Java eventutilities and runtime deps - https://phabricator.wikimedia.org/T326567 (10dancy) [19:30:56] 10Data-Engineering, 10Data-Services, 10cloud-services-team, 10Documentation: Provide documentation for toolforge users to request access to unexposed data through WikiReplicas - https://phabricator.wikimedia.org/T209992 (10fnegri) [19:36:33] btullis: yes im waiting for confirmation but it does seem like we've got events coming in [19:36:41] My data analyst is having power issues though [19:43:53] 10Quarry, 10cloud-services-team: Support queries against Quarry's own database and ToolsDB - https://phabricator.wikimedia.org/T151158 (10fnegri) [19:44:25] 10Quarry, 10cloud-services-team, 10Epic: Productionize quarry a bit - https://phabricator.wikimedia.org/T288982 (10fnegri) [19:44:32] 10Analytics-Radar, 10Data-Services, 10cloud-services-team: Mitigate breaking changes from the new Wiki Replicas architecture - https://phabricator.wikimedia.org/T280152 (10fnegri) [19:47:28] btullis: okay so have some data now. Looks like we've fired 17255 events for anonymous users since the deploy this morning ( about 5 hrs ago) so I'm expecting this to add about 4k events an hour. [19:54:38] 10Data-Engineering, 10Data-Services, 10cloud-services-team: Plan a replacement for wiki replicas that is better suited to typical OLAP use cases than the MediaWiki OLTP schema - https://phabricator.wikimedia.org/T215858 (10fnegri) [19:58:05] 10Data-Engineering-Planning, 10serviceops, 10Discovery-Search (Current work), 10Event-Platform Value Stream (Sprint 07), 10Patch-For-Review: Flink on Kubernetes Helm charts - https://phabricator.wikimedia.org/T324576 (10Ottomata) @akosiaris manually edited the flink-pod-k8s-api NetworkPolicy we added to... [19:58:15] 10Data-Engineering-Planning, 10serviceops, 10Discovery-Search (Current work), 10Event-Platform Value Stream (Sprint 07), 10Patch-For-Review: Flink on Kubernetes Helm charts - https://phabricator.wikimedia.org/T324576 (10Ottomata) @akosiaris to reproduce, just delete the flink-app-main pod in stream-enric... [20:51:18] 10Data-Engineering-Planning, 10Patch-For-Review, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): NEW FEATURE REQUEST: Upgrade superset to 1.5.3 - https://phabricator.wikimedia.org/T323458 (10Mayakp.wiki) Thanks @BTullis for the upgrade. I'll try to test 1.5.3 before the end of this week and let yo... [20:52:50] 10Data-Engineering-Planning, 10Product-Analytics (Kanban): Superset Date Filter fix needed - https://phabricator.wikimedia.org/T318299 (10Mayakp.wiki) Correction, Superset is being upgraded to v1.5.3 T323458 I will test the date filter box and report on any issues. [21:35:03] Jdlrobson: Thanks for that. Sounds good. [21:47:13] 10Analytics-Radar, 10Cloud-VPS, 10cloud-services-team: Report page views for labs instances - https://phabricator.wikimedia.org/T103726 (10fnegri) [21:54:31] 10Data-Engineering-Planning, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): Add an-presto10[06-15] to the presto cluster - https://phabricator.wikimedia.org/T323783 (10Stevemunene) Found a discussion on the presto github revolving around a similar issue. The number of worker nodes that a cluster ca... [22:05:40] 10Data-Engineering-Planning, 10Patch-For-Review, 10Shared-Data-Infrastructure (EQ2 Kanban (Sprints 04-05)): NEW FEATURE REQUEST: Upgrade superset to 1.5.3 - https://phabricator.wikimedia.org/T323458 (10Volans) >>! In T323458#8536080, @BTullis wrote: > @Mayakp.wiki , @Volans - If you'd like to do any final ch... [22:36:03] 10Data-Engineering, 10AQS 2.0 Roadmap, 10API Platform (API Platform Roadmap), 10Epic, and 2 others: AQS 2.0:Wikistats 2 service - https://phabricator.wikimedia.org/T288301 (10BPirkle) [22:44:19] 10Data-Engineering, 10Equity-Landscape: Affiliates input metric - https://phabricator.wikimedia.org/T309275 (10JAnstee_WMF) @KCVelaga the data replication/check looks good! I did spot some needed column label changes for alignment in the two inputs tables. table: **ntsako.affiliate_leadership_input_metrics**... [22:46:53] 10Data-Engineering, 10Pageviews-Anomaly: Massive spike in pageviews for a few enwiki pages beginning with "Index" - https://phabricator.wikimedia.org/T327027 (10Izno) [23:02:31] 10Data-Engineering, 10Equity-Landscape: Affiliates output rank metrics - https://phabricator.wikimedia.org/T306619 (10JAnstee_WMF) @KCVelaga I have no idea what's up with the two tiny discrepancies for subcon Australia & New Zealand and continent Africa, any ideas, could the SQL be dropping a comparison point... [23:04:10] 10Data-Engineering, 10Equity-Landscape: Overall Engagement output rank metric - https://phabricator.wikimedia.org/T306622 (10JAnstee_WMF) @KCVelaga_WMF Looks good! Signing off =)