[02:19:15] 10Data-Engineering, 10Platform Engineering Roadmap, 10User-Eevans: Pageviews integration testing - https://phabricator.wikimedia.org/T299735 (10BPirkle) [09:57:20] 10Analytics, 10SRE, 10Traffic: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10BTullis) Thanks @Vgutierrez for bringing this to our attention. I agree that we should try to find the cause of these errors and eradicate it if at all... [09:57:59] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, 10Traffic: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10BTullis) a:03BTullis [11:59:52] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, 10Traffic: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10BTullis) My suspicion is that these workers need more CPU and/or memory. We recently doubled the number of replica... [12:26:09] (03PS1) 10Snwachukwu: Create Hql script to generate API(rest and action) metrics. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/785153 (https://phabricator.wikimedia.org/T300028) [12:28:57] 10Data-Engineering, 10Product-Analytics: Request for March 2022 per-host traffic - https://phabricator.wikimedia.org/T306480 (10JAllemandou) a:03JAllemandou [12:29:23] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics: Request for March 2022 per-host traffic - https://phabricator.wikimedia.org/T306480 (10JAllemandou) [12:29:32] (03CR) 10Snwachukwu: "Hi Joseph," [analytics/refinery] - 10https://gerrit.wikimedia.org/r/785153 (https://phabricator.wikimedia.org/T300028) (owner: 10Snwachukwu) [12:38:26] (03PS2) 10Snwachukwu: Create Hql script to generate API(rest and action) metrics. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/785153 (https://phabricator.wikimedia.org/T300028) [12:41:43] 10Data-Engineering: Refactor `refinery-drop-older-than` so that it no longer uses _SUCCESS file - https://phabricator.wikimedia.org/T306611 (10JAllemandou) [12:56:56] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics: Request for March 2022 per-host traffic - https://phabricator.wikimedia.org/T306480 (10JAllemandou) This is done, data is available at `hdfs:///wmf/data/wmf/pageview/dataloss_2022-03` Please let us know when your analysis are complete so tha... [13:11:18] (03CR) 10Joal: Create Hql script to generate API(rest and action) metrics. (033 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/785153 (https://phabricator.wikimedia.org/T300028) (owner: 10Snwachukwu) [13:11:30] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10cmooney) Thanks @Cmjohnson Note the IP addresses assigned to the servers need to be updated to match those vlans. [13:39:15] 10Data-Engineering, 10Equity-Landscape: Transformations Flowchart - https://phabricator.wikimedia.org/T306614 (10KCVelaga_WMF) [13:59:16] 10Data-Engineering, 10Equity-Landscape: Readership Metrics Transformation - https://phabricator.wikimedia.org/T306617 (10KCVelaga_WMF) [14:00:01] 10Data-Engineering, 10Equity-Landscape: Editorship Metrics Transformation - https://phabricator.wikimedia.org/T306618 (10KCVelaga_WMF) [14:00:56] 10Data-Engineering, 10Equity-Landscape: Affiliate Metrics Transformation - https://phabricator.wikimedia.org/T306619 (10KCVelaga_WMF) [14:01:40] 10Data-Engineering, 10Equity-Landscape: Grants Metrics Transformation - https://phabricator.wikimedia.org/T306620 (10KCVelaga_WMF) [14:03:34] 10Data-Engineering, 10Equity-Landscape: Programs Metrics Transform - https://phabricator.wikimedia.org/T306621 (10KCVelaga_WMF) [14:03:36] 10Data-Engineering, 10Equity-Landscape: Overall Engagement Metric (Transformation) - https://phabricator.wikimedia.org/T306622 (10KCVelaga_WMF) [14:04:16] 10Data-Engineering, 10Equity-Landscape: Population Metrics Transformation - https://phabricator.wikimedia.org/T306624 (10KCVelaga_WMF) [14:04:50] 10Data-Engineering, 10Equity-Landscape: Population Metrics Transformation - https://phabricator.wikimedia.org/T306624 (10KCVelaga_WMF) [14:04:52] 10Data-Engineering, 10Equity-Landscape: Overall Engagement Metric (Transformation) - https://phabricator.wikimedia.org/T306622 (10KCVelaga_WMF) [14:05:33] 10Data-Engineering, 10Equity-Landscape: Editorship Metrics Transformation - https://phabricator.wikimedia.org/T306618 (10KCVelaga_WMF) [14:05:35] 10Data-Engineering, 10Equity-Landscape: Readership Metrics Transformation - https://phabricator.wikimedia.org/T306617 (10KCVelaga_WMF) [14:05:37] 10Data-Engineering, 10Equity-Landscape: Milestone: Ingest and Transform Input Data - https://phabricator.wikimedia.org/T305475 (10KCVelaga_WMF) [14:06:29] 10Data-Engineering, 10Equity-Landscape: Affiliate Metrics Transformation - https://phabricator.wikimedia.org/T306619 (10KCVelaga_WMF) [14:06:31] 10Data-Engineering, 10Equity-Landscape: Milestone: Ingest and Transform Input Data - https://phabricator.wikimedia.org/T305475 (10KCVelaga_WMF) [14:06:33] 10Data-Engineering, 10Equity-Landscape: Editorship Metrics Transformation - https://phabricator.wikimedia.org/T306618 (10KCVelaga_WMF) [14:06:37] 10Data-Engineering, 10Equity-Landscape: Grants Metrics Transformation - https://phabricator.wikimedia.org/T306620 (10KCVelaga_WMF) [14:06:39] 10Data-Engineering, 10Equity-Landscape: Affiliate Metrics Transformation - https://phabricator.wikimedia.org/T306619 (10KCVelaga_WMF) [14:06:41] 10Data-Engineering, 10Equity-Landscape: Milestone: Ingest and Transform Input Data - https://phabricator.wikimedia.org/T305475 (10KCVelaga_WMF) [14:06:46] 10Data-Engineering, 10Equity-Landscape: Programs Metrics Transform - https://phabricator.wikimedia.org/T306621 (10KCVelaga_WMF) [14:06:49] 10Data-Engineering, 10Equity-Landscape: Grants Metrics Transformation - https://phabricator.wikimedia.org/T306620 (10KCVelaga_WMF) [14:06:50] 10Data-Engineering, 10Equity-Landscape: Milestone: Ingest and Transform Input Data - https://phabricator.wikimedia.org/T305475 (10KCVelaga_WMF) [14:06:59] 10Data-Engineering, 10Equity-Landscape: Programs Metrics Transform - https://phabricator.wikimedia.org/T306621 (10KCVelaga_WMF) [14:07:01] 10Data-Engineering, 10Equity-Landscape: Overall Engagement Metric (Transformation) - https://phabricator.wikimedia.org/T306622 (10KCVelaga_WMF) [14:10:01] 10Data-Engineering, 10Equity-Landscape: Extract + Transformation Raw Data into Input Metrics - https://phabricator.wikimedia.org/T306625 (10KCVelaga_WMF) [14:12:43] 10Data-Engineering, 10Equity-Landscape: Readership Metrics Transformation - https://phabricator.wikimedia.org/T306617 (10KCVelaga_WMF) [14:26:04] 10Data-Engineering, 10Equity-Landscape: Population Metrics Transformation - https://phabricator.wikimedia.org/T306624 (10KCVelaga_WMF) [14:35:05] 10Data-Engineering, 10Equity-Landscape: Editorship Metrics Transformation - https://phabricator.wikimedia.org/T306618 (10KCVelaga_WMF) [14:37:09] 10Data-Engineering, 10Equity-Landscape: Affiliate Metrics Transformation - https://phabricator.wikimedia.org/T306619 (10KCVelaga_WMF) [14:38:24] 10Data-Engineering, 10Equity-Landscape: Grants Metrics Transformation - https://phabricator.wikimedia.org/T306620 (10KCVelaga_WMF) [14:39:09] 10Data-Engineering, 10Equity-Landscape: Programs Metrics Transform - https://phabricator.wikimedia.org/T306621 (10KCVelaga_WMF) [14:39:49] 10Data-Engineering, 10Equity-Landscape: Overall Engagement Metric (Transformation) - https://phabricator.wikimedia.org/T306622 (10KCVelaga_WMF) [14:52:12] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, 10Traffic, 10Patch-For-Review: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10BTullis) At the moment we are getting between ~30 and ~60 requests receiving 503 responses pe... [15:09:59] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1143.eqiad.wmnet with OS buster [15:10:03] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1143.eqiad.wmnet with OS buster [15:10:09] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1143.eqiad.wmnet with OS buster [15:10:15] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1143.eqiad.wmnet with OS buster [15:10:22] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1143.eqiad.wmnet with OS buster [15:10:27] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1143.eqiad.wmnet with OS buster [15:10:34] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1143.eqiad.wmnet with OS buster [15:10:44] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1143.eqiad.wmnet with OS buster exec... [15:10:49] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1143.eqiad.wmnet with OS buster exec... [15:11:00] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1143.eqiad.wmnet with OS buster exec... [15:11:06] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1143.eqiad.wmnet with OS buster exec... [15:11:12] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1143.eqiad.wmnet with OS buster exec... [15:11:18] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1143.eqiad.wmnet with OS buster exec... [15:11:24] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1143.eqiad.wmnet with OS buster exec... [15:11:36] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1145.eqiad.wmnet with OS buster [15:11:44] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1143.eqiad.wmnet with OS buster [15:11:55] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1144.eqiad.wmnet with OS buster [15:12:06] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1146.eqiad.wmnet with OS buster [15:12:32] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1146.eqiad.wmnet with OS buster exec... [15:12:41] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1145.eqiad.wmnet with OS buster exec... [15:12:45] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1144.eqiad.wmnet with OS buster exec... [15:13:33] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1144.eqiad.wmnet with OS buster [15:13:52] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1145.eqiad.wmnet with OS buster [15:14:19] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cmjohnson@cumin1001 for host an-worker1146.eqiad.wmnet with OS buster [15:39:55] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1143.eqiad.wmnet with OS buster exec... [15:41:43] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1144.eqiad.wmnet with OS buster exec... [15:42:01] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1145.eqiad.wmnet with OS buster exec... [15:42:30] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q2:(Need By: TBD) rack/setup/install an-worker11[42-48].eqiad.wmnet - https://phabricator.wikimedia.org/T293922 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cmjohnson@cumin1001 for host an-worker1146.eqiad.wmnet with OS buster exec... [15:42:37] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, 10Traffic, 10Patch-For-Review: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10BTullis) I have increased the amount of RAM available to the eventgate-analytics-external dep... [15:43:01] ty btullis [15:44:00] A pleasure. I'm hoping that it helps with the error rate, otherwise I think we're going to have to look inside eventgate. [15:45:22] I happened across the ticket where you were trying to work out what to call eventgate: https://phabricator.wikimedia.org/T206785#4786495 🙂 [15:46:53] hahah [15:47:37] check out line 1419 in https://etherpad.wikimedia.org/p/event-platform [15:50:37] Ha, excellent. Let's change it to Eventhole - not too late is it? :-) [15:51:53] Sadly though, it doesn't look like increasing the RAM has made much of a difference to the error rate: https://logstash.wikimedia.org/goto/495ef2b27b5da739b7c6e2db10f27ae6 [15:52:39] https://usercontent.irccloud-cdn.com/file/Cw2HIGGy/image.png [15:53:25] 10Analytics-Radar, 10Data-Engineering-Radar, 10Event-Platform, 10Platform Engineering, and 3 others: eventlogging_VisualEditorTemplateDialogUse: '.event.template_names[0]' should be string - https://phabricator.wikimedia.org/T299779 (10ppelberg) [15:58:33] :/ [16:09:13] 10Data-Engineering: Add projects to sqoop list when synced in clouddb - https://phabricator.wikimedia.org/T304632 (10Ottomata) [16:09:16] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for guwwiki - https://phabricator.wikimedia.org/T303761 (10Ottomata) [16:09:24] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team (Kanban): Prepare and check storage layer for shnwikivoyage - https://phabricator.wikimedia.org/T302798 (10Ottomata) [16:29:51] 10Data-Engineering, 10Event-Platform, 10SRE, 10Traffic, and 2 others: Banner sampling leading to a relatively wide site outage (mostly esams) - https://phabricator.wikimedia.org/T303036 (10Krinkle) [16:30:06] 10Data-Engineering, 10Event-Platform, 10SRE, 10Traffic, and 2 others: Banner sampling leading to a relatively wide site outage (mostly esams) - https://phabricator.wikimedia.org/T303036 (10Krinkle) [16:44:51] 10Data-Engineering: How to submit a Pyspark job with a keytab and principal? - https://phabricator.wikimedia.org/T306313 (10bmansurov) 05Open→03Resolved Creating a spark session directly (as opposed to using wmfdata) seems to have fixed the issue. [17:21:51] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics: Request for March 2022 per-host traffic - https://phabricator.wikimedia.org/T306480 (10kzimmerman) Thank you @JAllemandou !! [18:01:04] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics (Kanban): Request for March 2022 per-host traffic - https://phabricator.wikimedia.org/T306480 (10kzimmerman) [18:15:35] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, 10Traffic: intake-analytics is responsible for up to a 85% of varnish backend fetch errors - https://phabricator.wikimedia.org/T306181 (10BTullis) The RAM upgrade has not resulted in any improvement. {F35061891,width=60%} [18:59:56] 10Quarry: Quarry history feature not showing history - https://phabricator.wikimedia.org/T306658 (10rook) [19:35:12] (VarnishkafkaNoMessages) firing: ... [19:35:12] varnishkafka for instance cp2029:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp2029:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [19:48:52] 10Data-Engineering, 10Platform Engineering Roadmap, 10User-Eevans: Pageviews integration testing - https://phabricator.wikimedia.org/T299735 (10Eevans) @BPirkle I'm a bit surprised that the legacy service strips `www.` prefixes. It strips `.org` because the project is basically the hostname, but all of the... [20:39:12] 10Data-Engineering, 10Platform Engineering Roadmap, 10User-Eevans: Pageviews integration testing - https://phabricator.wikimedia.org/T299735 (10BPirkle) Yeah, I thought it was a little unusual as well. I tested the existing production service using [[ https://wikimedia.org/api/rest_v1/metrics/pageviews/aggre... [20:42:23] 10Data-Engineering, 10Platform Engineering Roadmap, 10User-Eevans: Pageviews integration testing - https://phabricator.wikimedia.org/T299735 (10Eevans) >>! In T299735#7872717, @BPirkle wrote: > Yeah, I thought it was a little unusual as well. I tested the existing production service using [[ https://wikimedi... [20:55:12] (VarnishkafkaNoMessages) resolved: ... [20:55:12] varnishkafka for instance cp2029:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp2029:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [21:23:52] (03CR) 10Sharvaniharan: Android schemas migrated from legacy (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/778603 (owner: 10Sharvaniharan) [21:53:33] 10Data-Engineering, 10Platform Engineering Roadmap, 10User-Eevans: Pageviews integration testing - https://phabricator.wikimedia.org/T299735 (10BPirkle) I dug a bit into the existing production code. The "www" stripping was added [[ https://gerrit.wikimedia.org/r/c/analytics/aqs/+/275681 | as its own commit... [22:26:35] !log killed browser_general oozie job and started corresponding airflow job [22:26:37] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log