[00:08:38] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [00:33:50] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [01:08:06] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [01:30:44] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [02:04:18] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [02:29:26] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [03:03:34] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [03:28:46] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [04:02:52] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [04:25:17] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [05:07:22] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [05:32:10] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [06:06:04] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [06:31:12] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [07:05:18] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [07:30:26] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [08:04:29] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [08:29:34] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [10:04:18] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [10:29:24] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [11:03:28] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [11:16:56] (03PS14) 10Btullis: Add configuration for deployment pipeline [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/762950 (https://phabricator.wikimedia.org/T301453) [11:19:18] Can I help with this flapping eventlogging_to_druid_network_flows_internal_hourly unit somehow? [11:22:06] We just keep seeing this error: `ERROR DataFrameToDruid: Druid ingestion task index_hadoop_network_flows_internal_hiamognl_2022-02-18T11:00:41.602Z for network_flows_internal failed.` [11:28:34] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [11:30:45] (03PS15) 10Btullis: Add configuration for deployment pipeline [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/762950 (https://phabricator.wikimedia.org/T301453) [12:00:31] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [12:23:03] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [12:58:20] (03PS16) 10Btullis: Add configuration for deployment pipeline [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/762950 (https://phabricator.wikimedia.org/T301453) [12:58:55] (03CR) 10jerkins-bot: [V: 04-1] Add configuration for deployment pipeline [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/762950 (https://phabricator.wikimedia.org/T301453) (owner: 10Btullis) [13:03:05] (03CR) 10Btullis: "Will need retesting after: https://gerrit.wikimedia.org/r/c/integration/config/+/763724" [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/762950 (https://phabricator.wikimedia.org/T301453) (owner: 10Btullis) [13:07:17] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [13:07:26] 10Analytics-Radar, 10Anti-Harassment, 10CheckUser, 10Privacy Engineering, and 4 others: Deal with Google Chrome User-Agent deprecation - https://phabricator.wikimedia.org/T242825 (10Seddon) [13:07:50] (03CR) 10Btullis: Add configuration for deployment pipeline (038 comments) [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/762950 (https://phabricator.wikimedia.org/T301453) (owner: 10Btullis) [13:55:57] (03PS17) 10Btullis: Add configuration for deployment pipeline [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/762950 (https://phabricator.wikimedia.org/T301453) [13:56:48] (03CR) 10jerkins-bot: [V: 04-1] Add configuration for deployment pipeline [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/762950 (https://phabricator.wikimedia.org/T301453) (owner: 10Btullis) [14:05:17] (03PS18) 10Btullis: Add configuration for deployment pipeline [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/762950 (https://phabricator.wikimedia.org/T301453) [14:20:22] (03CR) 10Btullis: [C: 03+2] Add configuration for deployment pipeline [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/762950 (https://phabricator.wikimedia.org/T301453) (owner: 10Btullis) [14:24:24] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [14:29:26] 10Analytics, 10Gerrit-Privilege-Requests: Request membership in Analytics group for Aqu - https://phabricator.wikimedia.org/T302069 (10Aklapper) [14:31:26] (03Merged) 10jenkins-bot: Add configuration for deployment pipeline [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/762950 (https://phabricator.wikimedia.org/T301453) (owner: 10Btullis) [14:46:10] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog, 10Release Pipeline, 10Patch-For-Review: Create DataHub containers with deployment pipeline - https://phabricator.wikimedia.org/T301453 (10BTullis) I'm happy with the deployment pipeline part now, so I'm calling this task done. The container... [14:47:33] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: Define the Kubernetes Deployments for Datahub - https://phabricator.wikimedia.org/T301454 (10BTullis) Thanks @akosiaris - I'll go for this subcharts pattern with one service and see where I get. [15:09:48] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [15:16:27] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog, 10Release Pipeline, 10Patch-For-Review: Create DataHub containers with deployment pipeline - https://phabricator.wikimedia.org/T301453 (10BTullis) Actually, it's not quite done. The containers are all built with the same name, so I'll need to... [15:28:38] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [16:02:04] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [16:15:14] 10Data-Engineering, 10Product-Analytics: Improvements to mediawiki_geoeditors_monthly dimensions - https://phabricator.wikimedia.org/T302079 (10mpopov) [16:20:41] 10Data-Engineering, 10Product-Analytics: Improvements to mediawiki_geoeditors_monthly dimensions - https://phabricator.wikimedia.org/T302079 (10mpopov) [16:27:12] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [17:01:22] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [17:26:28] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [17:40:45] (03PS1) 10Btullis: Update image names and the tag [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/763795 (https://phabricator.wikimedia.org/T301453) [17:42:00] (03CR) 10Btullis: [C: 03+2] Update image names and the tag [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/763795 (https://phabricator.wikimedia.org/T301453) (owner: 10Btullis) [18:08:33] (03Merged) 10jenkins-bot: Update image names and the tag [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/763795 (https://phabricator.wikimedia.org/T301453) (owner: 10Btullis) [18:10:21] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [18:28:54] 10Analytics-Radar, 10Data-Engineering, 10Event-Platform, 10TimedMediaHandler, 10Wikimedia-Video: Record and report metrics for audio and video playback - https://phabricator.wikimedia.org/T108522 (10brion) [19:08:52] 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics, 10Superset: Help with data that's not appearing on charts - https://phabricator.wikimedia.org/T301895 (10Iflorez) Adding: The 2016 data point has been failing to show up on charts for 14 days now and the 2020 data point has... [19:45:33] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [19:50:46] Hi a-team is anybody around and would like to look at this eventlogging_to_druid_network_flows_internal_hourly error with me? [19:51:19] I'm around razzi, I'm in the middle of something though, I'll ping when I'm free [19:51:31] Sounds good milimetric [20:08:17] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [20:16:53] Well, there's our resolution milimetric ^ [20:17:41] lol razzi... good things come to those who... can't get npm to do what they want and in the meantime other stuff fixes itself? [20:18:03] Hey maybe I can help you with npm though! milimetric [20:18:37] ooh... ok, talking to my brother now, will ping [20:33:21] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [20:33:40] ok razzi, I'm around [20:34:55] cool milimetric join me in the 'cave [20:35:13] where I'll be conspicuously absent, going to get my headphones [20:38:16] (03PS2) 10Razzi: Add Dockerfile to build production dist folder [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/744885 [20:45:19] (03PS1) 10Milimetric: Audit and fix security vulnerabilities [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/763817 [20:57:13] (03CR) 10Razzi: [C: 03+2] Audit and fix security vulnerabilities [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/763817 (owner: 10Milimetric) [21:07:27] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [21:12:17] (03Merged) 10jenkins-bot: Audit and fix security vulnerabilities [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/763817 (owner: 10Milimetric) [21:32:37] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [21:44:33] (03PS1) 10Milimetric: Fix annotation display [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/763847 [21:44:43] ok razzi: that fixes the annotations ^ [21:44:53] if you look at it, you can check out http://localhost:5000/dist-dev/#/en.wikipedia.org/contributing/edits/normal|line|2010-12-07~2014-04-30|~total|monthly [21:45:09] notice that if you change the timespan to squish the annotations together it merges them [21:45:33] I'm going to build with your docker patch and deploy to staging so I can mess with it on my phone [21:46:14] lgtm milimetric , next week is my ops week so I'll be happy to deploy it on tuesday [21:47:32] I'll link you when I get it on staging [22:05:00] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [22:05:01] ok razzi it's up at https://wikistats-canary.wmflabs.org/annotations your docker build worked great [22:06:36] oof, I didn't realize how much abuse was happening on these annotation pages [22:07:38] classic wiki issue [22:08:20] (03CR) 10Milimetric: [C: 03+2] "Tried build a few times, deployed to https://wikistats-canary.wmflabs.org/annotations, looks great on desktop and mobile" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/744885 (owner: 10Razzi) [22:09:05] Before I forget let me add the docker build steps to the README [22:23:08] (03Merged) 10jenkins-bot: Add Dockerfile to build production dist folder [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/744885 (owner: 10Razzi) [22:27:30] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [23:09:30] RECOVERY - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [23:30:39] PROBLEM - Check unit status of eventlogging_to_druid_network_flows_internal_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_network_flows_internal_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [23:32:36] Added the docker build instructions to the wikistats2 wiki: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Wikistats_2#Releasing_a_new_version_to_production [23:34:14] The eventlogging_to_druid_network_flows_internal_hourly unit is unfortunately recovering and then failing again... not sure what to do there. [23:40:52] I put what looks like the error here for anybody else to look at: https://wikitech.wikimedia.org/wiki/User:Razzi/Debugging_eventlogging_to_druid_network_flows_internal_hourly.service