[00:09:41] (SystemdUnitFailed) firing: (6) jupyter-aqu-singleuser-conda-analytics.service Failed on stat1005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:09:41] (SystemdUnitFailed) firing: (6) jupyter-aqu-singleuser-conda-analytics.service Failed on stat1005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:45:50] 10Data-Engineering, 10Abstract Wikipedia team, 10Anti-Harassment, 10Cloud-Services, and 18 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10Physikerwelt) [08:09:41] (SystemdUnitFailed) firing: (6) jupyter-aqu-singleuser-conda-analytics.service Failed on stat1005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:15:01] 10Data-Engineering, 10Advanced-Search, 10All-and-every-Wikisource, 10ArticlePlaceholder, and 67 others: Remove unnecessary targets definitions - https://phabricator.wikimedia.org/T328497 (10Iniquity) [08:16:50] 10Data-Engineering, 10Advanced-Search, 10All-and-every-Wikisource, 10ArticlePlaceholder, and 66 others: Remove unnecessary targets definitions - https://phabricator.wikimedia.org/T328497 (10Iniquity) [08:17:40] 10Data-Engineering, 10Advanced-Search, 10All-and-every-Wikisource, 10ArticlePlaceholder, and 66 others: Remove unnecessary targets definitions - https://phabricator.wikimedia.org/T328497 (10Iniquity) [08:48:22] (SystemdUnitFailed) firing: (7) produce_canary_events.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:50:00] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [09:00:18] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [09:03:22] (SystemdUnitFailed) firing: (7) produce_canary_events.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:04:41] (SystemdUnitFailed) firing: (6) jupyter-aqu-singleuser-conda-analytics.service Failed on stat1005:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:59:30] 10Data-Engineering, 10Abstract Wikipedia team, 10Anti-Harassment, 10Cloud-Services, and 18 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10Dbrant) [14:13:09] 10Data-Engineering, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 17 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10Jdforrester-WMF) [14:18:44] 10Data-Engineering, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 17 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10taavi) [14:22:06] 10Data-Engineering, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 17 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10ssastry) [14:31:03] 10Data-Engineering, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 17 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10taavi) [14:33:22] (SystemdUnitFailed) firing: (7) hadoop-yarn-nodemanager.service Failed on an-test-worker1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:03:52] 10Data-Engineering, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 17 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10bd808) [15:08:07] 10Data-Engineering, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 17 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10bd808) The main thing I can think of being a potential blocker for #toolhub and #wikimedia-developer-portal moving would be tr... [15:09:32] 10Data-Engineering-Planning, 10Data Pipelines, 10Epic: Support for Product Analytics Data Pipelines Migration to Airflow - https://phabricator.wikimedia.org/T332997 (10xcollazo) [15:09:37] (03PS3) 10Snwachukwu: Add referer_name field to pageview_hourly table in hive. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/906073 (https://phabricator.wikimedia.org/T334120) [15:24:08] 10Data-Engineering, 10DBA, 10Infrastructure-Foundations, 10Machine-Learning-Team, and 9 others: codfw row C switches upgrade - https://phabricator.wikimedia.org/T334049 (10MPhamWMF) [17:44:25] 10Data-Engineering, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 17 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10thcipriani) >>! In T332953#8768454, @bd808 wrote: > The main thing I can think of being a potential blocker for #toolhub and #... [18:08:39] (03PS1) 10Jameel Kaisar: Created experimental/geoip/network_latency 1.0.0 schema [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/907508 (https://phabricator.wikimedia.org/T334417) [18:19:41] (SystemdUnitFailed) firing: (8) produce_canary_events.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [18:21:48] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [18:27:51] 10Data-Engineering, 10Advanced-Search, 10All-and-every-Wikisource, 10ArticlePlaceholder, and 66 others: Remove unnecessary targets definitions - https://phabricator.wikimedia.org/T328497 (10Jdlrobson) [18:30:32] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [18:33:22] (SystemdUnitFailed) firing: (8) produce_canary_events.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:01:34] 10Data-Engineering, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 17 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10sbassett) [19:04:41] (SystemdUnitFailed) firing: (8) produce_canary_events.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:08:51] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [19:13:58] 10Data-Engineering, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 17 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10sbassett) >>! In T332953#8769056, @thcipriani wrote: > - Tricky part: recreate mediawiki-i18n-check, only run on changes from... [19:15:05] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [19:18:22] (SystemdUnitFailed) firing: (8) produce_canary_events.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:20:32] !log deployed airflow analytics to fix mediawiki wikitext history [19:20:34] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:59:40] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 11): Q4 eventutilities-python should bundle java deps. - https://phabricator.wikimedia.org/T327251 (10tchin) A less opaque place for inspiration is that the py4j library bundles the jar with its python wheel as well. They leave adding it to the... [20:12:00] is https://wikitech.wikimedia.org/wiki/Analytics/Data_access#Requesting_access still up to date? the kerberos access request form seems to be using a project tag that's been archived since [20:17:31] I think that #data-engineering is the replacement for #analytics in Phabricator. See https://phabricator.wikimedia.org/project/profile/5462/ [20:18:22] (SystemdUnitFailed) firing: (7) hadoop-yarn-nodemanager.service Failed on an-test-worker1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:34:41] (SystemdUnitFailed) firing: (8) produce_canary_events.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:48:22] (SystemdUnitFailed) firing: (7) produce_canary_events.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:07:23] 10Data-Engineering, 10Anti-Harassment, 10Cloud-Services, 10Content-Transform-Team, and 17 others: Migrate PipelineLib repos to GitLab - https://phabricator.wikimedia.org/T332953 (10Michaelcochez) [22:19:41] (SystemdUnitFailed) firing: (7) produce_canary_events.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:20:52] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [22:30:36] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [22:33:22] (SystemdUnitFailed) firing: (7) produce_canary_events.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:49:41] (SystemdUnitFailed) firing: (7) produce_canary_events.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:51:38] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [23:01:18] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [23:03:22] (SystemdUnitFailed) firing: (7) produce_canary_events.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed