[03:23:30] <jinxer-wm>	 (MediawikiPageContentChangeEnrichAvailability) firing: ...
[03:23:30] <jinxer-wm>	 Low percentage of enriched events produced by mw_page_content_change_enrich in codfw - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=codfw%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichAvailability
[04:56:51] <wikibugs>	 10Quarry, 10superset.wmcloud.org, 10cloud-services-team (FY2022/2023-Q4): Replace Quarry with an installation of Superset - https://phabricator.wikimedia.org/T169452 (10Frostly) See subtasks, documentation is being worked on.
[07:23:30] <jinxer-wm>	 (MediawikiPageContentChangeEnrichAvailability) firing: ...
[07:23:30] <jinxer-wm>	 Low percentage of enriched events produced by mw_page_content_change_enrich in codfw - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=codfw%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichAvailability
[08:02:42] <jinxer-wm>	 (SystemdUnitFailed) firing: produce_canary_events.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:04:46] <icinga-wm>	 PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[08:15:16] <icinga-wm>	 RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state
[08:17:42] <jinxer-wm>	 (SystemdUnitFailed) resolved: produce_canary_events.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[11:23:30] <jinxer-wm>	 (MediawikiPageContentChangeEnrichAvailability) firing: ...
[11:23:30] <jinxer-wm>	 Low percentage of enriched events produced by mw_page_content_change_enrich in codfw - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=codfw%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichAvailability
[15:23:30] <jinxer-wm>	 (MediawikiPageContentChangeEnrichAvailability) firing: ...
[15:23:31] <jinxer-wm>	 Low percentage of enriched events produced by mw_page_content_change_enrich in codfw - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=codfw%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichAvailability
[19:23:47] <jinxer-wm>	 (MediawikiPageContentChangeEnrichAvailability) firing: ...
[19:23:47] <jinxer-wm>	 Low percentage of enriched events produced by mw_page_content_change_enrich in codfw - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=codfw%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichAvailability
[19:47:19] <wikibugs>	 10Quarry, 10Wikifunctions: Wikifunctions seems to be inaccessible from Quarry - https://phabricator.wikimedia.org/T343665 (10Nintendofan885)
[20:29:56] <wikibugs>	 10Quarry, 10Wikifunctions: Wikifunctions seems to be inaccessible from Quarry - https://phabricator.wikimedia.org/T343665 (10Framawiki) 05Open→03Resolved a:03Framawiki now functional: https://quarry.wmcloud.org/query/75686 [[ https://github.com/framawiki/quarry/blob/99f7a7b72dfe51d903d43cd61a681fff65bb2d...
[20:33:42] <wikibugs>	 10Quarry, 10Abstract Wikipedia team, 10Wikifunctions: Wikifunctions seems to be inaccessible from Quarry - https://phabricator.wikimedia.org/T343665 (10Framawiki) I don't know what `sites` table contains, it's not managed by Quarry that just shows the content of the replicas db. See [[ https://www.mediawiki....
[22:12:13] <wikibugs>	 10Data-Engineering, 10Data-Engineering-Dashiki, 10MediaWiki-Platform-Team (Radar): Investigate surprising "10% Other" portion of Analytics Browsers report - https://phabricator.wikimedia.org/T342267 (10Krinkle)
[23:28:30] <jinxer-wm>	 (MediawikiPageContentChangeEnrichAvailability) firing: ...
[23:28:31] <jinxer-wm>	 Low percentage of enriched events produced by mw_page_content_change_enrich in codfw - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=codfw%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=mw_page_content_change_enrich - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichAvailability