[02:54:36] <wikibugs>	 10Data-Engineering, 10XTools, 10Chinese-Sites: Run maintain-views on zhwiki, newiki - https://phabricator.wikimedia.org/T334041 (10Shizhao)
[03:04:40] <jinxer-wm>	 (SystemdUnitFailed) firing: (6) jupyter-dsaez-singleuser-conda-analytics.service Failed on stat1004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status?orgId=1&forceLogin&editPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[04:43:22] <jinxer-wm>	 (SystemdUnitFailed) firing: (7) wmf_auto_restart_envoyproxy.timer Failed on an-test-ui1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status?orgId=1&forceLogin&editPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[06:27:08] <wikibugs>	 10Data-Engineering-Planning, 10DBA, 10Data-Persistence, 10Infrastructure-Foundations, and 11 others: eqiad row A switches upgrade - https://phabricator.wikimedia.org/T329073 (10hashar)
[06:51:30] <wikibugs>	 10Data-Engineering, 10serviceops-radar, 10Event-Platform Value Stream (Sprint 11): Store Flink HA metadata in Zookeeper - https://phabricator.wikimedia.org/T331283 (10dcausse) >>! In T331283#8759282, @Ottomata wrote: >> It would be nice to know how easy it is to switch between the two HA Service implementati...
[07:16:50] <elukey>	 Krinkle: o/ I added for people to the code change, there is usually a deployment happening on Tue/Wed
[07:17:07] <elukey>	 https://wikitech.wikimedia.org/wiki/Data_Engineering/Ops_week#The_Data_Engineering_deployment_train_%F0%9F%9A%82
[07:17:26] <elukey>	 once the change is merged it gets deployed during the next "train"
[08:38:28] <wikibugs>	 10Data-Engineering, 10serviceops, 10Event-Platform Value Stream (Sprint 11), 10Patch-For-Review: New Service Request: flink-kubernetes-operator - https://phabricator.wikimedia.org/T333464 (10JMeybohm) >>! In T333464#8759346, @Ottomata wrote: >> Could you please share resource requirements for the operator...
[08:44:49] <jinxer-wm>	 (SystemdUnitFailed) firing: (7) wmf_auto_restart_envoyproxy.timer Failed on an-test-ui1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status?orgId=1&forceLogin&editPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:32:20] <wikibugs>	 10Analytics-Radar, 10Data-Engineering-Radar, 10Event-Platform Value Stream: Move Kafka Jumbo's TLS clients to the new bundle - https://phabricator.wikimedia.org/T296064 (10elukey) Last steps:  * clean up certs in puppet private * verify if any change is needed in deployment-prep
[12:44:41] <jinxer-wm>	 (SystemdUnitFailed) firing: (7) wmf_auto_restart_envoyproxy.timer Failed on an-test-ui1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status?orgId=1&forceLogin&editPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[12:53:22] <jinxer-wm>	 (SystemdUnitFailed) firing: (7) wmf_auto_restart_envoyproxy.timer Failed on an-test-ui1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status?orgId=1&forceLogin&editPanel=13 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[13:10:19] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream, 10EventStreams, 10Patch-For-Review: Include image/file changes in page-links-change - https://phabricator.wikimedia.org/T333497 (10Isaac) > What do you think?  Hmm...what's the use-case for having wikilinks to articles and images in the same stream? On one...
[14:29:47] <icinga-wm_>	 PROBLEM - statsv Varnishkafka log producer on cp3064 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka
[14:30:03] <icinga-wm_>	 PROBLEM - eventlogging Varnishkafka log producer on cp3064 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/eventlogging.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka
[14:31:53] <icinga-wm_>	 RECOVERY - statsv Varnishkafka log producer on cp3064 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka
[14:32:11] <icinga-wm_>	 RECOVERY - eventlogging Varnishkafka log producer on cp3064 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/eventlogging.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka
[14:41:27] <wikibugs>	 (03PS1) 10Snwachukwu: Update pageview hourly and daily druid tables. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/906595 (https://phabricator.wikimedia.org/T334224)
[14:47:12] <wikibugs>	 (03PS2) 10Snwachukwu: Add referer_name field to pageview_hourly table in hive. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/906073 (https://phabricator.wikimedia.org/T334120)
[14:47:42] <wikibugs>	 (03CR) 10Snwachukwu: "mforms" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/906595 (https://phabricator.wikimedia.org/T334224) (owner: 10Snwachukwu)
[15:31:15] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream, 10EventStreams, 10Patch-For-Review: Include image/file changes in page-links-change - https://phabricator.wikimedia.org/T333497 (10TheresNoTime) >>! In T333497#8762380, @Isaac wrote: >> What do you think?  > Hmm...what's the use-case for having wikilinks t...
[15:58:28] <wikibugs>	 10Data-Engineering, 10Data-Engineering-Wikistats: Monthly pageview stats for March 2023 missing - https://phabricator.wikimedia.org/T333923 (10Antoine_Quhen) https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/352
[16:01:41] <wikibugs>	 (03CR) 10Aqu: Update pageview hourly and daily druid tables. (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/906595 (https://phabricator.wikimedia.org/T334224) (owner: 10Snwachukwu)
[16:06:55] <wikibugs>	 (03CR) 10Aqu: "Looks good" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/906073 (https://phabricator.wikimedia.org/T334120) (owner: 10Snwachukwu)
[16:54:40] <jinxer-wm>	 (SystemdUnitFailed) firing: (6) jupyter-dsaez-singleuser-conda-analytics.service Failed on stat1004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[16:56:21] <wikibugs>	 (03PS3) 10AikoChou: Add event schema for ML classification change on current page state [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/905965 (https://phabricator.wikimedia.org/T331401)
[17:00:55] <wikibugs>	 (03CR) 10AikoChou: Add event schema for ML classification change on current page state (032 comments) [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/905965 (https://phabricator.wikimedia.org/T331401) (owner: 10AikoChou)
[19:07:46] <wikibugs>	 10Data-Engineering-Planning, 10Data Pipelines, 10Editing-team, 10WMF-General-or-Unknown, 10Wikimedia-production-error: "Invalid revision ID -1" error for VisualEditorFeatureUse events, mostly from officewiki - https://phabricator.wikimedia.org/T322602 (10matmarex) I had a look at this today, as I was als...
[19:14:56] <wikibugs>	 10Data-Engineering-Planning, 10Data Pipelines, 10WMF-General-or-Unknown, 10Editing-team (Kanban Board), and 2 others: "Invalid revision ID -1" error for VisualEditorFeatureUse events, mostly from officewiki - https://phabricator.wikimedia.org/T322602 (10matmarex) a:03matmarex
[19:39:23] <wikibugs>	 10Data-Engineering, 10Data-Engineering-Wikistats: Monthly pageview stats for March 2023 missing - https://phabricator.wikimedia.org/T333923 (10Antoine_Quhen) The data has been regenerated and should be pushed automatically to the web endpoint at 5 am UTC.  Then it will appear here: https://dumps.wikimedia.org/...
[20:20:55] <wikibugs>	 10Data-Engineering, 10SRE, 10ops-eqiad, 10Patch-For-Review: Degraded RAID on an-worker1132 - https://phabricator.wikimedia.org/T333091 (10wiki_willy) a:05Cmjohnson→03Jclark-ctr
[20:54:41] <jinxer-wm>	 (SystemdUnitFailed) firing: (6) jupyter-dsaez-singleuser-conda-analytics.service Failed on stat1004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[21:41:42] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream, 10EventStreams, 10Patch-For-Review: Include image/file changes in page-links-change - https://phabricator.wikimedia.org/T333497 (10Isaac) @TheresNoTime thanks for explaining. I think I still lean towards separate streams all things equal then but ultimatel...