[00:30:05] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [07:56:45] Hi mforns - I'm reading your review, and I need help on understanding your suggestion - let me know when you have a minute [08:33:04] 10Data-Engineering-Operations, 10Data-Engineering-Planning, 10Mail, 10SRE, 10Patch-For-Review: Change the analytics-alerts email alias to a mailman distribution list - https://phabricator.wikimedia.org/T315486 (10BTullis) So far, all of the inbound messages to this mailing list have been held for moderat... [08:33:27] btullis: Good morning - let me know when you have a minute about alert emails :) [08:39:26] Now is good. I think that it is related to this change: https://gerrit.wikimedia.org/r/c/operations/puppet/+/843885 so godog: will probably want to know about this change in behaviour too. [08:40:39] Batcave? [08:46:17] btullis: Yes! I missed your message sorry [09:23:04] (03CR) 10Btullis: Update the email used for alerting the data engineering team (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/848269 (https://phabricator.wikimedia.org/T315486) (owner: 10Btullis) [09:30:34] 10Data-Engineering-Operations, 10Data-Engineering-Planning, 10Mail, 10SRE, 10Patch-For-Review: Change the analytics-alerts email alias to a mailman distribution list - https://phabricator.wikimedia.org/T315486 (10BTullis) @Dzahn - would it be acceptable for us to use the exim aliases file to forward the... [09:34:12] 10Data-Engineering: Check home/HDFS leftovers of bscarone - https://phabricator.wikimedia.org/T321542 (10MoritzMuehlenhoff) [09:37:26] 10Data-Engineering: Check home/HDFS leftovers of bscarone - https://phabricator.wikimedia.org/T321542 (10Miriam) Thank you so much for creating this @MoritzMuehlenhoff ! Yes, if possible, we would like to keep the data in Bruno's home directories until the project resumes in Jan/Feb. [10:13:25] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 03), 10Platform Team Initiatives (Modern Event Platform (TEC2)): Allow disabling/enabling configured streams via wgEventStreams config - https://phabricator.wikimedia.org/T259712 (10phuedx) Per-producer and per-consumer settings are acceptable because... [10:21:10] 10Data-Engineering, 10Data Pipelines: Migrate 1+ Refine jobs - https://phabricator.wikimedia.org/T307505 (10Aklapper) a:05NOkafor-WMF→03None Resetting inactive assignee. Please reassign tasks when offboarding - thanks.) [10:21:21] 10Data-Engineering, 10Event-Platform Value Stream: jsonschema-tools tests should fail if schema $id does not match title or path - https://phabricator.wikimedia.org/T300404 (10Aklapper) a:05NOkafor-WMF→03None Resetting inactive assignee. Please reassign tasks when offboarding - thanks.) [11:06:42] (03CR) 10Joal: [C: 04-1] Update the email used for alerting the data engineering team (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/848269 (https://phabricator.wikimedia.org/T315486) (owner: 10Btullis) [11:16:37] 10Data-Engineering-Operations, 10Data-Engineering-Planning, 10Mail, 10SRE, 10Patch-For-Review: Change the analytics-alerts email alias to a mailman distribution list - https://phabricator.wikimedia.org/T315486 (10Ladsgroup) What you did for accepting non-members looks good to me. I haven't seen any held... [11:21:08] 10Analytics-Kanban, 10Data-Engineering-Planning, 10Pageviews-Anomaly: Article on Carles Puigdemont has inflated pageviews in many projects - https://phabricator.wikimedia.org/T263908 (10jhsoby) @JArguello-WMF: Any update? :) [12:50:35] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 03), 10Platform Team Initiatives (Modern Event Platform (TEC2)): Allow disabling/enabling configured streams via wgEventStreams config - https://phabricator.wikimedia.org/T259712 (10Ottomata) > We could make the EventStreamConfigs extension override t... [12:52:54] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 03), 10Platform Team Initiatives (Modern Event Platform (TEC2)): Allow disabling/enabling configured streams via wgEventStreams config - https://phabricator.wikimedia.org/T259712 (10phuedx) >>! In T259712#8341374, @Ottomata wrote: >> We could make the... [12:54:18] 10Data-Engineering, 10Event-Platform Value Stream: EventBus' stream config destination_event_service setting should move into producers.mediawikI_eventbus specific settings. - https://phabricator.wikimedia.org/T321557 (10Ottomata) [13:11:34] heya joal, I'm here [13:15:27] Hi mforns [13:15:31] 10Data-Engineering, 10Event-Platform Value Stream: EventBus' stream config destination_event_service setting should move into producers.mediawikI_eventbus specific settings. - https://phabricator.wikimedia.org/T321557 (10Ottomata) [13:15:44] joal: bc? [13:21:55] (03Abandoned) 10Milimetric: [WIP] Add log-entry create schema [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/651635 (https://phabricator.wikimedia.org/T263055) (owner: 10Milimetric) [13:48:37] 10Data-Engineering-Planning, 10Shared-Data-Infrastructure, 10Event-Platform Value Stream (Sprint 03): [SPIKE] Deploy event driven stateless Flink service to DSE cluster - https://phabricator.wikimedia.org/T320812 (10EChetty) [13:50:23] mforns: do you have time? [13:50:31] yes! [13:50:35] bc? [13:50:46] wow sorry mforns - I didn't realized you had pinged me [13:50:51] sure, joining [13:51:04] no problemo! [13:56:09] 10Data-Engineering-Planning, 10Wikidata, 10Wikidata Analytics, 10Data Pipelines (Sprint 03): Some reliability metrics missing since June 20th '22 - https://phabricator.wikimedia.org/T314131 (10Michael) Thank you for looking into this! We seem to have full [reliability metrics](https://grafana.wikimedia.org... [14:00:10] (03CR) 10Mforns: [C: 03+1] Update mediawiki-history page and user computation (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/842922 (https://phabricator.wikimedia.org/T318589) (owner: 10Joal) [14:07:54] 10Analytics-Kanban, 10Data-Engineering-Planning, 10Pageviews-Anomaly: Article on Carles Puigdemont has inflated pageviews in many projects - https://phabricator.wikimedia.org/T263908 (10Pelajanela) >>! In T263908#8341166, @jhsoby wrote: > @JArguello-WMF: Any update? :) *//still interested in the topic and f... [14:23:50] Good morning milimetric - for when you have a minute, would you double check https://phabricator.wikimedia.org/T318589#8338832 and https://gerrit.wikimedia.org/r/c/analytics/refinery/source/+/842922 please? With your approval I'll call this done :) [14:39:14] 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: Obtain a security review of AQS 2.0 - https://phabricator.wikimedia.org/T288663 (10VirginiaPoundstone) [14:41:12] 10Data-Engineering-Planning, 10API Platform (Product Roadmap), 10Code-Health-Objective, 10Epic, and 3 others: AQS 2.0: Create repository for shared functions - https://phabricator.wikimedia.org/T311541 (10VirginiaPoundstone) a:05DAbad→03None [16:26:00] 10Data-Engineering, 10API Platform (Sprint 00), 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0:Wikistats 2 service - https://phabricator.wikimedia.org/T288301 (10VirginiaPoundstone) [16:29:09] 10Data-Engineering, 10API Platform (Sprint 00), 10Platform Engineering Roadmap, 10User-Eevans: Obtain a security review of AQS 2.0 - https://phabricator.wikimedia.org/T288663 (10VirginiaPoundstone) [17:19:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp4034 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4034%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [17:23:45] (03CR) 10Milimetric: [C: 03+2] "Code looks fine to me, thanks for doing all the testing and comparisons. I look forward to going over this as we think about an increment" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/842922 (https://phabricator.wikimedia.org/T318589) (owner: 10Joal) [17:24:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp4034 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4034%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [17:28:21] !log deployed refinery to the test cluster [17:28:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:29:45] Thanks milimetric for your review :) [17:33:29] (03Merged) 10jenkins-bot: Update mediawiki-history page and user computation [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/842922 (https://phabricator.wikimedia.org/T318589) (owner: 10Joal) [18:05:52] 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Add sql_tuple function to wmfdata-python - https://phabricator.wikimedia.org/T293706 (10nshahquinn-wmf) 05Open→03Resolved This has been released in [Wmfdata 1.4.0](https://github.com/wikimedia/wmfdata-python/blob/main/CHANGELOG.md#140-20-october-... [18:56:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp4047 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4047%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [19:01:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp4047 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4047%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [19:09:14] 10Data-Engineering-Planning, 10Shared-Data-Infrastructure, 10Data Pipelines (Sprint 03): Create Plan for Spark 2 Deprecation - https://phabricator.wikimedia.org/T318367 (10JArguello-WMF) @Miriam Could you please help Marcel review the inventory of all the remaining jobs still using spark 2 in your team? Than... [19:13:37] 10Data-Engineering, 10API Platform (API Platform Roadmap), 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Pageviews Service - https://phabricator.wikimedia.org/T288296 (10VirginiaPoundstone) [19:31:21] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 03): EventBus' stream config destination_event_service setting should move into producers.mediawikI_eventbus specific settings. - https://phabricator.wikimedia.org/T321557 (10Ottomata) [19:38:31] (03CR) 10Ottomata: "Merging this in prep for merge of I22d5e168fb320760061623e0e93b75efc3a74123" [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/807565 (https://phabricator.wikimedia.org/T308017) (owner: 10Ottomata) [19:38:42] (03CR) 10Ottomata: [C: 03+2] Add new mediawiki state entity and change fragments, and use them in new mediawiki page change schema [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/807565 (https://phabricator.wikimedia.org/T308017) (owner: 10Ottomata) [19:39:19] (03Merged) 10jenkins-bot: Add new mediawiki state entity and change fragments, and use them in new mediawiki page change schema [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/807565 (https://phabricator.wikimedia.org/T308017) (owner: 10Ottomata) [19:54:03] 10Data-Engineering, 10API Platform (Sprint 00), 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0:Wikistats 2 service - https://phabricator.wikimedia.org/T288301 (10VirginiaPoundstone) [19:54:39] 10Data-Engineering, 10API Platform (API Platform Roadmap), 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0:Wikistats 2 service - https://phabricator.wikimedia.org/T288301 (10VirginiaPoundstone) [22:10:28] 10Data-Engineering-Planning, 10Cassandra, 10Image-Suggestions, 10Section-Level-Image-Suggestions: Section Level Image Suggestions - Data Persistence Request - https://phabricator.wikimedia.org/T320831 (10Eevans) > The existing image suggestions data pipeline suggests images at an article level. There is a... [22:50:12] (VarnishkafkaNoMessages) firing: varnishkafka on cp4047 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4047%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [22:55:12] (VarnishkafkaNoMessages) resolved: varnishkafka on cp4047 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp4047%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [23:57:21] 10Data-Engineering-Operations, 10Data-Engineering-Planning, 10Mail, 10SRE, 10Patch-For-Review: Change the analytics-alerts email alias to a mailman distribution list - https://phabricator.wikimedia.org/T315486 (10Dzahn) @BTullis Yea, that is accetable. It's still progress over managing the group members...