[00:21:36] 14Analytics, 10AQS2.0, 06Tech-Docs-Team, 10Data Products (Epics Timeline), and 2 others: AQS 2.0 user documentation - https://phabricator.wikimedia.org/T288664#9924516 (10apaskulin) [00:21:44] 14Analytics, 10AQS2.0, 06Tech-Docs-Team, 10Data Products (Epics Timeline), and 2 others: AQS 2.0 user documentation - https://phabricator.wikimedia.org/T288664#9924518 (10apaskulin) [00:24:08] 14Analytics, 10AQS2.0, 06Tech-Docs-Team, 10Data Products (Epics Timeline), and 2 others: AQS 2.0 user documentation - https://phabricator.wikimedia.org/T288664#9924527 (10apaskulin) [10:23:19] !log debdeploy conda-analytics 0.0.32 to test cluster T356231 [10:23:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [10:23:23] T356231: Package versions in Conda-Analytics are not pinned - https://phabricator.wikimedia.org/T356231 [11:12:10] !log deploy conda-analytics v 0.0.32 to analytics hadoop worker hosts T356231 T356230 [11:12:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:12:15] T356231: Package versions in Conda-Analytics are not pinned - https://phabricator.wikimedia.org/T356231 [11:12:15] T356230: Conda-Analytics packages incompatible with latest versions of Pandas and Numpy - https://phabricator.wikimedia.org/T356230 [11:22:15] !log deploy conda-analytics v 0.0.32 to analytics hadoop coordinator hosts T356231 T356230 [11:22:19] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:22:19] T356231: Package versions in Conda-Analytics are not pinned - https://phabricator.wikimedia.org/T356231 [11:22:20] T356230: Conda-Analytics packages incompatible with latest versions of Pandas and Numpy - https://phabricator.wikimedia.org/T356230 [11:33:06] !log deploy conda-analytics v 0.0.32 to analytics stat hosts T356231 T356230 [11:33:09] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:33:10] T356231: Package versions in Conda-Analytics are not pinned - https://phabricator.wikimedia.org/T356231 [11:33:10] T356230: Conda-Analytics packages incompatible with latest versions of Pandas and Numpy - https://phabricator.wikimedia.org/T356230 [11:43:09] 06Data-Engineering, 10[DEPRECATED] wdwb-tech, 10Citoid, 06Content-Transform-Team, and 10 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118#9926076 (10MSantos) [11:47:41] !log deploy conda-analytics v 0.0.32 to analytics airflow hosts T356231 T356230 [11:47:45] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:47:45] T356231: Package versions in Conda-Analytics are not pinned - https://phabricator.wikimedia.org/T356231 [11:47:46] T356230: Conda-Analytics packages incompatible with latest versions of Pandas and Numpy - https://phabricator.wikimedia.org/T356230 [12:17:47] 06Data-Engineering, 10Temporary accounts, 10Data-Platform-SRE (2024.06.17 - 2024.07.07): Generate a list of Superset users affected by changes to IP masking/temp users - https://phabricator.wikimedia.org/T347510#9926165 (10BTullis) a:03BTullis [12:18:10] 06Data-Engineering, 10Cassandra, 10Data Pipelines, 10Data-Platform-SRE (2024.06.17 - 2024.07.07): Create puppet defined type for adding/updating/deleting secrets or other small files on HDFS - https://phabricator.wikimedia.org/T323692#9926167 (10BTullis) a:03BTullis [13:16:04] 06Data-Engineering, 06Data-Platform, 10Wikidata, 10Wikidata-Query-Service: WDQS updater missed some updates - https://phabricator.wikimedia.org/T362977#9926342 (10dcausse) Another instance of this issue was reported on wiki: > @DCausse (WMF): fwiw, I have 6 items updated on the 19 & 20 June - https://w.wik... [13:29:54] 06Data-Engineering, 10Dumps 2.0 (Kanban Board), 10Event-Platform: [Dumps 2] Spike: Figure root causes of missing rows when doing reconciliation - https://phabricator.wikimedia.org/T368176#9926398 (10Ottomata) [13:38:19] (03CR) 10Snwachukwu: [C:03+2] Update column definition for uniqueness check. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1049580 (https://phabricator.wikimedia.org/T354692) (owner: 10Snwachukwu) [13:57:07] (03Merged) 10jenkins-bot: Update column definition for uniqueness check. [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1049580 (https://phabricator.wikimedia.org/T354692) (owner: 10Snwachukwu) [14:36:38] 10Data-Engineering (Q4 2024 April 1st - June 30th), 06Data Products, 06Movement-Insights, 10Movement-Metrics: Data Quality Issue: Wikitext History Job fail / rerun in Airflow - https://phabricator.wikimedia.org/T342911#9926713 (10Snwachukwu) a:03Snwachukwu [14:43:54] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 07Wikimedia-production-error: Error: Call to a member function getPageAsLinkTarget() on null - https://phabricator.wikimedia.org/T368543 (10Lucas_Werkmeister_WMDE) 03NEW [14:45:59] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 07Wikimedia-production-error: Error: Call to a member function getPageAsLinkTarget() on null - https://phabricator.wikimedia.org/T368543#9926759 (10Lucas_Werkmeister_WMDE) I’m not really sure how this code could ever have worked: `lang=php,name=includ... [15:14:41] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 15), and 2 others: Investigate surprising "10% Other" portion of Analytics Browsers report - https://phabricator.wikimedia.org/T342267#9926893 (10VirginiaPoundstone) CC @Mayakp.wiki [15:30:44] (03PS1) 10Xcollazo: Update changelog.md [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1049971 [15:31:22] (03CR) 10Xcollazo: [V:03+2 C:03+2] "Just a changelog change before release. Self-merging." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1049971 (owner: 10Xcollazo) [15:32:01] 06Data-Engineering, 10Temporary accounts, 10Data-Platform-SRE (2024.06.17 - 2024.07.07): Generate a list of Superset users affected by changes to IP masking/temp users - https://phabricator.wikimedia.org/T347510#9926973 (10BTullis) I've made a start on this by preparing a SQL query that selects all of the da... [15:33:01] Starting build #9 for job analytics-refinery-maven-release [15:51:23] 06Data-Engineering, 10Data-Platform (Data Platform Ops Week Working Group): Fix gobblin GobblinKafkaRecordsExtractedNotEqualRecordsExpected alert for Kafka compacted topics - https://phabricator.wikimedia.org/T368555 (10Ottomata) 03NEW [15:52:00] 06Data-Engineering, 10Data-Platform (Data Platform Ops Week Working Group), 03Discovery-Search (Current work): Fix gobblin GobblinKafkaRecordsExtractedNotEqualRecordsExpected alert for Kafka compacted topics - https://phabricator.wikimedia.org/T368555#9927153 (10Ottomata) [15:52:10] 06Data-Engineering, 06Discovery-Search, 10Data-Platform (Data Platform Ops Week Working Group): Fix gobblin GobblinKafkaRecordsExtractedNotEqualRecordsExpected alert for Kafka compacted topics - https://phabricator.wikimedia.org/T368555#9927157 (10Ottomata) [15:57:22] 06Data-Engineering, 06Data-Platform, 10Wikidata, 10Wikidata-Query-Service: WDQS updater missed some updates - https://phabricator.wikimedia.org/T362977#9927168 (10Ottomata) Related: {T368176} [15:57:39] 06Data-Engineering, 06Data Products, 10MediaWiki-extensions-EventLogging, 07Wikimedia-production-error: Error: Call to a member function getPageAsLinkTarget() on null - https://phabricator.wikimedia.org/T368543#9927176 (10Ottomata) FWIW, we are SO CLOSE to not needing EventLogging\JsonSchemaContent for WMF... [15:58:00] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#9927182 (10Milimetric) Thanks for pinging us on this. The sqoop code should run without modification, so we're good downstream. Thank you! [15:58:24] Project analytics-refinery-maven-release build #9: 09SUCCESS in 25 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/9/ [15:58:38] 06Data-Engineering, 10CirrusSearch, 06Data Products, 06Discovery-Search, and 2 others: Error: Call to a member function getPageAsLinkTarget() on null - https://phabricator.wikimedia.org/T368543#9927183 (10Ottomata) Adding Search since this seems to be triggered from CirrusSearch? [15:59:41] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Drop deprecated abuse filter fields on wmf wikis - https://phabricator.wikimedia.org/T367781#9927186 (10Milimetric) Thanks for pinging us, we don't use abuse filter tables anywhere I'm aware of, so this shouldn't affect us. [15:59:45] 06Data-Engineering, 10Cassandra: Create keyspace and table for Knowledge Gaps - https://phabricator.wikimedia.org/T340494#9927188 (10Miriam) Hi @lbowmaker @Eevans we see that this task has been resolved. Could you clarify what are the next steps? Has this task been resolved and this now on AQS 2 or has the... [16:05:09] 06Data-Engineering, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Upgrade hosts to haproxy 2.8.10 - https://phabricator.wikimedia.org/T367756#9927223 (10Fabfur) Update on this investigation: apparently capturing the frame with tcpdump from HAProxy to Benthos, doesn't show the "log merging". The w... [16:08:20] Starting build #9 for job analytics-refinery-update-jars [16:10:30] (03PS1) 10Maven-release-user: Add refinery-source jars for v0.2.43 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1049985 [16:10:31] Project analytics-refinery-update-jars build #9: 09SUCCESS in 2 min 10 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/9/ [16:11:27] (03CR) 10Xcollazo: [V:03+2 C:03+2] "LGTM." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1049985 (owner: 10Maven-release-user) [16:12:36] !log Deployed refinery-source using jenkins [16:12:38] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:15:44] 07Analytics-Data-Problem, 06Data-Engineering, 10Data-Engineering-Dashiki, 10Data Products (Data Products Sprint 15), and 2 others: Investigate surprising "10% Other" portion of Analytics Browsers report - https://phabricator.wikimedia.org/T342267#9927263 (10WDoranWMF) a:05Milimetric→03mforns [16:16:30] 06Data-Engineering, 10Temporary accounts, 10Data-Platform-SRE (2024.06.17 - 2024.07.07): Generate a list of Superset users affected by changes to IP masking/temp users - https://phabricator.wikimedia.org/T347510#9927269 (10BTullis) I have looked up the users who own the objects affected by the changes now, w... [16:20:18] 06Data-Engineering, 10Dumps-Generation, 06SRE, 10Data Products (Data Products Sprint 15), and 2 others: Dumps generation without prefetch cause disruption to the production environment - https://phabricator.wikimedia.org/T368098#9927276 (10WDoranWMF) p:05Unbreak!→03High [16:26:17] !log Deploying refinery using scap [16:26:18] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:45:26] (03PS1) 10Xcollazo: Move canary to stat1008, remove stat targets that no longer exist. [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/1049991 [16:48:15] PROBLEM - eventlogging Varnishkafka log producer on cp1100 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/eventlogging.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [16:49:07] (03PS2) 10Btullis: Move canary to stat1008, remove stat targets that no longer exist. [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/1049991 (https://phabricator.wikimedia.org/T353785) (owner: 10Xcollazo) [16:49:29] (03CR) 10Btullis: [C:03+1] "Added a bug reference, for good measure." [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/1049991 (https://phabricator.wikimedia.org/T353785) (owner: 10Xcollazo) [16:50:50] (03CR) 10Xcollazo: [V:03+2 C:03+2] "Merging!" [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/1049991 (https://phabricator.wikimedia.org/T353785) (owner: 10Xcollazo) [16:56:45] (03PS6) 10KCVelaga: Add Automoderator metrics to WMF Product jobs [analytics/wmf-product/jobs] - 10https://gerrit.wikimedia.org/r/1049106 (https://phabricator.wikimedia.org/T362615) [17:15:30] 06Data-Engineering, 10MediaWiki-General, 10Event-Platform, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#9927618 (10Ottomata) Deployed on varnish and varnishkafk... [17:22:00] 06Data-Engineering, 10MediaWiki-General, 10Event-Platform, 10MediaWiki-Platform-Team (Radar), 13Patch-For-Review: Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#9927637 (10Ottomata) Verified that /beacon/statsv still... [18:01:26] !log Deployed refinery using scap, then deployed onto hdfs [18:01:28] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:01:40] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for WBrown (WMF) - https://phabricator.wikimedia.org/T368260#9927777 (10Dzahn) Thanks! Gotcha. We are waiting for approval from data engineering. [18:04:57] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for DMburugu - https://phabricator.wikimedia.org/T367872#9927779 (10Dzahn) 05Open→03In progress p:05Triage→03High a:03Dzahn [18:07:07] !log 'Deployed latest DAGs to analytics Airflow instance.' [18:07:09] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:15:27] 06Data-Engineering, 10LDAP-Access-Requests, 06SRE: Grant Access to analytics-privatedata-users for cwylo - https://phabricator.wikimedia.org/T368027#9927826 (10cwylo) >>! In T368027#9913044, @kamila wrote: > @cwylo Can you please confirm that you have read the [[ https://wikitech.wikimedia.org/wiki/Analytics... [18:37:15] 06Data-Engineering, 10MediaWiki-extensions-EventLogging, 10Event-Platform, 13Patch-For-Review: Decommission EventLogging backend components by migrating to MEP - https://phabricator.wikimedia.org/T238230#9927910 (10Ottomata) [19:17:29] 06Data-Engineering, 10MediaWiki-General, 10Event-Platform, 10MediaWiki-Platform-Team (Radar): Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#9928080 (10Ottomata) Today starting at 17:51 UTC we deployed a change that ena... [19:56:34] 06Data-Engineering, 10MediaWiki-General, 10Event-Platform, 10MediaWiki-Platform-Team (Radar): Create legacy EventLogging proxy HTTP intake (for MediaWikiPingback) endpoint to EventGate - https://phabricator.wikimedia.org/T353817#9928279 (10Ottomata) I think we are going to need a special routing rule somew... [20:29:43] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for WBrown (WMF) - https://phabricator.wikimedia.org/T368260#9928408 (10Dzahn) 05In progress→03Stalled [20:29:49] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to private data-based dashboards for Jsn.sherman - https://phabricator.wikimedia.org/T367295#9928409 (10Dzahn) p:05Triage→03High [20:32:20] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for Kgraessle - https://phabricator.wikimedia.org/T367747#9928412 (10Dzahn) 05Open→03Stalled p:05Triage→03High [21:09:21] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for cwylo - https://phabricator.wikimedia.org/T368027#9928609 (10Dzahn) Ok, thanks for confirming both things! Sounds good. I also made a mistake there in my original question. When I mentioned the "wmf" group that... [21:09:35] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for cwylo - https://phabricator.wikimedia.org/T368027#9928614 (10Dzahn) [21:09:38] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for cwylo - https://phabricator.wikimedia.org/T368027#9928615 (10Dzahn) [21:12:51] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for cwylo - https://phabricator.wikimedia.org/T368027#9928622 (10Ottomata) Approved. [21:14:24] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for WBrown (WMF) - https://phabricator.wikimedia.org/T368260#9928626 (10Ottomata) Approved. [21:14:46] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to private data-based dashboards for Jsn.sherman - https://phabricator.wikimedia.org/T367295#9928629 (10Ottomata) Approved [21:15:30] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for Kgraessle - https://phabricator.wikimedia.org/T367747#9928632 (10Ottomata) Approved [21:21:43] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for Kgraessle - https://phabricator.wikimedia.org/T367747#9928641 (10Dzahn) Thanks! Confirmed user with check_user script Now has all needed approvals. Uploading change to code review. [21:21:46] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for Kgraessle - https://phabricator.wikimedia.org/T367747#9928645 (10Dzahn) 05Stalled→03In progress [21:22:05] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for Kgraessle - https://phabricator.wikimedia.org/T367747#9928646 (10Dzahn) a:03Dzahn [21:26:48] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for Kgraessle - https://phabricator.wikimedia.org/T367747#9928655 (10Dzahn) Can I assume you mean "private data in Superset" and not just "Superset, dashboards without private data"? Because those are different requ... [21:27:34] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Grant Access to analytics-privatedata-users for Kgraessle - https://phabricator.wikimedia.org/T367747#9928657 (10Kgraessle) >>! In T367747#9928655, @Dzahn wrote: > Can I assume you mean "private data in Superset" and not just "Superset, dashboards without pri... [21:29:03] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Grant Access to analytics-privatedata-users for Kgraessle - https://phabricator.wikimedia.org/T367747#9928661 (10Dzahn) Alright, thanks. Code change uploaded. You can expect this to be merged tomorrow. [22:08:09] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to analytics-privatedata-users for WBrown (WMF) - https://phabricator.wikimedia.org/T368260#9928791 (10Dzahn) 05Stalled→03In progress a:03Dzahn [22:11:21] 06Data-Engineering, 06SRE, 10SRE-Access-Requests, 13Patch-For-Review: Requesting access to analytics-privatedata-users for WBrown (WMF) - https://phabricator.wikimedia.org/T368260#9928799 (10Dzahn) [22:14:22] 06Data-Engineering, 06SRE, 10SRE-Access-Requests: Requesting access to private data-based dashboards for Jsn.sherman - https://phabricator.wikimedia.org/T367295#9928803 (10Dzahn) 05Stalled→03In progress a:03Dzahn