[00:43:42] 10Data-Engineering, 10Product-Analytics, 10Wmfdata-Python: Improve df_to_remarkup formatting for wmfdata-python - https://phabricator.wikimedia.org/T341589 (10nshahquinn-wmf) Hi Andrew! Thank you so much for digging into this. I think this is a great plan, so please do create a [pull request on GitHub](https... [03:25:04] (03PS1) 10David Martin: Add wikilambda_zobject_labels query to sqoop [analytics/refinery] - 10https://gerrit.wikimedia.org/r/937568 (https://phabricator.wikimedia.org/T341725) [08:28:59] 10Data-Engineering, 10Product-Analytics, 10Wmfdata-Python: Improve df_to_remarkup formatting for wmfdata-python - https://phabricator.wikimedia.org/T341589 (10AndrewTavis_WMDE) a:03AndrewTavis_WMDE [08:30:32] 10Data-Engineering, 10Product-Analytics, 10Wmfdata-Python: Improve df_to_remarkup formatting for wmfdata-python - https://phabricator.wikimedia.org/T341589 (10AndrewTavis_WMDE) Great, @nshahquinn-wmf! Thanks for the direction on it :) Will try to get to this in the coming week and we can discuss in the PR 😊 [09:13:20] 10Data-Engineering, 10Data-Platform-SRE, 10LDAP-Access-Requests, 10Shared-Data-Infrastructure (2022-23 Q4 Wrap up): Grant temporary access to web based Data Engineering tools to Bishop Fox - https://phabricator.wikimedia.org/T336357 (10BTullis) >>! In T336357#9007551, @Peachey88 wrote: >>>! In T336357#884... [09:39:48] 10Data-Platform-SRE, 10Patch-For-Review: Deploy ceph osd processes to data-engineering cluster - https://phabricator.wikimedia.org/T330151 (10BTullis) I'm picking up my previous work on this. One of the things that I need to do is to be able to get `ceph-volume lvm list --format=json` to work, so that I can u... [09:49:03] (03PS3) 10D3r1ck01: Add RESTBase response headers to AQS 1.0 [analytics/aqs] - 10https://gerrit.wikimedia.org/r/937073 (https://phabricator.wikimedia.org/T341038) [09:50:15] (03CR) 10D3r1ck01: "Per https://phabricator.wikimedia.org/T341038#9004946, note that some response headers are set at the edge. See comment in Phabricator." [analytics/aqs] - 10https://gerrit.wikimedia.org/r/937073 (https://phabricator.wikimedia.org/T341038) (owner: 10D3r1ck01) [09:52:20] (03CR) 10CI reject: [V: 04-1] Add RESTBase response headers to AQS 1.0 [analytics/aqs] - 10https://gerrit.wikimedia.org/r/937073 (https://phabricator.wikimedia.org/T341038) (owner: 10D3r1ck01) [09:55:21] 10Data-Platform-SRE: Grant all authenticated users access to SQL Lab in Superset - https://phabricator.wikimedia.org/T328457 (10BTullis) >>! In T328457#9010006, @Isaac wrote: > @BTullis a few more (I was able to create a chart that some people now want to see)! > * @Maryana: https://wikitech.wikimedia.org/wiki/U... [11:18:37] 10Data-Engineering, 10Data Engineering and Event Platform Team, 10Epic, 10Event-Platform: [Event Platform] Design and Implement realtime enrichment pipeline for MW page change with content - https://phabricator.wikimedia.org/T307959 (10gmodena) [11:21:40] 10Data-Engineering, 10Data Engineering and Event Platform Team, 10Epic, 10Event-Platform: [Event Platform] Design and Implement realtime enrichment pipeline for MW page change with content - https://phabricator.wikimedia.org/T307959 (10gmodena) [11:22:21] 10Data-Engineering, 10Data Engineering and Event Platform Team, 10Epic, 10Event-Platform: [Event Platform] Design and Implement realtime enrichment pipeline for MW page change with content - https://phabricator.wikimedia.org/T307959 (10gmodena) [11:29:47] 10Data-Engineering, 10Data Engineering and Event Platform Team, 10Event-Platform: Rename and release stream as mediawiki.page_content_change.v1 - https://phabricator.wikimedia.org/T341783 (10gmodena) [11:44:37] (03CR) 10Milimetric: [V: 03+2 C: 03+2] "tests pass, merging" [analytics/aqs] - 10https://gerrit.wikimedia.org/r/937073 (https://phabricator.wikimedia.org/T341038) (owner: 10D3r1ck01) [12:21:35] 10Quarry, 10superset.wmcloud.org, 10cloud-services-team (FY2022/2023-Q4): Replace Quarry with an installation of Superset - https://phabricator.wikimedia.org/T169452 (10Graham87) I'm an irregular user of Quarry but find it very useful when I do need it. I found some major accessibility problems with Superset... [12:25:19] 10Quarry, 10superset.wmcloud.org, 10cloud-services-team (FY2022/2023-Q4): Replace Quarry with an installation of Superset - https://phabricator.wikimedia.org/T169452 (10rook) >>! In T169452#9012318, @Graham87 wrote: > I'm an irregular user of Quarry but find it very useful when I do need it. I found some maj... [13:02:52] 10Quarry, 10superset.wmcloud.org, 10cloud-services-team (FY2022/2023-Q4): Replace Quarry with an installation of Superset - https://phabricator.wikimedia.org/T169452 (10Graham87) >>! In T169452#9012321, @rook wrote: >>>! In T169452#9012318, @Graham87 wrote: >> I'm an irregular user of Quarry but find it very... [13:04:48] 10Data-Platform-SRE, 10Patch-For-Review: Deploy ceph osd processes to data-engineering cluster - https://phabricator.wikimedia.org/T330151 (10BTullis) Next I ran the activation command: ` btullis@cephosd1001:~$ fsid=$(sudo ceph-volume lvm list /dev/disk/by-id/wwn-0x58ce38ee21f3c50d --format=json | jq -r '.[]|.... [13:10:09] 10Data-Platform-SRE, 10Discovery-Search: Provision Zookeeper Cluster for storing Flink HA data - https://phabricator.wikimedia.org/T341792 (10bking) [13:28:24] (03PS1) 10DCausse: Bump eventutilities to 1.2.12 and use new shaded artifact [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/937954 [13:33:47] (03CR) 10CI reject: [V: 04-1] Bump eventutilities to 1.2.12 and use new shaded artifact [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/937954 (owner: 10DCausse) [13:38:09] 10Data-Platform-SRE: Grant all authenticated users access to SQL Lab in Superset - https://phabricator.wikimedia.org/T328457 (10Isaac) > Those two accounts have also had the role added now. 👍 Thanks! [13:41:30] 10Data-Platform-SRE, 10Patch-For-Review: Deploy ceph osd processes to data-engineering cluster - https://phabricator.wikimedia.org/T330151 (10BTullis) ` btullis@cephosd1001:~$ sudo ceph osd tree ID CLASS WEIGHT TYPE NAME STATUS REWEIGHT PRI-AFF -1 3.49309 root default... [13:47:58] (03PS2) 10DCausse: Bump eventutilities to 1.2.12 and use new shaded artifact [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/937954 [13:52:28] (03CR) 10CI reject: [V: 04-1] Bump eventutilities to 1.2.12 and use new shaded artifact [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/937954 (owner: 10DCausse) [13:55:08] 10Data-Platform-SRE, 10Patch-For-Review: Deploy ceph osd processes to data-engineering cluster - https://phabricator.wikimedia.org/T330151 (10BTullis) OK, I simply needed to add the `--destroy` option to `ceph-volume lvm zap` ` btullis@cephosd1001:~$ sudo ceph-volume lvm zap /dev/disk/by-id/wwn-0x58ce38ee21f3c... [14:08:18] 10Data-Engineering, 10Event-Platform: [session length] Change domain of event collection to avoid ad-blocker issue - https://phabricator.wikimedia.org/T280256 (10MSantos) [14:08:35] 10Data-Engineering, 10Event-Platform: [session length] Investigate slight drop at sessions of 30 minutes or more - https://phabricator.wikimedia.org/T280254 (10MSantos) [14:27:53] Hello, currently working on T341700 thus I am about to disable puppet and the airflow-scheduler@analytics_test.service on an-test-client1001 before adding profile::airflow to an-test-client1002. [14:27:56] T341700: Migrate analytics_test airflow instance to bullseye an-test-client1002 - https://phabricator.wikimedia.org/T341700 [14:34:27] (MediawikiPageContentChangeEnrichTaskManagerNotRunning) firing: ... [14:34:27] The mw-page-content-change-enrich Flink cluster in codfw has no registered TaskManagers - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=codfw%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=All - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichTaskManagerNotRunning [14:39:27] (MediawikiPageContentChangeEnrichTaskManagerNotRunning) resolved: ... [14:39:27] The mw-page-content-change-enrich Flink cluster in codfw has no registered TaskManagers - TODO - https://grafana.wikimedia.org/d/K9x0c4aVk/flink-app?orgId=1&var-datasource=codfw%20prometheus/k8s&var-namespace=mw-page-content-change-enrich&var-helm_release=main&var-operator_name=All&var-flink_job_name=All - https://alerts.wikimedia.org/?q=alertname%3DMediawikiPageContentChangeEnrichTaskManagerNotRunning [15:34:13] 10Data-Platform-SRE, 10Discovery-Search: Provision Zookeeper Cluster for storing Flink HA data - https://phabricator.wikimedia.org/T341792 (10bking) [17:47:33] (03PS1) 10Milimetric: Fix build by bumping versions [analytics/aqs] - 10https://gerrit.wikimedia.org/r/937995 [17:47:44] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Fix build by bumping versions [analytics/aqs] - 10https://gerrit.wikimedia.org/r/937995 (owner: 10Milimetric) [17:56:00] (03PS1) 10Milimetric: Update aqs to 161c239 [analytics/aqs/deploy] - 10https://gerrit.wikimedia.org/r/937996 [18:03:44] (03CR) 10Milimetric: [V: 03+2 C: 03+2] "merging to deploy" [analytics/aqs/deploy] - 10https://gerrit.wikimedia.org/r/937996 (owner: 10Milimetric) [18:30:06] (03PS1) 10David Martin: Create wikilambda_zobject_labels table in Hive [analytics/refinery] - 10https://gerrit.wikimedia.org/r/937997 (https://phabricator.wikimedia.org/T341729) [18:47:45] (03CR) 10Gmodena: "Are tests known/expected to fail? The error reported in CI does not seem related to the version bump?" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/937954 (owner: 10DCausse) [19:01:13] (03CR) 10Milimetric: "This change is ready for review." (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/937997 (https://phabricator.wikimedia.org/T341729) (owner: 10David Martin) [19:20:43] (03PS3) 10David Martin: Create wikilambda_zobject_labels table in Hive [analytics/refinery] - 10https://gerrit.wikimedia.org/r/937997 (https://phabricator.wikimedia.org/T341729) [19:25:23] (03CR) 10David Martin: Create wikilambda_zobject_labels table in Hive (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/937997 (https://phabricator.wikimedia.org/T341729) (owner: 10David Martin) [19:32:34] 10Data-Platform-SRE, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:rack/setup/install an-worker11[49-56] - https://phabricator.wikimedia.org/T327295 (10Jhancock.wm) got the firmware for all the servers updated for everything (bios,idrac,network,sas) last night. finding time and connectivity to make sure all the bios se... [20:38:20] !log deployed Airflow DAGs for analytics instance to pickup T335860 [20:38:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:38:24] T335860: Implement job to transform mediawiki.page_content_change - https://phabricator.wikimedia.org/T335860 [21:04:30] 10Data-Engineering, 10Data Engineering and Event Platform Team, 10Discovery-Search (Current work), 10Event-Platform, and 2 others: Add support for redirects in CirrusSearch - https://phabricator.wikimedia.org/T325315 (10pfischer) a:05pfischer→03None [21:12:31] 10Data-Engineering, 10Data-Engineering-Wikistats, 10Data Products, 10Data Pipelines (Sprint 12): Non-mobile UAs on mobile (2g/gprs, etc) IP-blocks - https://phabricator.wikimedia.org/T58628 (10VirginiaPoundstone) Hello @MarkAHershberger! As I am onboarding as the new Product Manager to Data Products, I am... [21:18:12] 10Data-Engineering, 10Data-Engineering-Wikistats, 10Data Pipelines, 10Data Products: "Pages to date" not loading with "daily" metric - https://phabricator.wikimedia.org/T312717 (10VirginiaPoundstone) @Milimetric looks like you squashed this bug last year. Is that true? If it is let's close the ticket.