[01:16:24] (03CR) 10Snwachukwu: Create HQL scripts to generate Wikidata's ArticlePlaceholder and Reliability metrics. (036 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/791394 (https://phabricator.wikimedia.org/T300021) (owner: 10Snwachukwu) [01:18:58] (03PS3) 10Snwachukwu: Add HQL scripts for wikidata graphite metrics [analytics/refinery] - 10https://gerrit.wikimedia.org/r/791394 (https://phabricator.wikimedia.org/T300021) [05:44:52] 10Data-Engineering, 10Data-Engineering-Kanban: Procure MaxMind GeoIP2 Database License - https://phabricator.wikimedia.org/T303453 (10odimitrijevic) The license has been extended to 7-13. They payment of the yearly subscription is still in progress. [08:44:59] 10Data-Engineering, 10Data-Engineering-Kanban: RAID battery malfunction in an-worker1081 - https://phabricator.wikimedia.org/T308267 (10BTullis) This incident has now been resolved, with sincere thanks to @wiki_willy and @Jclark-ctr. {F35178518,width=80%} [08:48:15] 10Data-Engineering-Radar, 10MW-1.39-notes (1.39.0-wmf.7; 2022-04-11): Decommission the UploadWizard* instruments - https://phabricator.wikimedia.org/T305238 (10phuedx) [08:56:32] 10Data-Engineering, 10Equity-Landscape: Readership input metrics - https://phabricator.wikimedia.org/T309273 (10ntsako) [08:57:05] 10Data-Engineering, 10Equity-Landscape: Editorship Input Metrics - https://phabricator.wikimedia.org/T309274 (10ntsako) [08:57:50] 10Data-Engineering, 10Equity-Landscape: Affiliates input metric - https://phabricator.wikimedia.org/T309275 (10ntsako) [08:58:35] 10Data-Engineering, 10Equity-Landscape: Grants input metric - https://phabricator.wikimedia.org/T309276 (10ntsako) [08:59:08] 10Data-Engineering, 10Equity-Landscape: Programs input metric - https://phabricator.wikimedia.org/T309277 (10ntsako) [09:00:12] 10Data-Engineering, 10Equity-Landscape: Programs input metric - https://phabricator.wikimedia.org/T309277 (10ntsako) [09:01:20] 10Data-Engineering, 10Equity-Landscape: Overall Engagement input metric - https://phabricator.wikimedia.org/T309278 (10ntsako) [09:02:07] 10Data-Engineering, 10Equity-Landscape: Population input metrics - https://phabricator.wikimedia.org/T309279 (10ntsako) [09:04:02] 10Data-Engineering, 10Equity-Landscape: Readership input metrics - https://phabricator.wikimedia.org/T309273 (10ntsako) Readership metric created for sample on ntsako.georeadership_metrics [09:04:21] 10Data-Engineering, 10Equity-Landscape: Readership input metrics - https://phabricator.wikimedia.org/T309273 (10ntsako) ` SELECT * FROM ntsako.georeadership_metrics WHERE year=2021 ` [09:04:34] 10Data-Engineering, 10Equity-Landscape: Extract + Transformation Raw Data into Input Metrics - https://phabricator.wikimedia.org/T306625 (10ntsako) [09:04:36] 10Data-Engineering, 10Equity-Landscape: Readership input metrics - https://phabricator.wikimedia.org/T309273 (10ntsako) 05Open→03In progress [09:05:34] 10Data-Engineering, 10Equity-Landscape: Editorship Input Metrics - https://phabricator.wikimedia.org/T309274 (10ntsako) Editorship input metrics created and stored in: ` SELECT * FROM ntsako.geoeditor_metrics_pivot WHERE year = 2021 AND country_code = '---' ` [09:05:50] 10Data-Engineering, 10Equity-Landscape: Extract + Transformation Raw Data into Input Metrics - https://phabricator.wikimedia.org/T306625 (10ntsako) [09:05:52] 10Data-Engineering, 10Equity-Landscape: Editorship Input Metrics - https://phabricator.wikimedia.org/T309274 (10ntsako) 05Open→03In progress [09:07:07] 10Data-Engineering-Radar, 10MW-1.39-notes (1.39.0-wmf.7; 2022-04-11): Decommission the UploadWizard* instruments - https://phabricator.wikimedia.org/T305238 (10phuedx) >>! In T305238#7957150, @Ottomata wrote: > @phuedx Should we also remove the current tables and data in the event database? There doesn't seem... [09:07:37] 10Data-Engineering-Radar, 10MW-1.39-notes (1.39.0-wmf.7; 2022-04-11): Decommission the UploadWizard* instruments - https://phabricator.wikimedia.org/T305238 (10phuedx) [09:07:39] 10Data-Engineering: Drop sanitized UploadWizard* data - https://phabricator.wikimedia.org/T305556 (10phuedx) [09:07:53] 10Data-Engineering: Drop sanitized UploadWizard* data - https://phabricator.wikimedia.org/T305556 (10phuedx) [09:07:55] 10Data-Engineering-Radar, 10MW-1.39-notes (1.39.0-wmf.7; 2022-04-11): Decommission the UploadWizard* instruments - https://phabricator.wikimedia.org/T305238 (10phuedx) [09:08:21] 10Data-Engineering: Drop sanitized UploadWizard* data - https://phabricator.wikimedia.org/T305556 (10phuedx) [09:08:26] 10Data-Engineering-Radar, 10MW-1.39-notes (1.39.0-wmf.7; 2022-04-11): Decommission the UploadWizard* instruments - https://phabricator.wikimedia.org/T305238 (10phuedx) 05Open→03Resolved a:03phuedx [09:08:28] 10Analytics-Kanban, 10Data-Engineering, 10Event-Platform, 10Fundraising-Backlog, and 3 others: Determine which remaining legacy EventLogging schemas need to be migrated or decommissioned - https://phabricator.wikimedia.org/T282131 (10phuedx) [09:11:14] 10Data-Engineering: Drop UploadWizard* data - https://phabricator.wikimedia.org/T305556 (10phuedx) [09:22:41] 10Data-Engineering, 10Equity-Landscape: Grants input metric - https://phabricator.wikimedia.org/T309276 (10ntsako) Loaded updated grants csv on: ` SELECT * FROM ntsako.grants WHERE year = 2021 ` [09:22:58] 10Data-Engineering, 10Equity-Landscape: Extract + Transformation Raw Data into Input Metrics - https://phabricator.wikimedia.org/T306625 (10ntsako) [09:23:00] 10Data-Engineering, 10Equity-Landscape: Grants input metric - https://phabricator.wikimedia.org/T309276 (10ntsako) 05Open→03In progress [09:24:12] 10Data-Engineering: Drop UploadWizard* data - https://phabricator.wikimedia.org/T305556 (10phuedx) >>! In T305238#7957150, @Ottomata wrote: > @phuedx Should we also remove the current tables and data in the event database? There doesn't seem to be much there anyway :) @MarkTraceur, as the manager for the team... [09:26:38] 10Data-Engineering: Drop GettingStarted* data - https://phabricator.wikimedia.org/T307774 (10phuedx) [09:26:57] 10Data-Engineering, 10Equity-Landscape: World Bank Data - https://phabricator.wikimedia.org/T309282 (10ntsako) [09:27:05] 10Data-Engineering: Drop GettingStarted* data - https://phabricator.wikimedia.org/T307774 (10phuedx) [09:27:14] 10Data-Engineering: Drop GettingStarted* data - https://phabricator.wikimedia.org/T307774 (10phuedx) [09:27:37] 10Data-Engineering, 10Equity-Landscape: Wiki DB Map - https://phabricator.wikimedia.org/T309283 (10ntsako) [09:28:35] 10Data-Engineering, 10Equity-Landscape: Wiki DB Map - https://phabricator.wikimedia.org/T309283 (10ntsako) Data loaded on ` SELECT * FROM ntsako.wiki_db_map ` [09:29:11] 10Data-Engineering, 10Equity-Landscape: World Bank Data - https://phabricator.wikimedia.org/T309282 (10ntsako) Data loaded on ` SELECT * FROM ntsako.world_bank_data ` [09:29:36] 10Data-Engineering, 10Equity-Landscape: World Bank Data - https://phabricator.wikimedia.org/T309282 (10ntsako) 05Open→03In progress [09:29:42] 10Data-Engineering, 10Equity-Landscape: Wiki DB Map - https://phabricator.wikimedia.org/T309283 (10ntsako) 05Open→03In progress [09:29:44] 10Data-Engineering, 10Equity-Landscape: Extract + Transformation Raw Data into Input Metrics - https://phabricator.wikimedia.org/T306625 (10ntsako) [10:03:11] 10Data-Engineering, 10Data-Catalog: Integrate Superset with DataHub - https://phabricator.wikimedia.org/T306903 (10BTullis) Good progress on this now. I have performed a manual ingestion from Superset. We now have metadata about 2,001 charts and 221 dashboards, as well as references to presto sources and linea... [10:38:03] 10Data-Engineering, 10Equity-Landscape: Grants input metric - https://phabricator.wikimedia.org/T309276 (10ntsako) Loaded input metrics on: ` SELECT * FROM ntsako.grants_leadership WHERE year=2021; ` Still need to do some data validation for some columns. [10:39:00] 10Data-Engineering, 10Data-Catalog: Integrate Superset with DataHub - https://phabricator.wikimedia.org/T306903 (10BTullis) The approach I took to carry out the ingestion was this: ==== Run a local superset instance ==== * activate my stacked conda environment on stat1008 - this is the same environment where... [10:39:27] 10Data-Engineering, 10Data-Catalog: Integrate Superset with DataHub - https://phabricator.wikimedia.org/T306903 (10BTullis) p:05Triage→03Medium [10:39:53] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Integrate Superset with DataHub - https://phabricator.wikimedia.org/T306903 (10BTullis) [11:14:21] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Integrate Superset with DataHub - https://phabricator.wikimedia.org/T306903 (10BTullis) I have created a paste with the errors that occurred during ingestion: P28592 There were some 500 errors with invalid character sequences, but there were als... [11:15:49] 10Data-Engineering, 10Equity-Landscape: Programs input metric - https://phabricator.wikimedia.org/T309277 (10ntsako) 05Open→03In progress [11:15:51] 10Data-Engineering, 10Equity-Landscape: Extract + Transformation Raw Data into Input Metrics - https://phabricator.wikimedia.org/T306625 (10ntsako) [11:48:13] 10Data-Engineering, 10SRE, 10Traffic-Icebox: Mobile redirects drop provenance parameters - https://phabricator.wikimedia.org/T252227 (10Isaac) [12:02:59] 10Data-Engineering, 10SRE, 10Traffic-Icebox: Mobile redirects drop provenance parameters - https://phabricator.wikimedia.org/T252227 (10Isaac) Thanks all for the input on this task and @BBlack especially for digging up what was happening. I finally updated the task description to reflect what I think is the... [13:24:06] 10Analytics-Radar, 10Dumps-Generation, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team): page_restrictions field incomplete in current and historical dumps - https://phabricator.wikimedia.org/T251411 (10Ladsgroup) [13:51:19] 10Analytics-Radar, 10Dumps-Generation, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team): page_restrictions field incomplete in current and historical dumps - https://phabricator.wikimedia.org/T251411 (10Ladsgroup) 05Stalled→03Open The field have been cleaned up in favor of the table an... [15:21:17] 10Data-Engineering, 10Equity-Landscape: Programs input metric - https://phabricator.wikimedia.org/T309277 (10ntsako) CSV loaded onto: ` SELECT * FROM programs_data ` [19:23:49] 10Data-Engineering-Radar, 10API Platform, 10Platform Engineering Roadmap: Retroactively fix logging to use a RequestScopedLogger where applicable - https://phabricator.wikimedia.org/T305504 (10FGoodwin) 05In progress→03Resolved [19:23:51] 10Data-Engineering, 10API Platform, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Implement pageviews endpoints - https://phabricator.wikimedia.org/T288296 (10FGoodwin) [19:28:38] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q4: rack/setup/install stat1010 - https://phabricator.wikimedia.org/T307399 (10Jclark-ctr) stat1010 E1 u24 cableid # 20220077 port24 [19:30:24] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q4: rack/setup/install stat1010 - https://phabricator.wikimedia.org/T307399 (10Jclark-ctr) [19:30:57] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q4: rack/setup/install stat1010 - https://phabricator.wikimedia.org/T307399 (10Jclark-ctr) a:05Jclark-ctr→03Cmjohnson [20:40:50] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q4: rack/setup/install stat1010 - https://phabricator.wikimedia.org/T307399 (10Jclark-ctr) @BTullis please confirm if New rows E- F are ok for this host. [20:41:46] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q4:(Need By: TBD) rack/setup/install an-presto10[06-15].eqiad.wmnet - https://phabricator.wikimedia.org/T306835 (10Jclark-ctr) @BTullis please confirm racking instructions and if New rows E- F are ok racking [21:00:37] 10Data-Engineering: Drop UploadWizard* data - https://phabricator.wikimedia.org/T305556 (10MarkTraceur) Yeah, it should be okay to drop the tables at this point. I don't see much need to keep them around. [21:04:08] 10Quarry: Quarry exports integers as floats to wikitable - https://phabricator.wikimedia.org/T151106 (10Abstract09) It happens in all formats but it looks like Quarry doesn't do anything about data type here, the integer in this example query is from the 'rank' function in SQL, and when we fetch in Quarry it is... [21:17:26] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q4:(Need By: TBD) rack/setup/install an-presto10[06-15].eqiad.wmnet - https://phabricator.wikimedia.org/T306835 (10cmooney) Should be good for rows E and F if that works for the team. [21:17:43] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q4: rack/setup/install stat1010 - https://phabricator.wikimedia.org/T307399 (10cmooney) These should be ok for rows E/F if that suits the team. [21:56:23] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Add better monitoring for Analytics UIs - https://phabricator.wikimedia.org/T277729 (10Dzahn) Brett fixed https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=an-tool1007&service=Check+Turnilo+node+appserver Now you _actually_ have the moni... [22:05:57] 10Quarry: Quarry exports integers as floats to wikitable - https://phabricator.wikimedia.org/T151106 (10rook) @Abstract09 Could you include some code links on where we fetch this in quarry as a float? I believe rank would only return an int, so we're probably processing it generally with a float rather than spe... [22:34:41] 10Data-Engineering, 10Data-Engineering-Kanban: Procure MaxMind GeoIP2 Database License - https://phabricator.wikimedia.org/T303453 (10Dzahn) >>! In T303453#7766646, @phuedx wrote: > #anti-harassment procured a subscription for the GeoIP2 Enterprise and Anonymous IP databases, presumably using a separate accoun... [22:52:39] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q4: rack/setup/install stat1010 - https://phabricator.wikimedia.org/T307399 (10BTullis) Yes, rows E and F are fine for this, thanks. [22:53:23] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q4:(Need By: TBD) rack/setup/install an-presto10[06-15].eqiad.wmnet - https://phabricator.wikimedia.org/T306835 (10BTullis) Yes, rows E and F are fine for these presto servers, thanks. [22:54:05] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q4:(Need By: TBD) rack/setup/install an-presto10[06-15].eqiad.wmnet - https://phabricator.wikimedia.org/T306835 (10BTullis)