[06:53:26] 10Data-Engineering: Check home/HDFS leftovers of eyener - https://phabricator.wikimedia.org/T316072 (10MoritzMuehlenhoff) [08:34:33] 10Data-Engineering, 10Data-Engineering-Operations, 10SRE, 10SRE-Access-Requests: Access request to analytics system(s) - https://phabricator.wikimedia.org/T315409 (10Ladsgroup) The dumps can be also accessed from WMCS. [09:13:05] (03CR) 10Joal: "One comment about the commit message, then comments on 4 daily hql files. Most of the comments seem to apply to all files, I let you do th" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/812095 (https://phabricator.wikimedia.org/T311507) (owner: 10NOkafor) [09:44:45] 10Data-Engineering, 10Discovery-Search, 10SRE-Access-Requests: Production Shell access for Peter - https://phabricator.wikimedia.org/T316090 (10Gehel) [09:49:00] 10Data-Engineering, 10Discovery-Search, 10SRE, 10SRE-Access-Requests: Production Shell access for Peter - https://phabricator.wikimedia.org/T316090 (10Gehel) [09:50:04] 10Data-Engineering, 10Discovery-Search, 10SRE, 10SRE-Access-Requests: Production Shell access for Peter - https://phabricator.wikimedia.org/T316090 (10Gehel) As Peter's manager, and owner of the Search and W[CD]QS services, I'm approving this request. [10:39:42] 10Data-Engineering, 10Equity-Landscape: Load language data - https://phabricator.wikimedia.org/T315886 (10ntsako) Loaded ` SELECT * FROM ntsako.brief_projects_edited_metrics WHERE year=2021 SELECT * FROM ntsako.official_language_metrics SELECT * FROM ntsako.unesco_endangered_lang_metrics WHERE ye... [11:48:48] 10Data-Engineering, 10Equity-Landscape: Load country data - https://phabricator.wikimedia.org/T310712 (10ntsako) Hi @JAnstee_WMF Please can you review this. Table loaded is: ` SELECT * FROM ntsako.country_meta_data; ` [11:49:05] 10Data-Engineering, 10Equity-Landscape: Load country data - https://phabricator.wikimedia.org/T310712 (10ntsako) a:05ntsako→03JAnstee_WMF [12:43:17] 10Data-Engineering, 10Event-Platform Value Stream: Document and Promote Image Suggestions Feedback > Cassandra Flink Job - https://phabricator.wikimedia.org/T316112 (10lbowmaker) [12:44:22] 10Data-Engineering, 10Epic, 10Event-Platform Value Stream (Sprint 00): Integrate Image Suggestions Feedback with Cassandra - https://phabricator.wikimedia.org/T306627 (10lbowmaker) [12:45:03] 10Data-Engineering, 10Event-Platform Value Stream: Document and Promote Image Suggestions Feedback > Cassandra Flink Job - https://phabricator.wikimedia.org/T316112 (10lbowmaker) [12:59:07] 10Data-Engineering, 10Discovery-Search: Production Shell access for Peter - https://phabricator.wikimedia.org/T316090 (10Ladsgroup) Hi, I remove SRE tags so it doesn't clutter our dashboards yet. Please re-add them once the request has the information. [13:04:45] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 00): Remove materialized .json files from event schema repositories - https://phabricator.wikimedia.org/T315674 (10Ottomata) [13:20:17] 10Data-Engineering, 10Discovery-Search: Production Shell access for Peter - https://phabricator.wikimedia.org/T316090 (10Ottomata) Approved for analytics-privatedata-users and analytics-search-users. [14:19:16] (03PS11) 10NOkafor: Add cassandra loading queries for aiflow in the hql folder Bug: T311507 [analytics/refinery] - 10https://gerrit.wikimedia.org/r/812095 (https://phabricator.wikimedia.org/T311507) [14:39:38] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 00): Remove materialized .json files from event schema repositories - https://phabricator.wikimedia.org/T315674 (10DLynch) > The schemas are often linked for humans to read. Perhaps if we had a nice UI for them (or Datahub integration), we wouldn't do t... [14:57:53] Any chance this error I'm seeing on the new clouddumps servers is a kerberos thing? [14:58:03] https://phabricator.wikimedia.org/T316123 [14:59:33] andrewbogott: AFAIK clouddumps need to connect to HDFS to pull data, and therefore need some kerberos creds - But I think there is such thing as kerberos for user auth [15:01:21] Yeah, I was wondering if it's a side-effect of the hdfs/kerberos thing -- like maybe something needs to be activated or registered or similar [15:07:13] andrewbogott: could be, but I'm afraid I can't really help on that :S [15:07:29] moritz seems to know what's happening, over in _security [15:07:43] There is a dumpsgen user with a keytab on that box. [15:08:22] ack andrewbogott - thanks for letting me know [15:45:37] 10Data-Engineering, 10Data-Engineering-Operations, 10SRE, 10SRE-Access-Requests: Access request to analytics system(s) - https://phabricator.wikimedia.org/T315409 (10Tchanders) > Is this monthly data dump script something that runs in Hadoop or perhaps on the stat boxes? If so, analytics-privatedata-users... [16:05:30] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q4:(Need By: TBD) rack/setup/install an-presto10[06-15].eqiad.wmnet - https://phabricator.wikimedia.org/T306835 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1001 for host an-presto1007.eqiad.wmnet with OS bullseye [16:15:42] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q4:(Need By: TBD) rack/setup/install an-presto10[06-15].eqiad.wmnet - https://phabricator.wikimedia.org/T306835 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1001 for host an-presto1007.eqiad.wmnet with OS bullseye exec... [16:17:34] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q4:(Need By: TBD) rack/setup/install an-presto10[06-15].eqiad.wmnet - https://phabricator.wikimedia.org/T306835 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by btullis@cumin1001 for host an-presto1009.eqiad.wmnet with OS bullseye [16:36:31] (03PS12) 10NOkafor: Add cassandra loading queries for aiflow in the hql folder [analytics/refinery] - 10https://gerrit.wikimedia.org/r/812095 (https://phabricator.wikimedia.org/T311507) [17:06:16] 10Data-Engineering, 10DC-Ops, 10SRE, 10ops-eqiad: Q4:(Need By: TBD) rack/setup/install an-presto10[06-15].eqiad.wmnet - https://phabricator.wikimedia.org/T306835 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by btullis@cumin1001 for host an-presto1009.eqiad.wmnet with OS bullseye exec... [18:01:41] (03CR) 10Joal: "Still some inconsistencies - I pointed everything I found in all files for references, so that you can refer to it one by one." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/812095 (https://phabricator.wikimedia.org/T311507) (owner: 10NOkafor) [18:32:21] 10Data-Engineering, 10Data-Engineering-Operations, 10SRE, 10SRE-Access-Requests: Access request to analytics system(s) - https://phabricator.wikimedia.org/T315409 (10gmodena) >>! In T315409#8181985, @Tchanders wrote: >> Is this monthly data dump script something that runs in Hadoop or perhaps on the stat b... [19:01:13] anyone got time to help me add a plugin to airflow and configure something? [19:01:26] https://datahubproject.io/docs/lineage/airflow/ [19:07:18] ah, I see https://github.com/wikimedia/puppet/blob/85604a1934d48fa610ee183a36cb5fea7d87823c/modules/airflow/manifests/instance.pp, I'll send a change [19:07:24] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [19:08:46] PROBLEM - Check unit status of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [19:16:40] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [19:20:08] RECOVERY - Check unit status of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [19:34:29] 10Analytics-Radar, 10Machine-Learning-Team, 10SRE: Using docker in WMF production network outside of kubernetes - https://phabricator.wikimedia.org/T275551 (10gmodena) >>! In T275551#8178081, @Ottomata wrote: >> will it be possible to consume e.g. events from kafka infra, or read/write to swift? > Nopers :/... [19:52:05] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [19:56:28] PROBLEM - Check unit status of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [20:01:32] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [20:02:48] (03PS1) 10Bearloga: [WIP] Retroactively add http.client_ip to Android schemas [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/826373 [20:03:41] (03CR) 10CI reject: [V: 04-1] [WIP] Retroactively add http.client_ip to Android schemas [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/826373 (owner: 10Bearloga) [20:07:50] RECOVERY - Check unit status of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [20:23:04] 10Data-Engineering, 10Equity-Landscape: Load country data - https://phabricator.wikimedia.org/T310712 (10JAnstee_WMF) @ntsako Looks good - I would suggest we drop the 2 columns which have become fully redundant following the list cleaning a while back: un_iso3_alpha_code exclusive_code These are now fully red... [20:24:07] 10Data-Engineering, 10Equity-Landscape: Load country data - https://phabricator.wikimedia.org/T310712 (10JAnstee_WMF) a:05JAnstee_WMF→03ntsako [20:30:02] 10Data-Engineering: Add the requestctl element of the x-analytics map to turnlio's webrequest_sampled_128 - https://phabricator.wikimedia.org/T314578 (10CDanis) also cc @EChetty [21:00:15] 10Data-Engineering: Add the requestctl element of the x-analytics map to turnlio's webrequest_sampled_128 - https://phabricator.wikimedia.org/T314578 (10Ottomata) @CDanis if you want to give it a go https://github.com/wikimedia/analytics-refinery/tree/master/oozie/webrequest/druid has the files you need to mod... [21:03:15] (03PS2) 10Sharvaniharan: [WIP] Retroactively add http.client_ip to Android schemas [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/826373 (owner: 10Bearloga) [21:08:06] 10Data-Engineering-Kanban, 10Data Engineering Planning, 10Data Pipelines (Sprint 00): Create conda-base-env with last pyspark - https://phabricator.wikimedia.org/T309227 (10Ottomata) @Antoine_Quhen something to think about, is if we will include the conda `pkgs` dir and files in this base conda env debian.... [21:10:25] (03PS3) 10Sharvaniharan: Retroactively add http.client_ip to Android schemas [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/826373 (owner: 10Bearloga) [21:11:05] (03CR) 10Sharvaniharan: "Hi @Ottomata. Please let me know if it all looks good :-)" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/826373 (owner: 10Bearloga) [21:12:03] (03PS4) 10Sharvaniharan: Retroactively add http.client_ip to Android schemas [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/826373 (https://phabricator.wikimedia.org/T316047) (owner: 10Bearloga) [22:55:26] 10Data-Engineering, 10Equity-Landscape: Load country data - https://phabricator.wikimedia.org/T310712 (10Mayakp.wiki) Hi @ntsako / @JAnstee_WMF is there a difference between these 2 columns? or are they redundant as well ? - `iso2_country_code` - `iso3166_1_alpha_2_code` FYI, `iso3166_1_alpha_2_code...