[01:37:01] 10Analytics-Clusters, 10Analytics-Kanban: Set yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds - https://phabricator.wikimedia.org/T269616 (10Ottomata) Hm, I don't think this is working. I checked a [[ https://hue.wikimedia.org/hue/jobbrowser/#!id=0059995-210426062240701-oozie-oozi-W@denormal... [06:22:38] Good morning [06:38:01] bonjour [06:39:39] qq - there are warnings about the new aqs nodes in icinga since days ago (I tried to ping Hugh but I think he may be afk) [06:39:42] /analytics.wikimedia.org/v1/edits/per-page/{project}/{page-title}/{editor-type}/{granularity}/{start}/{end} (Get daily edits for english wikipedia page 0) is WARNING: Test Get daily edits for english wikipedia page 0 responds with unexpected value at path /items[0]/results => Expected 1 array elements, gotten 0 [06:40:03] I suppose this is ok due to the loading issue, should I ack? [06:40:45] Hi elukey - I think you're right, this is due to test data [06:40:53] elukey: I can quickly add test data if you wish [06:41:46] nono just acked the alerts, it was just to double check :) [09:35:33] elukey: my bad, I was afk on friday and missed your message. They're indeed safe to ack when the tables are empty [09:41:42] hnowlan: o/ nono please no issue at all, they were just warnings :) [10:51:07] 10Analytics, 10WMDE-Templates-FocusArea, 10WMDE-TechWish (Sprint-2021-02-03), 10WMDE-TechWish-Sprint-2021-05-26, 10WMDE-TechWish-Sprint-2021-06-09: Add missing normalization to CodeMirror Grafana board - https://phabricator.wikimedia.org/T273748 (10awight) [12:48:03] 10Analytics, 10Analytics-EventLogging, 10Performance-Team, 10Patch-For-Review: EventLogging background queue beforeunload event handler blocks Back-Forward cache - https://phabricator.wikimedia.org/T285220 (10Gilles) p:05Triage→03Medium [14:12:18] !log start updating analytics firewall rules to capirca generated ones on cr1-eqiad - T279429 [14:12:22] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:12:22] T279429: Audit analytics firewall filters - https://phabricator.wikimedia.org/T279429 [14:21:05] 10Analytics, 10WMDE-Templates-FocusArea, 10WMDE-TechWish (Sprint-2021-02-03), 10WMDE-TechWish-Sprint-2021-05-26, 10WMDE-TechWish-Sprint-2021-06-09: Add missing normalization to CodeMirror Grafana board - https://phabricator.wikimedia.org/T273748 (10mforns) Will do today @awight, sorry for making you wait. [14:28:40] !log remove decom hosts from the analytics firewall filter on cr1-eqiad - T279429 [14:28:43] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:28:43] T279429: Audit analytics firewall filters - https://phabricator.wikimedia.org/T279429 [14:29:22] 10Analytics, 10Analytics-Data-Quality, 10Datasets-Webstatscollector: Add alarms for high volume of views to pages with replacement characters - https://phabricator.wikimedia.org/T117945 (10mforns) @Nikerabbit I think we still could work on this in the future. However, as we have to prioritize it against othe... [14:35:20] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Metrics-Platform: wgEventStreams (EventStreamConfig) should support per wiki overrides - https://phabricator.wikimedia.org/T277193 (10mforns) @Ottomata I think it would be interesting to do this in the next quarter or two, because of mediawiki_client_s... [14:37:33] !log start updating analytics firewall rules to capirca generated ones on cr2-eqiad - T279429 [14:37:35] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:37:36] T279429: Audit analytics firewall filters - https://phabricator.wikimedia.org/T279429 [14:46:34] !log remove decom hosts from the analytics firewall filter on cr2-eqiad - T279429 [14:46:36] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:46:37] T279429: Audit analytics firewall filters - https://phabricator.wikimedia.org/T279429 [14:51:30] ottomata: all done for me, everything looks good to you? [14:54:18] XioNoX: looks ok from here so far anyway [14:54:19] thanks! [14:54:30] 10Analytics, 10Better Use Of Data, 10Event-Platform, 10Metrics-Platform: wgEventStreams (EventStreamConfig) should support per wiki overrides - https://phabricator.wikimedia.org/T277193 (10Ottomata) Agree, I think a general EventStreamConfig improvement focus would be good. Related - {T282004} - {T273235}... [14:55:02] XioNoX: o/ qq - what is the new procedure to say open an ip/port combination in analytics-in{6,4} [14:55:37] elukey: https://wikitech.wikimedia.org/wiki/Homer#How_to_use_it? [14:56:13] 10Analytics, 10observability: Need a list of AQS Kibana dashboards and searches - https://phabricator.wikimedia.org/T285318 (10colewhite) [14:56:42] elukey: so most of the cases, run https://netbox.wikimedia.org/extras/scripts/capirca.GetHosts/ and then run homer [14:59:14] 10Analytics, 10Infrastructure-Foundations, 10SRE, 10netops, 10Patch-For-Review: Audit analytics firewall filters - https://phabricator.wikimedia.org/T279429 (10ayounsi) 05Open→03Resolved [15:07:51] XioNoX: but do I have to modify a repo before? [15:08:42] elukey: for example if you add a new host to an existing rule, most likely no [15:09:28] if you add a new rule, yep [15:10:26] elukey: and the only file you should need to care about is https://github.com/wikimedia/operations-homer-public/blob/master/policies/cr-analytics.inc [15:11:30] ahh interesting thanks [15:39:44] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: page titles not decoding properly through the webrequest pipeline - https://phabricator.wikimedia.org/T285324 (10Milimetric) [15:54:33] Hi, I want to query this table, how can I know what is the most recent snapshot? https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Content/Wikidata_entity [15:57:16] joal: ^ [15:58:00] nvm [15:58:05] Found it with google [15:58:10] show partitions wmf.wikidata_entity; [15:58:42] the most recent one is 06-07, that's a bit old [15:58:48] Is it broken? [16:04:16] Amir1: Heya - It's not broken, I don't think :) [16:04:50] Amir1: It contains data for the week starting 2021-06-07 (meaning up to 2021-06-14), and the week after is the one just dinishing - new data should show up soon [16:05:02] oh I see [16:05:03] thanks [16:05:15] you're welcome Amir1 :) [16:59:23] 10Analytics, 10WMDE-Templates-FocusArea, 10WMDE-TechWish (Sprint-2021-02-03), 10WMDE-TechWish-Sprint-2021-05-26, 10WMDE-TechWish-Sprint-2021-06-09: Add missing normalization to CodeMirror Grafana board - https://phabricator.wikimedia.org/T273748 (10awight) >>! In T273748#7169617, @mforns wrote: > Will do... [17:28:58] 10Analytics-Clusters, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install an-web1001 - https://phabricator.wikimedia.org/T281787 (10RobH) [17:29:57] 10Analytics-Clusters, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install an-web1001 - https://phabricator.wikimedia.org/T281787 (10RobH) [18:09:27] 10Analytics-Clusters, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install an-web1001 - https://phabricator.wikimedia.org/T281787 (10RobH) [18:10:18] 10Analytics-Clusters, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install an-web1001 - https://phabricator.wikimedia.org/T281787 (10RobH) [18:19:04] milimetric: Heya - would you have a minute for me? [18:19:17] to the batcave joal ! [18:19:21] OMW [18:24:22] * razzi lunch [18:38:06] Gone for tonight team [18:38:18] bye jjjooaaaall :] [18:40:40] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: page titles in the webrequest pipeline - https://phabricator.wikimedia.org/T285324 (10Milimetric) [18:54:57] 10Analytics-Clusters, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install an-web1001 - https://phabricator.wikimedia.org/T281787 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts: ` an-web1001.eqiad.wmnet ` The log can be found in `/var/log/w... [19:06:21] 10Analytics-Clusters, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install an-web1001 - https://phabricator.wikimedia.org/T281787 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['an-web1001.eqiad.wmnet'] ` Of which those **FAILED**: ` ['an-web1001.eqiad.wmnet'] ` [19:10:47] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Top edited pages list on enwiktionary contains nonexistent pages with titles made up of question marks - https://phabricator.wikimedia.org/T284623 (10Milimetric) The apparent cause for this is our use of `convert(... using utf8)` which should be `conver... [19:43:22] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban: Import of MediaWiki tables into the Data Lakes mangles usernames - https://phabricator.wikimedia.org/T230915 (10Milimetric) Indeed this seems to be the same issue as in the other task, namely wrong charset used in sqooping. We should be using `utf8m... [19:55:22] 10Analytics-Clusters, 10DC-Ops, 10SRE, 10ops-eqiad, 10Patch-For-Review: (Need By: TBD) rack/setup/install an-web1001 - https://phabricator.wikimedia.org/T281787 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by robh on cumin1001.eqiad.wmnet for hosts: ` an-web1001.eqiad.wmnet ` The log can b... [20:20:03] 10Analytics, 10Analytics-EventLogging, 10Performance-Team, 10Patch-For-Review: EventLogging background queue beforeunload event handler blocks Back-Forward cache - https://phabricator.wikimedia.org/T285220 (10Krinkle) >>! In T285220#7167247, @DLynch wrote: > […] As pagehide is always fired before unload, a... [20:23:24] 10Analytics, 10Analytics-EventLogging, 10Performance-Team, 10Patch-For-Review: EventLogging background queue beforeunload event handler blocks Back-Forward cache - https://phabricator.wikimedia.org/T285220 (10DLynch) > Yes – unless there is code adding events from an unload handler itself, and/or otherwise... [20:26:37] 10Analytics-Clusters, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install an-web1001 - https://phabricator.wikimedia.org/T281787 (10RobH) [20:26:57] 10Analytics-Clusters, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install an-web1001 - https://phabricator.wikimedia.org/T281787 (10RobH) @Ottomata an-web1001 is now staged for your use! [20:27:00] 10Analytics-Clusters, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install an-web1001 - https://phabricator.wikimedia.org/T281787 (10RobH) 05Open→03Resolved [20:28:52] 10Analytics, 10MobileFrontend, 10Performance-Team (Radar), 10Readers-Web-Backlog (Tracking), 10Technical-Debt: Figure out XAnalytics stuff - https://phabricator.wikimedia.org/T190381 (10Jdlrobson) [20:38:19] 10Analytics-Clusters, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install an-web1001 - https://phabricator.wikimedia.org/T281787 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['an-web1001.eqiad.wmnet'] ` and were **ALL** successful. [20:42:00] 10Analytics-Clusters: Set up an-web1001 and decommission thorium - https://phabricator.wikimedia.org/T285355 (10Ottomata) [20:42:21] 10Analytics-Clusters: Set up an-web1001 and decommission thorium - https://phabricator.wikimedia.org/T285355 (10Ottomata) [20:42:32] 10Analytics-Kanban: Analytics Hardware for Fiscal Year 2020/2021 - https://phabricator.wikimedia.org/T255145 (10Ottomata) [20:52:09] (03CR) 10Mholloway: [WIP] Metrics Platform schema (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/676392 (https://phabricator.wikimedia.org/T276379) (owner: 10Jason Linehan) [21:06:53] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Services, 10Patch-For-Review: EventGate should use recent service-runner (^2.8.1) with Prometheus support - https://phabricator.wikimedia.org/T272714 (10Ottomata) Alright, https://gerrit.wikimedia.org/r/c/eventgate-wikimedia/+/700945 is ready for revie... [21:07:01] (03CR) 10Mholloway: [WIP] Metrics Platform schema (032 comments) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/676392 (https://phabricator.wikimedia.org/T276379) (owner: 10Jason Linehan) [21:07:36] (03PS1) 10Mforns: Advance start date of codemirror job [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/700990 (https://phabricator.wikimedia.org/T273748) [21:08:13] (03CR) 10Mforns: [V: 03+2 C: 03+2] "Self merging to unbreak production" [analytics/reportupdater-queries] - 10https://gerrit.wikimedia.org/r/700990 (https://phabricator.wikimedia.org/T273748) (owner: 10Mforns) [21:13:34] (03CR) 10Ottomata: [C: 03+1] "+1 from me, let's double check with Petr to see if any thoughts. If +1 from him I will merge." [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/700253 (https://phabricator.wikimedia.org/T285113) (owner: 10Urbanecm) [21:16:55] (03CR) 10Ottomata: [WIP] Metrics Platform schema (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/676392 (https://phabricator.wikimedia.org/T276379) (owner: 10Jason Linehan) [21:30:15] (03CR) 10Mholloway: [WIP] Metrics Platform schema (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/676392 (https://phabricator.wikimedia.org/T276379) (owner: 10Jason Linehan) [21:33:42] (03CR) 10Mholloway: [WIP] Metrics Platform schema (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/676392 (https://phabricator.wikimedia.org/T276379) (owner: 10Jason Linehan) [21:35:34] (03CR) 10Ppchelko: [C: 03+1] "No objections here, but I'll leave some comments on the accompanying change for core." [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/700253 (https://phabricator.wikimedia.org/T285113) (owner: 10Urbanecm) [21:41:31] (03CR) 10Mholloway: [WIP] Metrics Platform schema (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/676392 (https://phabricator.wikimedia.org/T276379) (owner: 10Jason Linehan) [21:59:38] (03CR) 10Mholloway: [WIP] Metrics Platform schema (032 comments) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/676392 (https://phabricator.wikimedia.org/T276379) (owner: 10Jason Linehan) [23:53:26] 10Analytics, 10Analytics-EventLogging, 10Performance-Team, 10Patch-For-Review: EventLogging background queue beforeunload event handler blocks Back-Forward cache - https://phabricator.wikimedia.org/T285220 (10Krinkle) I believe we're referring to scenarios like VisualEditor or NavigationTiming having instr...