[00:57:54] 10Data-Engineering, 10Product-Analytics: wikidata unique devices per-project-family overcount offset - https://phabricator.wikimedia.org/T301403 (10kzimmerman) @odimitrijevic and I discussed the priority for this and do not think it should be prioritized above current work. We're not regularly reporting on Wik... [05:20:09] (03PS1) 10Sharvaniharan: Add a required field in mobile_apps fragment [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/765671 [05:23:39] (03CR) 10Sharvaniharan: "Please review this patch at your convenience. Thank you :-)" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/765671 (owner: 10Sharvaniharan) [05:26:11] (03PS2) 10Sharvaniharan: Add a required field in mobile_apps fragment Added a new required field in a new fragment that will be used only by the android app. We will maintain any future android related fields here. [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/765671 (https://phabricator.wikimedia.org/T299239) [10:35:49] (03CR) 10Tchanders: [C: 03+1] "Looks good - thanks for working on this AGueyte" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/753548 (https://phabricator.wikimedia.org/T296415) (owner: 10AGueyte) [13:55:27] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-test-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-test-coord1001:10100 - https://alerts.wikimedia.org [14:15:27] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-test-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-test-coord1001:10100 - https://alerts.wikimedia.org [15:22:14] joal: I'm around if you wanna chat, I'm just cleaning up dashiki some more [15:31:37] (03PS2) 10Milimetric: Fix as many security vulnerabilities as possible [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/765647 [15:57:27] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-test-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-test-coord1001:10100 - https://alerts.wikimedia.org [16:12:27] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-test-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-test-coord1001:10100 - https://alerts.wikimedia.org [16:13:07] 10Analytics-Dashiki, 10Data-Engineering: Dashiki fixes needed for Pingback dashboard - https://phabricator.wikimedia.org/T298929 (10odimitrijevic) [16:38:14] (03PS1) 10Milimetric: Fix gulp build [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/766154 [16:38:31] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Fix as many security vulnerabilities as possible [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/765647 (owner: 10Milimetric) [16:38:46] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Fix gulp build [analytics/dashiki] - 10https://gerrit.wikimedia.org/r/766154 (owner: 10Milimetric) [16:40:00] anyone around? I'm debating releasing refinery just to sync up the new list of wikis, since sqooping will start Monday night so we won't have a lot of time [16:40:38] oh! It's Andrew's week, so I guess it's up to me and razzi. What do you think, razzi? [16:40:47] it's Friday and all, but it's just an hdfs-sync [16:42:13] oh! nvm, bigger problem: Data Loss ERROR - Workflow webrequest-load-check_sequence_statistics-wf-text-2022-2-25-15 [16:42:26] checking false positives, razzi [16:45:08] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, 10observability, and 2 others: Upgrade Kafka to 2.x - https://phabricator.wikimedia.org/T300102 (10EChetty) [16:47:05] 10Data-Engineering, 10Data-Engineering-Kanban, 10SRE, 10observability, and 2 others: Upgrade Kafka Risk Evaluation - https://phabricator.wikimedia.org/T302610 (10EChetty) [16:48:23] 10Data-Engineering, 10SRE, 10observability, 10serviceops, 10Epic: Upgrade Kafka to 2.x - https://phabricator.wikimedia.org/T300102 (10EChetty) [17:00:03] !log rerunning webrequest-load-wf-text-2022-2-25-15 after confirming all false positive loss [17:00:05] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [17:00:28] fyi data loss check was: [17:00:31] https://www.irccloud.com/pastebin/59SUpCh7/ [17:03:12] Hi milimetric! [17:03:51] sorry I was away at you messages time [17:04:00] Do you wish we sync up now on ops and others? [17:10:02] ping again milimetric? [17:32:52] sorry joal, was dealing with lunch and waiting for the job to fail [17:32:55] around now [17:32:59] (it failed again) [17:34:01] hey here too, reading scrollback [17:34:18] milimetric: in meeting now, then in music-rehearsal - may I ask if we could meet later in the day? [17:34:41] ofc joal, no rush, I got the webrequest failure, will run with force now [17:34:49] mforns: wanna sanity check me in bc? [17:34:54] yessss [17:34:55] omw [17:59:27] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-test-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-test-coord1001:10100 - https://alerts.wikimedia.org [18:08:10] job is fine after marcel and I restarted [18:14:27] milimetric: are you ok to switch opsweeks with Andrew? [18:14:27] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-test-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-test-coord1001:10100 - https://alerts.wikimedia.org [18:49:44] olja_: yep, sounds good, I'll watch for stuff breaking anyway [20:01:27] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-test-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-test-coord1001:10100 - https://alerts.wikimedia.org [20:11:27] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-test-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-test-coord1001:10100 - https://alerts.wikimedia.org [22:02:27] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-test-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-test-coord1001:10100 - https://alerts.wikimedia.org [22:07:27] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-test-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-test-coord1001:10100 - https://alerts.wikimedia.org [22:47:54] (03PS1) 10Mayakpwiki: movement_metrics: Migrate editing tables and ETL - editor_month [analytics/wmf-product/jobs] - 10https://gerrit.wikimedia.org/r/766196 (https://phabricator.wikimedia.org/T295332) [23:51:23] (03PS2) 10Mayakpwiki: movement_metrics: Migrate editing tables and ETL [analytics/wmf-product/jobs] - 10https://gerrit.wikimedia.org/r/766196 (https://phabricator.wikimedia.org/T295332)