[12:58:30] <wikibugs>	 10Quarry: See if query is running or complete in browser tab - https://phabricator.wikimedia.org/T316307 (10rook) 05Open→03In progress a:03rook
[13:05:15] <wikibugs>	 10Quarry: See if query is running or complete in browser tab - https://phabricator.wikimedia.org/T316307 (10rook) https://github.com/toolforge/quarry/pull/7
[13:36:44] <wikibugs>	 (03CR) 10Joal: [C: 03+1] "I didn't check everything, the ones I check are were correct - Let's do that!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/830260 (owner: 10Milimetric)
[13:37:38] <wikibugs>	 (03CR) 10Milimetric: [V: 03+2 C: 03+2] "+1 + +1 = +2?  :P" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/830260 (owner: 10Milimetric)
[13:38:03] <joal>	 \o/
[13:39:01] <milimetric>	 I love me a good purge :P
[13:39:03] <joal>	 milimetric: added to the deployment plan
[13:39:13] <milimetric>	 beat me to it, ty!
[13:39:25] <joal>	 milimetric: I wonder how faster it'll make the deploy  - Thank you so much!
[13:40:10] <milimetric>	 hopefully it doesn't break it, but should be fast yea, looking forward to it
[13:54:11] <wikibugs>	 (03PS14) 10Joal: Update refine to use Iceberg for event_sanitize [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/811212 (https://phabricator.wikimedia.org/T311739)
[14:03:56] <wikibugs>	 10Quarry: Comment on phabricator task on github update - https://phabricator.wikimedia.org/T317566 (10Aklapper)
[14:47:35] <wikibugs>	 10Quarry: Comment on phabricator task on github update - https://phabricator.wikimedia.org/T317566 (10github-toolforge-bot) vivian-rook synchronize https://github.com/toolforge/quarry/pull/8
[14:50:45] <wikibugs>	 10Quarry: Comment on phabricator task on github update - https://phabricator.wikimedia.org/T317566 (10github-toolforge-bot) vivian-rook synchronize https://github.com/toolforge/quarry/pull/8
[15:00:45] <wikibugs>	 10Quarry: Comment on phabricator task on github update - https://phabricator.wikimedia.org/T317566 (10rook)
[15:04:17] <wikibugs>	 10Analytics-Wikistats, 10Data Engineering Planning, 10Data Pipelines: "Pages to date" not loading with "daily" metric - https://phabricator.wikimedia.org/T312717 (10Milimetric) I'm going to go out of process here and try to fix this.  It's not really ok for a production bug to sit around this long.
[15:19:48] <wikibugs>	 10Quarry: Comment on phabricator task on github update - https://phabricator.wikimedia.org/T317566 (10Aklapper)
[16:30:50] <jinxer-wm>	 (HdfsRpcQueueLength) firing: RPC call queue length on the analytics-hadoop cluster is too high. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Namenode_RPC_length_queue - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=54&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsRpcQueueLength
[16:36:06] <wikibugs>	 10Quarry: Comment on phabricator task on github update - https://phabricator.wikimedia.org/T317566 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/quarry/pull/9
[16:46:14] <joal>	 This hdfs alert is due to me doing some iceberg tests - I'm overwhelming the system, I'll keep monitoring
[16:46:26] <joal>	 btullis: in case you notice it --^
[16:59:15] <btullis>	 Thanks Joel. Just seeing it now.
[17:00:08] <joal>	 and actually I just realize I made a typo: I'm NOT overwhelming the system - the queue is just a bit high
[17:01:35] <btullis>	 Autocorrect made my typo joal :-)
[17:01:42] <joal>	 huhu :)
[17:15:50] <jinxer-wm>	 (HdfsRpcQueueLength) resolved: RPC call queue length on the analytics-hadoop cluster is too high. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Namenode_RPC_length_queue - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=54&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsRpcQueueLength
[17:45:34] <wikibugs>	 10Quarry: Comment on phabricator task on github update - https://phabricator.wikimedia.org/T317566 (10github-toolforge-bot) vivian-rook synchronize https://github.com/toolforge/quarry/pull/10
[17:50:19] <wikibugs>	 (03PS1) 10Milimetric: Fix some npm vulnerabilities [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/833064
[17:50:21] <wikibugs>	 (03PS1) 10Milimetric: Avoid infinite loop when loading pages-to-date [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/833065 (https://phabricator.wikimedia.org/T312717)
[18:04:50] <jinxer-wm>	 (HdfsRpcQueueLength) firing: RPC call queue length on the analytics-hadoop cluster is too high. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Namenode_RPC_length_queue - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=54&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsRpcQueueLength
[18:14:50] <jinxer-wm>	 (HdfsRpcQueueLength) resolved: RPC call queue length on the analytics-hadoop cluster is too high. - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hadoop/Alerts#HDFS_Namenode_RPC_length_queue - https://grafana.wikimedia.org/d/000000585/hadoop?var-hadoop_cluster=analytics-hadoop&orgId=1&panelId=54&fullscreen - https://alerts.wikimedia.org/?q=alertname%3DHdfsRpcQueueLength
[18:32:42] <wikibugs>	 10Quarry: Comment on phabricator task on github update - https://phabricator.wikimedia.org/T317566 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/quarry/pull/11
[19:16:25] <wikibugs>	 (03CR) 10Mforns: "Hey Xabriel! First of all, huge thanks for taking this very annoying task and implementing all changes with great precision! As I mention " [analytics/refinery] - 10https://gerrit.wikimedia.org/r/832336 (https://phabricator.wikimedia.org/T317124) (owner: 10Xcollazo)
[19:19:23] <wikibugs>	 (03PS1) 10Milimetric: Update webpack build [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/833090
[19:19:36] <wikibugs>	 (03CR) 10Milimetric: [C: 03+2] Update webpack build [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/833090 (owner: 10Milimetric)
[19:19:43] <wikibugs>	 (03CR) 10Milimetric: [C: 03+2] Avoid infinite loop when loading pages-to-date [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/833065 (https://phabricator.wikimedia.org/T312717) (owner: 10Milimetric)
[19:19:49] <wikibugs>	 (03CR) 10Milimetric: [C: 03+2] Fix some npm vulnerabilities [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/833064 (owner: 10Milimetric)
[19:20:57] <wikibugs>	 (03CR) 10Milimetric: [V: 03+2 C: 03+2] Update webpack build [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/833090 (owner: 10Milimetric)
[19:28:49] <icinga-wm>	 PROBLEM - SSH on analytics1077.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[19:45:13] <wikibugs>	 10Data-Engineering, 10Product-Analytics: Millions of "user" page views to a single French Wikipedia article - https://phabricator.wikimedia.org/T318115 (10nshahquinn-wmf)
[19:45:49] <wikibugs>	 10Data-Engineering, 10Product-Analytics: Millions of "user" page views to a single French Wikipedia article - https://phabricator.wikimedia.org/T318115 (10nshahquinn-wmf) @odimitrijevic asked whether someone from Product Analytics could investigate.
[20:51:36] <wikibugs>	 10Quarry: Comment on phabricator task on github update - https://phabricator.wikimedia.org/T317566 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/quarry/pull/11
[20:52:19] <wikibugs>	 10Quarry: Comment on phabricator task on github update - https://phabricator.wikimedia.org/T317566 (10rook) 05Open→03Resolved
[21:12:26] <wikibugs>	 10Analytics-Radar, 10Domains, 10SRE, 10Traffic-Icebox, 10WMF-General-or-Unknown: Don't set cookies in traffic layer for non-user facing domains (avoid false third-party cookie warning) - https://phabricator.wikimedia.org/T262996 (10BCornwall) a:03BCornwall
[21:15:05] <wikibugs>	 (03PS1) 10Nettrom: Add Welcome Survey reminder to the module list [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/833108 (https://phabricator.wikimedia.org/T310320)
[21:28:28] <wikibugs>	 10Data-Engineering, 10Product-Analytics: Millions of "user" page views to a single French Wikipedia article - https://phabricator.wikimedia.org/T318115 (10Vahurzpu)
[21:56:57] <icinga-wm>	 RECOVERY - SSH on analytics1077.mgmt is OK: SSH OK - OpenSSH_7.4 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[22:20:23] <wikibugs>	 (03PS1) 10Milimetric: Fix UI component build [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/833122
[22:20:25] <wikibugs>	 (03PS1) 10Milimetric: Release 2.9.6 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/833123
[22:20:40] <wikibugs>	 (03CR) 10Milimetric: [C: 03+2] Release 2.9.6 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/833123 (owner: 10Milimetric)
[22:20:43] <wikibugs>	 (03CR) 10Milimetric: [C: 03+2] Fix UI component build [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/833122 (owner: 10Milimetric)
[22:23:47] <wikibugs>	 (03Merged) 10jenkins-bot: Fix UI component build [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/833122 (owner: 10Milimetric)
[22:23:49] <wikibugs>	 (03Merged) 10jenkins-bot: Release 2.9.6 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/833123 (owner: 10Milimetric)
[22:28:06] <milimetric>	 !log Wikistats: improved build a little and deployed fix to T312717
[22:28:08] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[22:28:09] <stashbot>	 T312717: "Pages to date" not loading with "daily" metric - https://phabricator.wikimedia.org/T312717
[22:30:42] <wikibugs>	 10Analytics-Wikistats, 10Data Engineering Planning, 10Data Pipelines: "Pages to date" not loading with "daily" metric - https://phabricator.wikimedia.org/T312717 (10Milimetric) a:03Milimetric @Nevmit: [[ https://stats.wikimedia.org/#/tr.wikipedia.org/content/pages-to-date/normal|line|2-year|~total|daily |...
[22:35:57] <wikibugs>	 10Analytics-Wikistats, 10Data Engineering Planning, 10Data Pipelines: "Pages to date" not loading with "daily" metric - https://phabricator.wikimedia.org/T312717 (10Milimetric) For the record, this was an interesting bug.  I'll describe here for anyone interested.  The code was doing something like `(get all...