[00:45:58] 10Quarry, 10Documentation: Improve Superset documentation - https://phabricator.wikimedia.org/T337342 (10EpicPupper) [00:46:02] 10Quarry, 10cloud-services-team (FY2022/2023-Q4): Move Quarry to be an installation of Superset - https://phabricator.wikimedia.org/T169452 (10EpicPupper) [00:54:15] 10Quarry, 10cloud-services-team (FY2022/2023-Q4): Move Quarry to be an installation of Superset - https://phabricator.wikimedia.org/T169452 (10EpicPupper) Just wanted to add that Superset is fantastic and adds so many great features! Graphs, previewing table schemas, dashboards, datasets, oh my... [01:18:41] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Liz) Any status updates on this issue? Replag is at 19 hours now. [02:28:13] (DiskSpace) firing: Disk space stat1008:9100:/ 5.912% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=stat1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [05:04:09] 10Quarry, 10Documentation, 10User-EpicPupper: Improve Superset documentation - https://phabricator.wikimedia.org/T337342 (10EpicPupper) [05:11:03] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) We are still working on getting the sanitarium hosts sanitized, should be finished in the next few hours. Once that is done, they'll need to catch up with t... [05:11:48] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [05:19:46] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [05:24:25] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [05:25:49] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [06:28:13] (DiskSpace) firing: Disk space stat1008:9100:/ 5.955% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=stat1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [06:36:57] !log `truncate /var/log/kerberos/krb5kdc.log -s 10g` on krb1001 to avoid the root partition to fill up [06:36:58] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:42:19] !log `apt-get clean` on stat1008 to clean up some space in the root partition [06:42:20] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [06:43:13] (DiskSpace) resolved: Disk space stat1008:9100:/ 5.955% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=stat1008 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [06:43:34] Thnanks elukey - do you have an idea as to why this log fills up? Should rotate/delete it more frequently, or remove the level of verbosity of the logger? [06:43:47] elukey: many thanks for these ops [06:44:48] joal: if I remember correctly, one cause of the increased Kerberos logging was the increased presto cluster size. Not sure of any others though. [06:45:05] hello folks! Moritz is already checking :) [06:45:30] for stat1008 maybe there is some extra cleanup needed, the root is around 80% now [06:45:39] I'm traveling to London today, so I'll be online, but a bit patchy. [06:46:02] ack, all good atm, I'll check if others alarms are raised btullis [06:46:52] I think there is another copy of conda-analytics in /tmp that can be deleted from all stats servers. [06:52:35] for kerberos logs I think there's two separate things, for once there's really a lot of requests from Presto servers, they hammer the KDCs a lot [06:53:21] IIRC Nicholas checked with upstream about it, but there are no plans to implement any local caching by them [06:55:52] there are also warning about some deprecated crypto settings, which are probably related to the recent updates of the KDC [06:56:04] 10Data-Engineering, 10Data Pipelines (Sprint 13): Druid Webrequest sampled 128 has missing data data for 1 hour - https://phabricator.wikimedia.org/T337088 (10JAllemandou) Thanks a lot for having raised the issue, this is now solved (see parent ticket). [06:56:20] likely the Kerberos release in Buster didn't yet warn about these, but now does which leads to more logs in general [06:56:27] I'll file a task about this [06:58:59] Ack, thanks moritzm. [07:36:04] 10Data-Engineering: stat1008's /srv partition is getting full due to home dirs - https://phabricator.wikimedia.org/T337246 (10MGerlach) `mgerlach`: 447G -> 182G [08:04:27] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [08:29:09] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A): Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10dcausse) >>! In T328925#8881625, @Ottomata wrote: > @dcausse @bking @gmodena @tchin I'm feeling good about this [[ https://grafana-rw.wikimedia.org/d/K9x0c4aV... [08:38:41] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10MusikAnimal) [08:54:27] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [09:42:14] 10Data-Engineering, 10Infrastructure-Foundations, 10SRE: Investigate crypto deprecations after Bullseye update - https://phabricator.wikimedia.org/T337544 (10MoritzMuehlenhoff) [09:42:26] 10Data-Engineering, 10Infrastructure-Foundations, 10SRE: Investigate crypto KDC deprecations after Bullseye update - https://phabricator.wikimedia.org/T337544 (10MoritzMuehlenhoff) p:05Triage→03Medium [09:43:53] for the Kerberos deprecation warnings I created https://phabricator.wikimedia.org/T337544, we can maybe start refreshing principals/keytabs for the Hadoop test cluster next week [09:51:40] that should tell us the scope (whether this also needs additional config settings or just refreshed keys) [09:55:56] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [09:56:34] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [09:59:11] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [11:23:09] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) @BTullis can you please set up the views on clouddb1021:3315 for all databases? The users should be there as I have copied them from the other clouddb hosts... [11:23:41] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [11:36:28] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) @BTullis can you please set up the views on clouddb1021:3312 for all databases? The users should be there as I have copied them from the other clouddb hosts... [11:36:53] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [11:43:07] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, 10Patch-For-Review: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [11:44:39] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, 10Patch-For-Review: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) The idea is to fully reclone clouddb1021 for all the sections, with grants and `p` databases and once all the views are created there... [12:43:11] (03PS3) 10Amire80: languages.json: Consistent indenting [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/921161 (owner: 10Reedy) [12:56:23] 10Data-Engineering, 10Event-Platform Value Stream, 10Machine-Learning-Team: Create new mediawiki.page_links_change stream based on fragment/mediawiki/state/change/page - https://phabricator.wikimedia.org/T331399 (10daniel) Quick recap on what LinkTarget does... MediaWiki distinguishes between several differ... [12:59:43] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A): Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10Ottomata) > flink_taskmanager_job_task_operator_lastSeenRev_rocksdb_estimate_live_data_size Oh I see, yeah, that's too bad. I have been able to parameterize... [13:00:32] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A): Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10Ottomata) @dcausse also, the Kafka specific panels assume the Kafka Source and Sinks, not the older Source/SinkFunctions, so I think not all of the panels the... [13:05:38] 10Data-Engineering, 10Event-Platform Value Stream, 10Machine-Learning-Team: Create new mediawiki.page_links_change stream based on fragment/mediawiki/state/change/page - https://phabricator.wikimedia.org/T331399 (10Ottomata) Aye, thanks Daniel! This is great. Similar to but with more detail to [[ https://pha... [13:29:12] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) @BTullis can you please set up the views on clouddb1021:3311 for all databases (there is just one)? The users should be there as I have copied them from the... [13:29:32] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [14:17:10] 10Quarry, 10Documentation, 10User-EpicPupper: Improve Superset documentation - https://phabricator.wikimedia.org/T337342 (10rook) [14:17:15] 10Quarry, 10cloud-services-team (FY2022/2023-Q4): Move Quarry to be an installation of Superset - https://phabricator.wikimedia.org/T169452 (10rook) [14:36:43] (SystemdUnitFailed) firing: rsync-published.service Failed on stat1009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:41:31] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Legoktm) [14:45:44] 10Data-Engineering, 10AQS2.0, 10API Platform (AQS 2.0 Roadmap), 10Epic, and 2 others: Page analytics: per-article should return expected data for monthly granularity - https://phabricator.wikimedia.org/T337564 (10FGoodwin) [15:01:18] (03PS1) 10DCausse: [WIP] Use eventutilites shaded jar [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/923646 (https://phabricator.wikimedia.org/T337421) [15:04:59] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: produce_canary_events.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [15:06:35] (03CR) 10CI reject: [V: 04-1] [WIP] Use eventutilites shaded jar [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/923646 (https://phabricator.wikimedia.org/T337421) (owner: 10DCausse) [15:06:43] (SystemdUnitFailed) firing: (2) produce_canary_events.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:08:31] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-views run by nskaggs: Started updating wikireplica views [15:15:53] RECOVERY - Check systemd state on an-launcher1002 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [15:16:07] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A): Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10dcausse) >>! In T328925#8882933, @Ottomata wrote: > @dcausse also, the Kafka specific panels assume the Kafka Source and Sinks, not the older Source/SinkFunct... [15:18:01] (03PS2) 10DCausse: [WIP] Use eventutilites shaded jar [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/923646 (https://phabricator.wikimedia.org/T337421) [15:21:43] (SystemdUnitFailed) firing: (2) produce_canary_events.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:22:35] (03CR) 10CI reject: [V: 04-1] [WIP] Use eventutilites shaded jar [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/923646 (https://phabricator.wikimedia.org/T337421) (owner: 10DCausse) [15:31:54] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10ops-monitoring-bot) Cookbook cookbooks.sre.wikireplicas.update-views for section s7 started by nskaggs executed with errors: - dbproxy1018.eqiad.wmnet (**FAIL**) - Co... [15:42:52] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) >>! In T337446#8883466, @ops-monitoring-bot wrote: > Cookbook cookbooks.sre.wikireplicas.update-views for section s7 started by nskaggs executed with errors... [16:17:10] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10CodeReviewBot) otto opened https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/56 EventProcessFu... [16:17:21] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10CodeReviewBot) [16:34:13] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10Ottomata) > flink_taskmanager_job_task_operator_lastSeenRev_rocksdb_estimate_live_data_size @dcausse, maybe you know this, but TIL for... [17:01:44] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10CodeReviewBot) otto merged https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/56 EventProcessFu... [17:38:13] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10CodeReviewBot) otto opened https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/58 Upgrade t... [17:38:22] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10CodeReviewBot) [17:42:29] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10CodeReviewBot) otto merged https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/58 Upgrade t... [18:26:17] 10Quarry, 10Cloud-VPS: List of admin users for superset? - https://phabricator.wikimedia.org/T337578 (10rook) [18:26:32] 10Quarry, 10Cloud-VPS: List of admin users for superset? - https://phabricator.wikimedia.org/T337578 (10rook) [18:42:24] (03PS3) 10Xcollazo: Add iceberg version of referrer_daily table. [analytics/refinery] - 10https://gerrit.wikimedia.org/r/917404 (https://phabricator.wikimedia.org/T335305) [18:45:04] (03CR) 10Xcollazo: [V: 03+2] "Ok modified target database as per conversations at T337562." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/917404 (https://phabricator.wikimedia.org/T335305) (owner: 10Xcollazo) [19:00:05] 10Data-Engineering, 10serviceops, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: New Service Request: flink-kubernetes-operator - https://phabricator.wikimedia.org/T333464 (10Ottomata) [[ https://grafana-rw.wikimedia.org/d/H-sRgqLVk/flink-kubernetes-operator?orgId=1&from=now-7d&to=now&var... [19:16:54] 10Quarry, 10Cloud-VPS: List of admin users for superset? - https://phabricator.wikimedia.org/T337578 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/superset-deploy/pull/4 [19:19:16] 10Quarry, 10Cloud-VPS: List of admin users for superset? - https://phabricator.wikimedia.org/T337578 (10rook) Seems to set my user as admin on account creation (not login). As a bonus gives other users "Alpha" and "sql_lab" roles on account creation. Thus may remove the need for the generation of the OAuth rol... [19:21:43] (SystemdUnitFailed) firing: rsync-published.service Failed on stat1009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:54:26] 10Quarry: Unable to login Quarry - https://phabricator.wikimedia.org/T337588 (10I) [19:57:21] 10Quarry: Unable to login Quarry - https://phabricator.wikimedia.org/T337588 (10I) After you follow this link, you will see a link that looks like this https://meta.wikimedia.org/w/index.php?title=Special%3AOAuth%2Fauthenticate&oauth_token=1c05c87a4c271b63cfebaf3dc936784c&oauth_consumer_key=cb1d846203c9197922573... [19:58:11] 10Quarry: Unable to login Quarry - https://phabricator.wikimedia.org/T337588 (10rook) This is perhaps T333043 [20:00:09] 10Quarry: Unable to login Quarry - https://phabricator.wikimedia.org/T337588 (10I) >>! In T337588#8884089, @rook wrote: > This is perhaps T333043 It may not be caused by this task. Because the error does not occur at the time of the call back, but before. [20:01:20] 10Quarry: Unable to login Quarry - https://phabricator.wikimedia.org/T337588 (10rook) Following the silly advice in T333043 could you try to log in (perhaps in a fresh private window) about 10 times and see if any are a success? [20:03:25] 10Quarry: Unable to login Quarry - https://phabricator.wikimedia.org/T337588 (10I) >>! In T337588#8884095, @rook wrote: > Following the silly advice in T333043 could you try to log in (perhaps in a fresh private window) about 10 times and see if any are a success? I am in mainland China and therefore cannot ful... [20:05:11] 10Quarry: [bug] "Internal Server Error" when logging into Quarry - https://phabricator.wikimedia.org/T333043 (10I) There may not be this problem now, because the problem occurred in the previous step, the reason is that the callback URL could not be found. [20:06:27] 10Quarry: Unable to login Quarry - https://phabricator.wikimedia.org/T337588 (10rook) >>! In T337588#8884097, @I wrote: > I am in mainland China and therefore cannot fully answer these questions at the official site, however when I try to pass OAuth Checker, a gadget room I developed myself, each attempt shows t... [20:08:08] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10CodeReviewBot) otto opened https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/57 EventProcessFu... [20:08:17] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10CodeReviewBot) [20:09:17] 10Quarry: Unable to login Quarry - https://phabricator.wikimedia.org/T337588 (10I) >>! In T337588#8884102, @rook wrote: > Could you describe a little more on what you're looking to do? For instance is a login necessary for your task? https://quarry.wmcloud.org/query/73899 should be accessible without login for i... [20:11:13] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10CodeReviewBot) otto merged https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/57 EventProcessFu... [20:13:26] 10Quarry: Unable to login Quarry - https://phabricator.wikimedia.org/T337588 (10rook) >>! In T337588#8884109, @I wrote: > I cannot view the results of this query. I'm trying to score a fork so I can see what's going on, and this glitch prevents me from doing that. I apologize I don't follow. You're trying to lo... [20:17:23] 10Quarry: Unable to login Quarry - https://phabricator.wikimedia.org/T337588 (10I) >>! In T337588#8884125, @rook wrote: >>>! In T337588#8884109, @I wrote: >> I cannot view the results of this query. I'm trying to score a fork so I can see what's going on, and this glitch prevents me from doing that. > > I apolo... [20:20:41] 10Quarry: Unable to login Quarry - https://phabricator.wikimedia.org/T337588 (10rook) @I Why do you need the callback URL. It sounds like you're trying to get an authenticated quarry connection though something other than a browser. Which is fine, but not something that quarry normally does. Is that what you're... [20:25:50] 10Quarry: Unable to login Quarry - https://phabricator.wikimedia.org/T337588 (10I) >>! In T337588#8884132, @rook wrote: > @I Why do you need the callback URL? It sounds like you're trying to get an authenticated quarry connection though something other than a browser. Which is fine, but not something that quarry... [20:28:01] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10CodeReviewBot) otto opened https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/59 Bump to e... [20:32:04] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10CodeReviewBot) otto merged https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/59 Bump to e... [21:23:29] 10Quarry, 10cloud-services-team (FY2022/2023-Q4): Move Quarry to be an installation of Superset - https://phabricator.wikimedia.org/T169452 (10Smartse) The ability to search for tables and then get a preview in Superset is really useful. @Liz It's super easy to copy over - just head to https://superset.wmclo... [23:21:43] (SystemdUnitFailed) firing: rsync-published.service Failed on stat1009:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed