[00:20:10] PROBLEM - Check systemd state on an-launcher1002 is CRITICAL: CRITICAL - degraded: The following units failed: monitor_refine_eventlogging_analytics.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:21:43] (SystemdUnitFailed) firing: monitor_refine_eventlogging_analytics.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:55:03] 10Data-Engineering, 10DBA, 10Data-Services, 10TaxonBot, and 3 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Liz) I'm not sure what's causing it (regarding s1), but I'm finding some bots are not returning up-to-date reports. With s1 down for 5 days, there should be a backl... [01:42:27] 10Data-Engineering, 10DBA, 10Data-Services, 10TaxonBot, and 3 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) >>! In T337446#8894111, @Liz wrote: > I'm not sure what's causing it (regarding s1), but I'm finding some bots are not returning up-to-date reports. Wit... [02:42:13] (DiskSpace) firing: Disk space an-test-worker1002:9100:/ 3.369% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [03:58:27] 10Data-Engineering, 10DBA, 10Data-Services, 10TaxonBot, and 3 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [04:21:43] (SystemdUnitFailed) firing: monitor_refine_eventlogging_analytics.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:07:55] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) [05:10:11] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) [05:20:37] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) [05:57:50] 10Data-Engineering, 10Infrastructure-Foundations: krb1001: krb5kdc.log excessive size - https://phabricator.wikimedia.org/T337906 (10ayounsi) p:05Triage→03High [06:24:52] 10Quarry, 10superset.wmcloud.org, 10cloud-services-team (FY2022/2023-Q4): Move Quarry to be an installation of Superset - https://phabricator.wikimedia.org/T169452 (10Rtnf) >>! In T169452#8858692, @Rchard2scout wrote: > Is this the right place to complain about features from Quarry that I'm missing from Supe... [06:42:13] (DiskSpace) firing: Disk space an-test-worker1002:9100:/ 3.18% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [06:42:25] (03CR) 10Nikerabbit: Show a warning for unused languages with localization over 75% (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/921329 (https://phabricator.wikimedia.org/T336752) (owner: 10Nik Gkountas) [08:00:45] mforns: Good morning - I have seen the errors of cassandra loading jobs, and on purpose decided not to act on them - let me know if you wish me to help :) [08:04:07] Hi stevemunene: could you please check which folder is making an-test-worker1002 root folder being full? [08:04:20] It's getting more and more full alert after alert [08:21:43] (SystemdUnitFailed) firing: monitor_refine_eventlogging_analytics.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:44:25] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) [08:46:43] (SystemdUnitFailed) firing: (2) monitor_refine_eventlogging_analytics.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:48:22] 10Data-Engineering, 10DBA, 10Data-Services, 10TaxonBot, and 3 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) p:05Unbreak!→03High I am reducing the priority of this as all the hosts have been recloned now and data should be up to date. We shouldn't be surpris... [08:51:38] 10Data-Engineering, 10DBA, 10Data-Services, 10TaxonBot, and 3 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) Just posted on wikitech-l https://lists.wikimedia.org/hyperkitty/list/wikitech-l@lists.wikimedia.org/message/WR36DST3Z6NV4YDZ4ODVPKHEK7VEGM4B/ [08:52:47] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) Need to re-run it again to take https://gerrit.wikimedia.org/r/c/operations/puppet/+/925547 into account... [08:56:28] 10Analytics-Radar, 10Data-Engineering-Icebox, 10Data-Persistence (work done), 10Platform Engineering Roadmap Decision Making, and 7 others: Remove revision_comment_temp and revision_actor_temp - https://phabricator.wikimedia.org/T215466 (10Zabe) [08:57:57] hello folks, I am roll rebooting kafka-test for kernel upgrades [09:02:08] 10Analytics-Radar, 10Data-Engineering-Icebox, 10Data-Persistence (work done), 10Platform Engineering Roadmap Decision Making, and 7 others: Remove revision_comment_temp and revision_actor_temp - https://phabricator.wikimedia.org/T215466 (10Ladsgroup) Running maintain-views and maintain-indexes on clouddb h... [09:07:42] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) This is going to alter revision table in enwiki and wikidatawiki. It'll take a while to finish. [09:16:44] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Marostegui) Wikidatawiki shouldn't be touched, s8 has not been part of the breakage. So we can skip it [09:26:43] (SystemdUnitFailed) firing: (2) monitor_refine_eventlogging_analytics.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:31:43] (SystemdUnitFailed) firing: (2) monitor_refine_eventlogging_analytics.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:35:46] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) >>! In T337734#8894652, @Marostegui wrote: > Wikidatawiki shouldn't be touched, s8 has not been part of... [09:46:24] joal: Steve is out today (public holiday). [09:47:26] Looking at an-test-worker1002, there is 14G in /var/cache/apt, with some really big packages that seem to be WMF specific: [09:47:59] https://www.irccloud.com/pastebin/7PaAjYBq/ [09:48:38] I'm not super up to date on how all that is deployed, but I suspect I can just `apt-get autoclean`. Any objection? [09:49:37] actually, `apt-get clean` [09:51:34] no objection! [09:51:37] thank you gehel [09:52:08] And apologizes stevemunene - I forgot about the public holiday in Kenya [09:53:18] done, disk usage down to 78% on an-test-worker1002 [09:54:08] * gehel wonders if we should have a systemd timer doing regular cleanup. It seems mostly useless to keep a local cache when we have our own APT repo that should contain everything that is installed. Probably a micro optimization. [09:57:13] (DiskSpace) resolved: Disk space an-test-worker1002:9100:/ 3.017% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [10:22:55] 10Data-Engineering, 10Anti-Harassment, 10CheckUser, 10Privacy Engineering, and 2 others: SPIKE: consider all problems that might happen when we handle Google's privacy changes - https://phabricator.wikimedia.org/T265057 (10kostajh) @Milimetric @odimitrijevic is there anything else you want to do with this... [10:25:22] 10Data-Engineering, 10Anti-Harassment, 10Metrics-Platform-Planning, 10Privacy Engineering, and 3 others: Measure user-agent client hints already sent in browsers requests - https://phabricator.wikimedia.org/T299397 (10kostajh) 05Open→03Resolved a:03kostajh @odimitrijevic @Milimetric I'm marking this... [10:25:40] 10Analytics-Radar, 10Data-Engineering-Icebox, 10Anti-Harassment, 10CheckUser, and 6 others: Deal with Google Chrome User-Agent deprecation - https://phabricator.wikimedia.org/T242825 (10kostajh) [10:32:42] 10Data-Engineering, 10Product-Analytics (Kanban): Model impact of User-Agent deprecation on top line metrics - https://phabricator.wikimedia.org/T336084 (10kostajh) >>! In T336084#8875151, @Mayakp.wiki wrote: > For next steps, we should re-start the discussion to request User-Agent Client-Hints and get High En... [10:44:12] 10Analytics-Radar, 10Data-Engineering-Icebox, 10Anti-Harassment, 10CheckUser, and 6 others: Deal with Google Chrome User-Agent deprecation - https://phabricator.wikimedia.org/T242825 (10kostajh) [10:44:42] 10Analytics-Radar, 10Data-Engineering-Icebox, 10Anti-Harassment, 10CheckUser, and 6 others: Deal with Google Chrome User-Agent deprecation - https://phabricator.wikimedia.org/T242825 (10kostajh) [11:42:02] 10Data-Engineering, 10DBA, 10Data-Services, 10TaxonBot, and 3 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [11:56:43] (SystemdUnitFailed) firing: (3) monitor_refine_eventlogging_analytics.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:06:43] (SystemdUnitFailed) firing: (4) monitor_refine_eventlogging_analytics.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:13:41] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) So I repooled the ones that have been maintained and have the indexes (and no view on rev_comment) and d... [12:17:24] 10Quarry, 10superset.wmcloud.org, 10cloud-services-team (FY2022/2023-Q4): Move Quarry to be an installation of Superset - https://phabricator.wikimedia.org/T169452 (10Wbm1058) >>! In T169452#8894304, @Rtnf wrote: >>>! In T169452#8858692, @Rchard2scout wrote: >> Is this the right place to complain about featu... [12:32:09] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10KylieTastic) I just tested my Quarry queries that were failing or taking hours when I last tested 3 hours ago and n... [12:54:35] 10Analytics-Radar, 10Data-Engineering-Icebox, 10Anti-Harassment, 10CheckUser, and 7 others: Deal with Google Chrome User-Agent deprecation - https://phabricator.wikimedia.org/T242825 (10kostajh) [12:56:05] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Marostegui) Thanks for letting us know! [12:58:16] 10Data-Engineering, 10Anti-Harassment, 10CheckUser, 10Privacy Engineering, and 2 others: SPIKE: consider problems to data pipelines as a result of reduced user agent entropy in Google Chrome - https://phabricator.wikimedia.org/T265057 (10kostajh) [13:57:14] 10Analytics-Radar, 10Data-Engineering-Icebox, 10Anti-Harassment, 10CheckUser, and 7 others: Deal with Google Chrome User-Agent deprecation - https://phabricator.wikimedia.org/T242825 (10kostajh) [14:02:30] 10Analytics-Radar, 10Data-Engineering-Icebox, 10Anti-Harassment, 10CheckUser, and 7 others: Deal with Google Chrome User-Agent deprecation - https://phabricator.wikimedia.org/T242825 (10kostajh) [14:13:45] !log deployed airflow analytics to fix anomaly detection ooms [14:13:46] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [14:16:13] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) [14:22:22] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Diannaa) Not sure if tyou are aware, but CopyPatrol is still not working. https://copypatrol.toolforge.org/en [14:26:43] (SystemdUnitFailed) firing: (2) monitor_refine_eventlogging_analytics.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:29:21] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Marostegui) >>! In T337734#8895779, @Diannaa wrote: > Not sure if tyou are aware, but CopyPatrol is still not worki... [14:31:43] (SystemdUnitFailed) firing: (2) monitor_refine_eventlogging_analytics.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:47:10] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10MusikAnimal) >>! In T337734#8895190, @Ladsgroup wrote: > So I repooled the ones that have been maintained and have... [14:56:59] (03CR) 10Milimetric: [C: 03+1] "+1 for now until we figure out what's going on with the totals. Everything else seems sorted out." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/914799 (https://phabricator.wikimedia.org/T337059) (owner: 10Nick Ifeajika) [15:01:36] (03CR) 10Milimetric: "Looks good. Just maybe find a different name other than timestamp. Then all we have left is to figure out the whole totals thing, otherw" [analytics/aqs] - 10https://gerrit.wikimedia.org/r/915678 (https://phabricator.wikimedia.org/T337059) (owner: 10Nick Ifeajika) [15:14:03] 10Data-Engineering, 10Anti-Harassment, 10Traffic: Include User-Agent Client Hints in WebRequest logs - https://phabricator.wikimedia.org/T337947 (10kostajh) [15:17:49] 10Data-Engineering, 10Anti-Harassment, 10CheckUser, 10Privacy Engineering, and 2 others: SPIKE: consider problems to data pipelines as a result of reduced user agent entropy in Google Chrome - https://phabricator.wikimedia.org/T265057 (10kostajh) Please see also {T337947}. [15:46:52] 10Data-Engineering, 10Event-Platform Value Stream, 10Release-Engineering-Team: eventutillities-python should publish python doc to doc.wikimedia.org - https://phabricator.wikimedia.org/T337475 (10jnuche) Just created a page with more details to help with this: https://www.mediawiki.org/wiki/GitLab/Publishing... [16:35:10] (03PS2) 10Sergio Gimeno: Add section title and ordinal in the image_suggestion_interaction schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/924879 (https://phabricator.wikimedia.org/T335716) [16:39:52] (03CR) 10Sergio Gimeno: Add section title and ordinal in the image_suggestion_interaction schema (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/924879 (https://phabricator.wikimedia.org/T335716) (owner: 10Sergio Gimeno) [17:03:58] mforns: By quickly looking at the code of cassandra-monthly loading dags, it seems passing application-jars are leftovers from a previous way of doin [17:04:23] joal: cool! will remove it then and test [17:04:35] thanks :] [17:04:35] mforns: thank you :) [17:09:44] actually, joal, I think I'm not going to test it manually, it seems like too big of an effort... the other jobs are the proof that the fix should work. Plus, the unit tests show the only thing that changes in the generated command is the application path, which points to the correct location. What do you think? [17:10:13] nFeel free to test in prod mforns - worst case, it fails again :) [17:10:49] yea joal, I had the same thought, cool will do [17:22:19] joal: can I take more of your time? :D https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/412 [17:24:38] 10Data-Engineering, 10DBA: Clean up clouddb1021 - https://phabricator.wikimedia.org/T337961 (10Ladsgroup) [17:26:43] (SystemdUnitFailed) firing: (2) monitor_refine_eventlogging_analytics.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:31:43] (SystemdUnitFailed) firing: (2) monitor_refine_eventlogging_analytics.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [17:36:26] 10Data-Engineering, 10Data-Engineering-Jupyter, 10Product-Analytics: Remove anaconda-wmf package from the cluster - https://phabricator.wikimedia.org/T337963 (10xcollazo) [17:50:57] !log deployed airflow analytics to unbreak monthly cassandra loading DAGs [17:50:59] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:02:37] joal: I self-merged it, I think we already discussed enough, it was a 1-liner change... [18:19:24] It worked, it fixed the problem! Now, there's another problem that already showed up last month... I think we need to bump up spark config for load_pageview_top_articles_to_cassandra, will do [18:26:28] 10Data-Engineering, 10Equity-Landscape, 10Movement-Insights: Transformations Flowchart - https://phabricator.wikimedia.org/T306614 (10JAnstee_WMF) [18:38:34] 10Data-Engineering, 10Equity-Landscape, 10Movement-Insights: Add country_meta_data - https://phabricator.wikimedia.org/T324681 (10nshahquinn-wmf) [18:53:01] 10Data-Engineering, 10Equity-Landscape, 10Movement-Insights: Transformations Flowchart - https://phabricator.wikimedia.org/T306614 (10JAnstee_WMF) p:05Triage→03Medium [18:58:47] 10Data-Engineering, 10Equity-Landscape: Load country data - https://phabricator.wikimedia.org/T310712 (10Mayakp.wiki) [19:09:46] !log deploy airflow analytics to bump up cassandra load monthly for top articles [19:09:47] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:10:35] 10Data-Engineering, 10DBA, 10Data-Services, 10TaxonBot, and 3 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Quiddity) >>! In T337446#8886722, @MusikAnimal wrote: > Something should be in Tech News Please could someone suggest //how// to summarize this for Tech News? Draft... [19:19:11] 10Data-Engineering, 10DBA, 10Data-Services, 10TaxonBot, and 3 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Ladsgroup) Production databases didn't have lag. Only cloud replicas but lags in order of days. Basically it stopped getting any updates for around a week due to dat... [19:25:49] 10Data-Engineering, 10DBA, 10Data-Services, 10TaxonBot, and 3 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) >>! In T337446#8897087, @Quiddity wrote: >>>! In T337446#8886722, @MusikAnimal wrote: >> Something should be in Tech News > > Please could someone sugge... [19:26:43] (SystemdUnitFailed) firing: (2) monitor_refine_eventlogging_analytics.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:29:45] 10Data-Engineering, 10DBA, 10Data-Services, 10TaxonBot, and 3 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Ladsgroup) The issue of slow down is resolved by now. There are still some replicas that don't have the index yet but all of them are depooled so no user-facing slow... [19:31:43] (SystemdUnitFailed) firing: (2) monitor_refine_eventlogging_analytics.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [19:42:50] 10Data-Engineering, 10DBA, 10Data-Services, 10TaxonBot, and 3 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Legoktm) >>! In T337446#8897087, @Quiddity wrote: > Please could someone suggest //how// to summarize this for Tech News? Draft wording //always// helps immensely!... [20:18:29] 10Data-Engineering, 10DBA, 10Data-Services, 10TaxonBot, and 3 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Quiddity) Thank you immensely @Legoktm that's exactly what I needed. :) [[ https://meta.wikimedia.org/wiki/Tech/News/2023/23 | Now added ]]. If anyone has changes, p... [20:26:43] (SystemdUnitFailed) firing: (2) monitor_refine_eventlogging_analytics.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:31:43] (SystemdUnitFailed) firing: (2) monitor_refine_eventlogging_analytics.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [20:36:45] 10Quarry, 10superset.wmcloud.org, 10cloud-services-team (FY2022/2023-Q4): Move Quarry to be an installation of Superset - https://phabricator.wikimedia.org/T169452 (10Stuartyeates) > Third. But, but but... we were told that Quarry "lacks support" and Superset has support overflowing with more support... who... [20:40:26] 10Quarry, 10superset.wmcloud.org, 10cloud-services-team (FY2022/2023-Q4): Move Quarry to be an installation of Superset - https://phabricator.wikimedia.org/T169452 (10EpicPupper) I think it’s definitely within the realm of possibility to contribute functionality for different export options upstream (wikitex... [20:41:50] 10Quarry, 10superset.wmcloud.org, 10cloud-services-team (FY2022/2023-Q4): Move Quarry to be an installation of Superset - https://phabricator.wikimedia.org/T169452 (10IKhitron) Conclusion: Until now if there was a feature that only wikimedia users needed, for example wikitable download, multilingual download... [21:16:30] 10Quarry, 10superset.wmcloud.org, 10cloud-services-team (FY2022/2023-Q4): Move Quarry to be an installation of Superset - https://phabricator.wikimedia.org/T169452 (10rook) >>! In T169452#8897369, @IKhitron wrote: > Conclusion: Until now if there was a feature that only wikimedia users needed, for example wi... [21:19:38] 10Quarry, 10superset.wmcloud.org, 10cloud-services-team (FY2022/2023-Q4): Move Quarry to be an installation of Superset - https://phabricator.wikimedia.org/T169452 (10IKhitron) >>>! In T169452#8897369, @IKhitron wrote: >> Conclusion: Until now if there was a feature that only wikimedia users needed, for exam... [21:56:43] (SystemdUnitFailed) firing: (2) monitor_refine_eventlogging_analytics.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:01:43] (SystemdUnitFailed) firing: (2) monitor_refine_eventlogging_analytics.service Failed on an-launcher1002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:09:52] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) [22:10:20] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) Everything is done, tomorrow I will repool them back. [23:52:30] 10Quarry, 10superset.wmcloud.org, 10cloud-services-team (FY2022/2023-Q4): Move Quarry to be an installation of Superset - https://phabricator.wikimedia.org/T169452 (10SWinxy) I would //hope// that there is the ability to create an extension in the 'Share' menu that may be used to create a wikitable.