[00:16:43] (SystemdUnitFailed) firing: hardsync-published.service Failed on an-web1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [00:18:36] PROBLEM - Check systemd state on an-web1001 is CRITICAL: CRITICAL - degraded: The following units failed: hardsync-published.service https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:30:58] RECOVERY - Check systemd state on an-web1001 is OK: OK - running: The system is fully operational https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state [00:36:43] (SystemdUnitFailed) resolved: hardsync-published.service Failed on an-web1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:23:27] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [02:37:13] (DiskSpace) firing: Disk space an-test-worker1002:9100:/ 4.464% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [02:40:34] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10JJMC89) [03:33:27] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-coord1001:10100 - https://alerts.wikimedia.org/?q=alertname%3DHiveServerHeapUsage [04:56:04] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [04:59:04] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [04:59:20] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) s1 is fully recloned, and catching up. I am going to start with s4 to be on the safe side. [05:03:21] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [05:11:43] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) a:03Marostegui [06:00:19] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Marostegui) a:03Marostegui I am running this on clouddb1021 for now. [06:00:28] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Marostegui) p:05Triage→03Medium [06:06:06] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [06:37:13] (DiskSpace) firing: Disk space an-test-worker1002:9100:/ 4.599% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [07:25:07] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [07:26:39] 10Quarry: Superset not exporting csv as utf-8 - https://phabricator.wikimedia.org/T337790 (10Aklapper) [07:26:45] 10Quarry, 10Cloud-VPS: List of admin users for superset? - https://phabricator.wikimedia.org/T337578 (10Aklapper) [07:26:51] 10Quarry, 10Documentation, 10User-EpicPupper: Improve Superset documentation - https://phabricator.wikimedia.org/T337342 (10Aklapper) [07:26:57] 10Quarry: superset to external db - https://phabricator.wikimedia.org/T336588 (10Aklapper) [07:27:03] 10Quarry: Assign multiple roles on superset login - https://phabricator.wikimedia.org/T336539 (10Aklapper) [07:27:09] 10Quarry: Public viewing of superset - https://phabricator.wikimedia.org/T336522 (10Aklapper) [07:27:19] 10Quarry, 10cloud-services-team (FY2022/2023-Q4): Move Quarry to be an installation of Superset - https://phabricator.wikimedia.org/T169452 (10Aklapper) [07:29:04] 10Quarry, 10superset.wmcloud.org: Superset not exporting csv as utf-8 - https://phabricator.wikimedia.org/T337790 (10Aklapper) [07:29:18] 10Quarry, 10superset.wmcloud.org, 10Documentation, 10User-EpicPupper: Improve Superset documentation - https://phabricator.wikimedia.org/T337342 (10Aklapper) [07:29:24] 10Quarry, 10Cloud-VPS, 10superset.wmcloud.org: List of admin users for superset? - https://phabricator.wikimedia.org/T337578 (10Aklapper) [07:29:30] 10Quarry, 10superset.wmcloud.org: Add examples to superset - https://phabricator.wikimedia.org/T336945 (10Aklapper) [07:29:36] 10Quarry, 10superset.wmcloud.org: Assign multiple roles on superset login - https://phabricator.wikimedia.org/T336539 (10Aklapper) [07:29:42] 10Quarry, 10superset.wmcloud.org: superset to external db - https://phabricator.wikimedia.org/T336588 (10Aklapper) [07:29:42] !log set "loadByPeriod(P8D+future), dropForever" for webrequest_sampled_live in druid-analytics - T337460 [07:29:44] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [07:29:45] T337460: Increase webrequest_sampled_live Druid datasource's retention - https://phabricator.wikimedia.org/T337460 [07:29:48] 10Quarry, 10superset.wmcloud.org: Public viewing of superset - https://phabricator.wikimedia.org/T336522 (10Aklapper) [07:29:54] 10Quarry, 10superset.wmcloud.org, 10cloud-services-team (FY2022/2023-Q4): Move Quarry to be an installation of Superset - https://phabricator.wikimedia.org/T169452 (10Aklapper) [07:38:38] joal: bonjour! Thanks a lot for the comment in the webrequest_sampled_live task [07:38:41] <3 [07:39:09] If you are ok I'll keep ramping up the retention to 30 days, and I'll file a change to drop data older than 90 days in puppet as wel [08:16:05] (03PS1) 10Sergio Gimeno: Add schema for section_image_suggestion_interaction [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/924879 (https://phabricator.wikimedia.org/T335716) [08:37:12] 10Data-Engineering, 10Traffic: Move varnishkafka to PKI - https://phabricator.wikimedia.org/T337825 (10elukey) [08:37:31] 10Data-Engineering, 10Data-Platform-SRE, 10Traffic: Move varnishkafka to PKI - https://phabricator.wikimedia.org/T337825 (10elukey) [09:18:47] (03PS3) 10Nick Ifeajika: query finetuning [analytics/refinery] - 10https://gerrit.wikimedia.org/r/914799 (https://phabricator.wikimedia.org/T337059) [09:30:18] 10Data-Engineering, 10AQS2.0, 10API Platform (AQS 2.0 Roadmap), 10Epic, and 2 others: AQS 2.0: Device Analytics service - https://phabricator.wikimedia.org/T288298 (10SGupta-WMF) [09:32:55] joal: Hey! Would you have a minute to jump in https://meet.google.com/vpz-jeud-rdm [09:36:51] 10Data-Engineering, 10AQS2.0, 10API Platform (AQS 2.0 Roadmap), 10Epic, and 2 others: AQS 2.0: Device Analytics service - https://phabricator.wikimedia.org/T288298 (10SGupta-WMF) Hi @hnowlan We have made really good progress on device-analytics code and this should be production ready in few days. What are... [10:03:14] (03PS5) 10Nick Ifeajika: Remove metrics parameter in endpoint Change time_bucket to timestamp for consistency [analytics/aqs] - 10https://gerrit.wikimedia.org/r/915678 (https://phabricator.wikimedia.org/T337059) [10:06:57] (03CR) 10CI reject: [V: 04-1] Remove metrics parameter in endpoint Change time_bucket to timestamp for consistency [analytics/aqs] - 10https://gerrit.wikimedia.org/r/915678 (https://phabricator.wikimedia.org/T337059) (owner: 10Nick Ifeajika) [10:10:33] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [10:12:27] 10Data-Engineering, 10API Platform (AQS 2.0 Roadmap), 10Epic, 10Platform Engineering Roadmap, 10User-Eevans: AQS 2.0: Media Analytics service - Unit testing - https://phabricator.wikimedia.org/T336383 (10codebug) 05Open→03In progress [10:17:39] (03PS2) 10Milimetric: Add several languages [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/921337 (https://phabricator.wikimedia.org/T337029) (owner: 10Amire80) [10:18:42] (03CR) 10Milimetric: [C: 03+2] "Thanks for the patch! We update the dist/ files as part of our build process, not with each commit. This way we can isolate those build " [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/921337 (https://phabricator.wikimedia.org/T337029) (owner: 10Amire80) [10:18:57] (03CR) 10Milimetric: [V: 03+2 C: 03+2] Add several languages [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/921337 (https://phabricator.wikimedia.org/T337029) (owner: 10Amire80) [10:19:24] (03CR) 10Milimetric: [C: 03+2] Add text direction to app container [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/921439 (https://phabricator.wikimedia.org/T251376) (owner: 10Nik Gkountas) [10:19:47] (03PS4) 10Milimetric: Add text direction to app container [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/921439 (https://phabricator.wikimedia.org/T251376) (owner: 10Nik Gkountas) [10:20:01] (03CR) 10Milimetric: [V: 03+2] Add text direction to app container [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/921439 (https://phabricator.wikimedia.org/T251376) (owner: 10Nik Gkountas) [10:20:09] (03CR) 10Milimetric: [C: 03+2] Add Hebrew "he" to languages.json [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/921476 (owner: 10Nik Gkountas) [10:20:14] (03PS4) 10Milimetric: Add Hebrew "he" to languages.json [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/921476 (owner: 10Nik Gkountas) [10:20:16] (03CR) 10Milimetric: [V: 03+2] Add Hebrew "he" to languages.json [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/921476 (owner: 10Nik Gkountas) [10:20:45] (03Abandoned) 10Milimetric: Update build files after adding Hebrew [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/921477 (owner: 10Nik Gkountas) [10:22:32] (03CR) 10Milimetric: [C: 03+2] languages.json: Consistent indenting [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/921161 (owner: 10Reedy) [10:23:14] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10MusikAnimal) I would argue this is high priority if not UBN (but acknowledging that Manuel isn't familiar with this... [10:24:23] (03PS3) 10Milimetric: SiteLanguage.vue: Make string translateable [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/921160 (owner: 10Reedy) [10:24:48] (03CR) 10Milimetric: [C: 03+2] "just a little typo, thanks for the change!" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/921160 (owner: 10Reedy) [10:24:54] (03CR) 10Milimetric: [V: 03+2 C: 03+2] SiteLanguage.vue: Make string translateable [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/921160 (owner: 10Reedy) [10:26:21] (03CR) 10Milimetric: "I have a patch adding something similar. Niklas the translatewiki warning wouldn't be visible here while building wikistats, so I can see" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/921329 (https://phabricator.wikimedia.org/T336752) (owner: 10Nik Gkountas) [10:26:32] (03CR) 10Milimetric: [C: 03+2] Remove zh.json [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/921339 (https://phabricator.wikimedia.org/T337029) (owner: 10Amire80) [10:27:10] (03PS2) 10Milimetric: Make consistent identation in languages.json [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/920275 (owner: 10Amire80) [10:27:34] (03Abandoned) 10Milimetric: Make consistent identation in languages.json [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/920275 (owner: 10Amire80) [10:27:57] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) I can take care of it. I think it's not hard to fix. Probably some schema updates have not been reflecte... [10:37:13] (DiskSpace) firing: Disk space an-test-worker1002:9100:/ 4.38% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [10:40:34] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [10:40:57] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) So it actually works: ` ladsgroup@clouddb1018:~$ sudo /usr/local/sbin/maintain-replica-indexes --databas... [10:42:00] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Marostegui) I was running it but it just broke, I am not sure if by running it twice there was a race condition but... [10:43:07] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) I was running it on clouddb1018 so it should not affect you. I take a look. [10:47:36] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Marostegui) a:05Marostegui→03None I would like to see involvement from either #cloud-services-team and/or #dat... [10:47:43] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Marostegui) p:05Medium→03High [10:56:05] (03PS1) 10Aqu: Release 2.10.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/924907 [10:58:23] (03CR) 10Milimetric: [C: 03+2] Release 2.10.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/924907 (owner: 10Aqu) [10:59:50] (03Merged) 10jenkins-bot: Release 2.10.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/924907 (owner: 10Aqu) [11:02:06] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [11:44:03] (03PS7) 10Peter Fischer: Encode redirect targets in page change events. [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/914867 (https://phabricator.wikimedia.org/T325315) [11:52:34] (03CR) 10Peter Fischer: "Rewrote `link_target` entity to support multiple types via `oneOf` with a type discriminator `link_target.link_type ∈ {local, interwiki, …" [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/914867 (https://phabricator.wikimedia.org/T325315) (owner: 10Peter Fischer) [11:52:53] Hi gehel - I'm sorry I was unavailable at the time you pinged :( May I help now? [11:58:21] elukey: ok for me :) [12:06:04] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 A): eventutilities-python: review and clean up in preparation for a GA release. - https://phabricator.wikimedia.org/T336488 (10CodeReviewBot) gmodena updated https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/5... [12:07:45] 10Data-Engineering, 10Event-Platform Value Stream: jsonschema-tools deterministic schema test should fail if a schema uses oneOf with different types - https://phabricator.wikimedia.org/T337855 (10Ottomata) [12:07:55] joal: yeah, I realized too late that you're not working this morning. stevemunene [12:08:11] joal: stevemunene will ping you with a few questions. [12:08:33] ack gehel :) [12:14:11] 10Data-Engineering, 10Product-Analytics, 10Research: Use Hive/Spark timestamps in Refined event data - https://phabricator.wikimedia.org/T278467 (10Ottomata) Nope! I think this was tagged with yall long ago because it would change some basic query semantics of timestamps in Hive tables. We should do it, bu... [12:15:50] ottomata: do you know why Superset keeps record of who has viewed a dashboard? Is this something we intentionally choose to turn on? It feels somewhat invasive that after clicking https://superset.wmcloud.org/superset/dashboard/1/ somewhere, everyone can now see at https://superset.wmcloud.org/superset/profile/Krinkle/ that I merely clicked that link. [12:16:11] The same applies to our private instance as well, but it's perhaps more surprising on the public one [12:17:21] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 A): eventutilities-python: review and clean up in preparation for a GA release. - https://phabricator.wikimedia.org/T336488 (10Ottomata) Think we can publish pydocs as part of this task!? https://gitlab.wikimedia.org/repos/releng/docpub [12:27:55] 10Data-Engineering, 10Product-Analytics, 10Research: Use Hive/Spark timestamps in Refined event data - https://phabricator.wikimedia.org/T278467 (10JAllemandou) It'll become higher priority when we need timestamp typed fields for Iceberg :) [12:29:38] 10Data-Engineering, 10Data-Platform-SRE, 10Event-Platform Value Stream, 10Platform Team Workboards (Clinic Duty Team): Avoid accepting Kafka messages with whacky timestamps - https://phabricator.wikimedia.org/T282887 (10Ottomata) Relevant: https://cwiki.apache.org/confluence/display/KAFKA/KIP-937%3A+Impr... [12:30:14] 10Data-Engineering, 10Product-Analytics, 10Research: Use Hive/Spark timestamps in Refined event data - https://phabricator.wikimedia.org/T278467 (10Ottomata) OO, maybe we should do this as part of the Iceberg migration then, since we will be creating new tables for that. [12:30:33] 10Data-Engineering, 10Product-Analytics, 10Research: Use Hive/Spark timestamps in Refined event data - https://phabricator.wikimedia.org/T278467 (10Ottomata) cc @xcollazo [12:35:16] (03CR) 10Ottomata: Encode redirect targets in page change events. (031 comment) [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/914867 (https://phabricator.wikimedia.org/T325315) (owner: 10Peter Fischer) [12:36:47] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) I have a rather educated guess why it's erroring out. The way we do schema changes is by looking up dbli... [12:47:59] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 A): eventutilities-python: review and clean up in preparation for a GA release. - https://phabricator.wikimedia.org/T336488 (10gmodena) > Think we can publish pydocs as part of this task!? Sure, seems easy enough. [12:58:33] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, and 2 others: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) Yup, with the patch works like a charm: ` ... 2023-05-31 12:57:55,295 DEBUG SHOW INDEX from afwiki.revision_com... [13:35:21] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [13:35:46] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) s4 on clouddb1021 has been recloned, added views, heartbeat, grants etc. Once it has caught up I will reclone the other two hosts from it. [13:50:25] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [14:02:32] 10Data-Engineering, 10Event-Platform Value Stream, 10EventStreams: Include image/file changes in page-links-change - https://phabricator.wikimedia.org/T333497 (10pfischer) I wrote a [[ https://docs.google.com/spreadsheets/d/1ao5HaKaZvAneM2zrTS1r3ZjXSaGnK9M61EQtTR2cALg/edit?usp=sharing | schema comparison ]]... [14:05:59] 10Data-Engineering-Planning, 10Event-Platform Value Stream, 10Machine-Learning-Team, 10Patch-For-Review: Add a new outlink topic stream for EventGate main - https://phabricator.wikimedia.org/T328899 (10pfischer) I wrote a [[ https://docs.google.com/spreadsheets/d/1ao5HaKaZvAneM2zrTS1r3ZjXSaGnK9M61EQtTR2cAL... [14:18:19] 10Data-Engineering, 10Event-Platform Value Stream, 10Machine-Learning-Team: Create new mediawiki.page_links_change stream based on fragment/mediawiki/state/change/page - https://phabricator.wikimedia.org/T331399 (10pfischer) I wrote a [[ https://docs.google.com/spreadsheets/d/1ao5HaKaZvAneM2zrTS1r3ZjXSaGnK9M... [14:24:53] 10Data-Engineering, 10Product-Analytics, 10Research: Use Hive/Spark timestamps in Refined event data - https://phabricator.wikimedia.org/T278467 (10xcollazo) Agreed that the Hive tables can stay as they are, and the new Iceberg tables can do proper `DATE`s and `TIMESTAMP`s. When inserting into Iceberg, we ca... [14:37:13] (DiskSpace) firing: Disk space an-test-worker1002:9100:/ 4.158% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [14:46:08] 10Data-Engineering, 10Product-Analytics, 10Research: Use Hive/Spark timestamps in Refined event data - https://phabricator.wikimedia.org/T278467 (10Ottomata) If we migrate Refine to write to Iceberg, we'll need to modify JsonSchemaConverter as noted in the description do the right thing. Also, maybe we sh... [14:54:09] 10Data-Engineering-Planning, 10Epic, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Deploy mediawiki-page-content-change-enrichment to wikikube k8s - https://phabricator.wikimedia.org/T325303 (10CodeReviewBot) otto opened https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-... [14:57:10] Starting build #21 for job wikimedia-event-utilities-maven-release-docker [14:58:20] 10Data-Engineering, 10Product-Analytics (Kanban): Check home/HDFS leftovers of xihua - https://phabricator.wikimedia.org/T337711 (10mpopov) a:03mpopov [14:58:43] 10Data-Engineering, 10Product-Analytics (Kanban): Check home/HDFS leftovers of xihua - https://phabricator.wikimedia.org/T337711 (10mpopov) p:05Triage→03Medium [15:01:16] Project wikimedia-event-utilities-maven-release-docker build #21: 09SUCCESS in 4 min 6 sec: https://integration.wikimedia.org/ci/job/wikimedia-event-utilities-maven-release-docker/21/ [15:06:29] 10Data-Engineering-Planning, 10Epic, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Deploy mediawiki-page-content-change-enrichment to wikikube k8s - https://phabricator.wikimedia.org/T325303 (10CodeReviewBot) otto merged https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-... [15:34:06] PROBLEM - Webrequests Varnishkafka log producer on cp2035 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [15:34:16] PROBLEM - eventlogging Varnishkafka log producer on cp2035 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/eventlogging.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [15:35:00] PROBLEM - statsv Varnishkafka log producer on cp2035 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [15:35:10] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10CodeReviewBot) otto opened https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/60 Bump to e... [15:35:20] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10CodeReviewBot) [15:41:31] 10Data-Engineering-Planning, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Flink App Deployment monitoring - https://phabricator.wikimedia.org/T328925 (10CodeReviewBot) otto merged https://gitlab.wikimedia.org/repos/data-engineering/mediawiki-event-enrichment/-/merge_requests/60 Bump to e... [15:47:14] Heya SRE folks - btullis, stevemunene or ottomata - we received alerts about a varnishkafka not behaving - can anyone of you have a quick look? [15:47:26] If it's real alert, we possibly lose data [15:47:55] joal: Valentin is working on the node, it is depooled, all good :) [15:48:09] Thanks a milion elukey - I'm reassured <3 [15:48:11] joal: what elukey said, known :) [15:48:19] yep... the host is having some issues and it's currently unable to run puppet [15:48:32] ack vgutierrez - thank you for the detail :) [15:52:52] 10Data-Engineering, 10Product-Analytics: Use Hive/Spark timestamps in Refined event data - https://phabricator.wikimedia.org/T278467 (10leila) [15:53:51] 10Data-Engineering, 10Product-Analytics: Use Hive/Spark timestamps in Refined event data - https://phabricator.wikimedia.org/T278467 (10leila) Thanks, all. Then I remove this from our todo lane. Please do add #research back if you want us to support you in some direct way. And thank you for your work. :) [15:59:11] 10Data-Engineering, 10Product-Analytics: Use Hive/Spark timestamps in Refined event data - https://phabricator.wikimedia.org/T278467 (10JAllemandou) >>! In T278467#8892762, @Ottomata wrote: > If we migrate Refine to write to Iceberg, we'll need to modify JsonSchemaConverter as noted in the description do the r... [16:03:47] RECOVERY - Webrequests Varnishkafka log producer on cp2035 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/webrequest.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [16:03:57] RECOVERY - eventlogging Varnishkafka log producer on cp2035 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/eventlogging.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [16:04:17] RECOVERY - statsv Varnishkafka log producer on cp2035 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka [16:14:15] 10Data-Engineering, 10serviceops, 10Event-Platform Value Stream (Sprint 14 A), 10Service-deployment-requests: New Service Request mediawiki-page-content-change-enrichment - https://phabricator.wikimedia.org/T330507 (10Ottomata) [16:14:23] 10Data-Engineering-Planning, 10Epic, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Deploy mediawiki-page-content-change-enrichment to wikikube k8s - https://phabricator.wikimedia.org/T325303 (10Ottomata) [16:14:54] 10Data-Engineering-Planning, 10Epic, 10Event-Platform Value Stream (Sprint 14 A), 10Patch-For-Review: Deploy mediawiki-page-content-change-enrichment to wikikube k8s - https://phabricator.wikimedia.org/T325303 (10Ottomata) [16:15:05] 10Data-Engineering, 10serviceops, 10Event-Platform Value Stream (Sprint 14 A), 10Service-deployment-requests: New Service Request mediawiki-page-content-change-enrichment - https://phabricator.wikimedia.org/T330507 (10Ottomata) We are live in wikikube eqiad and codfw! [16:36:32] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [16:37:00] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) Since it's fixed, running it on clouddb1013 now [16:38:47] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [16:58:07] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) And it's failing waiting for table lock. Someone need to depool clouddb1013 [17:59:10] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, and 2 others: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) Aaand it had another issue making it really hard to do the schema changes: It doesn't stop replication and thus it... [18:37:28] (DiskSpace) firing: Disk space an-test-worker1002:9100:/ 3.937% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [19:27:50] 10Data-Engineering, 10DBA, 10Data-Services, 10cloud-services-team, and 2 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10TheresNoTime) [19:35:28] 10Data-Engineering, 10DBA, 10Data-Services, 10TaxonBot, and 3 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10doctaxon) [19:36:32] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 A): eventutilities-python: review and clean up in preparation for a GA release. - https://phabricator.wikimedia.org/T336488 (10CodeReviewBot) otto merged https://gitlab.wikimedia.org/repos/data-engineering/eventutilities-python/-/merge_requests/58 m... [19:56:09] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) [20:10:43] 10Data-Engineering, 10DBA, 10Data-Services, 10TaxonBot, and 3 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [20:11:09] 10Data-Engineering, 10DBA, 10Data-Services, 10TaxonBot, and 3 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) [20:11:34] 10Data-Engineering, 10DBA, 10Data-Services, 10TaxonBot, and 3 others: Rebuild sanitarium hosts - https://phabricator.wikimedia.org/T337446 (10Marostegui) s4 has been fully recloned, clouddb1019:3314 is now catching up with its master [20:12:24] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) [20:16:12] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Marostegui) @Ladsgroup clouddb1013:3311 has replication stopped, was that you I guess? Just double checking [20:18:04] 10Data-Engineering, 10Data-Engineering-Wikistats, 10I18n, 10Wikimedia-Hackathon-2023: Untranslatable strings in Wikistats - https://phabricator.wikimedia.org/T337032 (10Kaganer) Message `Wikistats-metrics-edited-pages-description` is exist, but [[ https://translatewiki.net/w/i.php?title=Wikimedia:Wikistat... [20:18:43] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) yeah, the ALTERs couldn't go through because of [[https://github.com/wikimedia/operations-puppet/blob/f6... [20:20:13] !log starting refinery deployment [20:20:14] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:23:35] 10Data-Engineering, 10Data-Engineering-Wikistats, 10I18n, 10Wikimedia-Hackathon-2023: Untranslatable strings in Wikistats - https://phabricator.wikimedia.org/T337032 (10Kaganer) Some hardcoded/untranslatable messages: (//in Sidebar//) - Wiki - Filter/split - Dimensions - Filter by - Split by th... [20:26:11] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) [20:28:45] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) [20:30:02] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) [20:38:27] 10Data-Engineering, 10DBA, 10Data-Persistence, 10Data-Services, 10cloud-services-team: Investigate if maintain-replica-indexes is still needed - https://phabricator.wikimedia.org/T337734 (10Ladsgroup) had to the same on clouddb1015:3314 now [20:38:48] 10Data-Engineering, 10Data-Engineering-Wikistats, 10I18n, 10Wikimedia-Hackathon-2023: Untranslatable strings in Wikistats - https://phabricator.wikimedia.org/T337032 (10Reedy) [20:41:50] !log finished refinery deployment [20:41:52] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:50:01] 10Data-Engineering, 10Data-Engineering-Wikistats, 10I18n, 10Wikimedia-Hackathon-2023: Untranslatable strings in Wikistats - https://phabricator.wikimedia.org/T337032 (10Kaganer) `Wikistats-metrics-page-views-by-country-description` - The dot character did not get inside the translatable message. This is an... [20:50:56] 10Data-Engineering, 10Data-Engineering-Wikistats, 10I18n: Wiki selector: "All wikis" is not translated - https://phabricator.wikimedia.org/T246911 (10Reedy) This also includes "All Wikibooks", "All Wikinews" etc... [20:55:09] 10Data-Engineering, 10Event-Platform Value Stream (Sprint 14 A): eventutilities-python: review and clean up in preparation for a GA release. - https://phabricator.wikimedia.org/T336488 (10gmodena) [20:55:40] 10Data-Engineering, 10Data-Engineering-Wikistats, 10I18n, 10Wikimedia-Hackathon-2023: Untranslatable strings in Wikistats - https://phabricator.wikimedia.org/T337032 (10Kaganer) Country names may be translated, this is common data, IMHO: {F37086759} Also names of months in a charts: {F37086763} [21:06:39] (03CR) 10Gergő Tisza: Add schema for section_image_suggestion_interaction (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/924879 (https://phabricator.wikimedia.org/T335716) (owner: 10Sergio Gimeno) [21:09:24] (03PS3) 10Reedy: README.md: Code isn't in Phabricator repo :) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/925029 [21:19:19] 10Data-Engineering, 10Data-Engineering-Wikistats, 10I18n, 10Patch-For-Review, 10Wikimedia-Hackathon-2023: Untranslatable strings in Wikistats - https://phabricator.wikimedia.org/T337032 (10Kaganer) This is may be design issue, but this is relates with i18n topic. Title case mode is may be good in English... [21:28:27] 10Data-Engineering, 10Data-Engineering-Wikistats, 10I18n, 10Patch-For-Review, 10Wikimedia-Hackathon-2023: Untranslatable strings in Wikistats - https://phabricator.wikimedia.org/T337032 (10Reedy) https://plugins.jetbrains.com/plugin/17212-i18n-ally - Why find them manually when you can automate it? Not... [21:31:15] (03PS4) 10Reedy: README.md: Code isn't in Phabricator repo :) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/925029 [21:32:35] (03CR) 10Reedy: "Looks like the package-lock.json update was missed/not commited?" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/924907 (owner: 10Aqu) [21:55:12] 10Data-Engineering, 10Data-Engineering-Wikistats, 10I18n, 10Patch-For-Review, 10Wikimedia-Hackathon-2023: Untranslatable strings in Wikistats - https://phabricator.wikimedia.org/T337032 (10Reedy) >>! In T337032#8893795, @Kaganer wrote: > Also names of months in a charts: > {F37086763} Translations exist... [22:06:49] 10Data-Engineering, 10Data-Engineering-Wikistats, 10I18n, 10Patch-For-Review, 10Wikimedia-Hackathon-2023: Untranslatable strings in Wikistats - https://phabricator.wikimedia.org/T337032 (10Kaganer) >clearly see the months hardcoded in English Ok, this is just proposal, on future. For users would be usef... [22:17:38] 10Data-Engineering, 10Data-Engineering-Wikistats, 10I18n, 10Patch-For-Review, 10Wikimedia-Hackathon-2023: Untranslatable strings in Wikistats - https://phabricator.wikimedia.org/T337032 (10Reedy) Yeah, in an ideal world, everything practically should be translateable. I think the intention of this task... [22:17:46] 10Data-Engineering, 10Data-Engineering-Wikistats, 10I18n, 10Patch-For-Review, 10Wikimedia-Hackathon-2023: Untranslatable strings in Wikistats - https://phabricator.wikimedia.org/T337032 (10Kaganer) >>! In T337032#8893869, @Reedy wrote: > Why find them manually when you can automate it? I'm don't find th... [22:18:22] 10Data-Engineering, 10Data-Engineering-Wikistats: Wikistats 2 should translate month names and abbreviations - https://phabricator.wikimedia.org/T336815 (10Reedy) Months are already in the translation files... I think we're just not doing a good job of using them. See https://translatewiki.net/wiki/Special:Pr... [22:27:53] 10Data-Engineering, 10Data-Engineering-Wikistats, 10I18n, 10Patch-For-Review, 10Wikimedia-Hackathon-2023: Untranslatable strings in Wikistats - https://phabricator.wikimedia.org/T337032 (10Reedy) >>! In T337032#8893923, @Kaganer wrote: >>>! In T337032#8893869, @Reedy wrote: >> Why find them manually when... [22:31:29] (03PS1) 10Reedy: Remove unused consts from *.config.js [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/925064 [22:32:34] (03PS1) 10Reedy: package-lock.json: Update version to 2.10.0 [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/925065 [22:33:43] (03CR) 10Reedy: Release 2.10.0 (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/924907 (owner: 10Aqu) [22:41:53] (03PS3) 10Reedy: Update footer links [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/925031 (https://phabricator.wikimedia.org/T336765) [22:42:13] (DiskSpace) firing: Disk space an-test-worker1002:9100:/ 3.715% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=an-test-worker1002 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [22:42:51] (03CR) 10Reedy: Update footer links (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/925031 (https://phabricator.wikimedia.org/T336765) (owner: 10Reedy) [22:44:46] 10Data-Engineering, 10Data-Engineering-Wikistats: Number of anonymous edits - https://phabricator.wikimedia.org/T323562 (10Reedy) [22:47:21] 10Data-Engineering, 10Data-Engineering-Wikistats, 10I18n: In Wikistats, `npm run dev` only shows strings in English - https://phabricator.wikimedia.org/T337033 (10Reedy) If you run `npm build` first, does it have them? [22:47:56] 10Data-Engineering, 10Data-Engineering-Wikistats, 10I18n: Metrics tooltip in detail page is not localized - https://phabricator.wikimedia.org/T287908 (10Reedy) [22:48:42] 10Data-Engineering-Planning, 10Data-Engineering-Wikistats, 10Data Pipelines, 10I18n: Merge Ks-Arab and Ks-Deva to ks - https://phabricator.wikimedia.org/T314476 (10Reedy) [22:49:39] 10Data-Engineering, 10Data-Engineering-Wikistats, 10I18n, 10Patch-For-Review, 10Wikimedia-Hackathon-2023: Untranslatable strings in Wikistats - https://phabricator.wikimedia.org/T337032 (10Reedy) [22:49:56] 10Data-Engineering, 10Data-Engineering-Wikistats: "Active editors by country" doesn't display numbers for Belarus, Kazakhstan, Russia - https://phabricator.wikimedia.org/T333716 (10Reedy) [22:50:00] 10Data-Engineering-Planning, 10Data-Engineering-Wikistats: Get visibility which pages are being heavily edited, plundered, which need patrolling - https://phabricator.wikimedia.org/T315196 (10Reedy) [22:50:07] 10Data-Engineering, 10Data-Engineering-Wikistats: Define reduce calculations needed to compute active editors per project family - https://phabricator.wikimedia.org/T249751 (10Reedy) [22:50:14] 10Data-Engineering, 10Data-Engineering-Wikistats, 10I18n: Fixed time range names are forced to capitalized regardless of locale in sidebar - https://phabricator.wikimedia.org/T287910 (10Reedy) [22:50:22] 10Data-Engineering, 10Data-Engineering-Wikistats, 10Commons: Creating tools for compiling list of Wikimedia Commons users by contributions/uploads - https://phabricator.wikimedia.org/T263377 (10Reedy) [22:50:29] 10Data-Engineering, 10Data-Engineering-Wikistats: Group pageview data per family in AQS so we can surface it in wikistats per-family pageview metrics - https://phabricator.wikimedia.org/T205730 (10Reedy) [22:50:35] 10Data-Engineering, 10Data-Engineering-Wikistats: Wikimedia Statistics - Horizontal (time) axis wrongly formatted when the option "Monthly" is choosen - https://phabricator.wikimedia.org/T290551 (10Reedy) [22:50:41] 10Data-Engineering, 10Data-Engineering-Wikistats: Render Wikistats graphs server-side, so that they cam be embedded on-wikis - https://phabricator.wikimedia.org/T178016 (10Reedy) [22:50:46] 10Data-Engineering-Planning, 10Data-Engineering-Wikistats: Determine total number of external links in all Wikipedias - https://phabricator.wikimedia.org/T137984 (10Reedy) [22:52:14] 10Data-Engineering, 10Data-Engineering-Wikistats: User stats page - https://phabricator.wikimedia.org/T324993 (10Reedy) [22:52:40] 10Data-Engineering, 10Data-Engineering-Wikistats, 10Wikimedia-Hackathon-2023: Add several new LTR languages to wikistats (May 2023) - https://phabricator.wikimedia.org/T337029 (10Reedy) 05Open→03Resolved [22:53:05] 10Data-Engineering, 10Data-Engineering-Wikistats, 10I18n, 10Patch-For-Review: Wikistats 2 should translate month names and abbreviations - https://phabricator.wikimedia.org/T336815 (10Reedy) [22:57:17] (03PS5) 10Reedy: README.md: Code isn't in Phabricator repo :) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/925029 [23:00:15] (03CR) 10Reedy: "I'm not sure if https://wikitech.wikimedia.org/wiki/Data_Engineering/Systems/Wikistats_2 really is "right", but https://github.com/milimet" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/925032 (owner: 10Reedy) [23:15:12] 10Data-Engineering, 10Data-Engineering-Wikistats: Render Wikistats graphs server-side, so that they can be embedded on-wikis - https://phabricator.wikimedia.org/T178016 (10Reedy) [23:26:27] (03PS6) 10Reedy: README.md: Code isn't in Phabricator repo :) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/925029