[00:09:10] <jinxer-wm> FIRING: [2x] VarnishkafkaNoMessages: [WARNING] - varnishkafka on cp7009 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[00:10:10] <jinxer-wm> FIRING: VarnishkafkaNoMessages: [WARNING] - varnishkafka on cp4041 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp4041%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[00:11:54] <sukhe> uhm?
[00:15:10] <sukhe> brett: ^ any idea why varnishkafka-webrequest.service is not running?
[00:15:21] <sukhe> seems to be somewhat related to the upgrade
[00:15:24] <sukhe> the timing that is
[00:24:10] <jinxer-wm> RESOLVED: [2x] VarnishkafkaNoMessages: [WARNING] - varnishkafka on cp7009 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[00:25:10] <jinxer-wm> RESOLVED: VarnishkafkaNoMessages: [WARNING] - varnishkafka on cp7003 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=magru%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp7003%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[00:29:14] <sukhe> all looks goodnow
[01:31:11] <wikibugs> 06Data-Engineering: Analytics Cluster Dataset Usage Discovery Task - https://phabricator.wikimedia.org/T389903 (10VirginiaPoundstone) 03NEW
[01:33:04] <wikibugs> 06Data-Engineering: Analytics Cluster Dataset Usage Discovery Task - https://phabricator.wikimedia.org/T389903#10672154 (10VirginiaPoundstone)
[07:17:58] <wikibugs> 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Wikidata: The latest wikidata entity dump (latest-all.ttl.bz2) contains each triple twice - https://phabricator.wikimedia.org/T389787#10672512 (10Aklapper)
[07:20:49] <wikibugs> (03PS1) 10Joal: Update changelog to v0.2.59 for deploy [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1130885
[07:21:08] <wikibugs> (03CR) 10Joal: [V:03+2 C:03+2] "Self merging for release" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1130885 (owner: 10Joal)
[07:21:42] <wmf-insecte> Starting build #29 for job analytics-refinery-maven-release
[07:39:18] <wmf-insecte> Project analytics-refinery-maven-release build #29: 09SUCCESS in 17 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release/29/
[07:42:28] <wmf-insecte> Starting build #26 for job analytics-refinery-update-jars
[07:42:57] <wikibugs> (03PS1) 10Maven-release-user: Add refinery-source jars for v0.2.59 to artifacts [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1130943
[07:42:57] <wmf-insecte> Project analytics-refinery-update-jars build #26: 09SUCCESS in 28 sec: https://integration.wikimedia.org/ci/job/analytics-refinery-update-jars/26/
[07:45:48] <wikibugs> (03CR) 10Joal: [V:03+2 C:03+2] "Merging for next deploy" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/1130943 (owner: 10Maven-release-user)
[08:28:25] <joal> !log refinery-source released (v0.2.59)
[08:28:27] <stashbot> Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[08:28:39] <joal> !log refinery deployed (scap + hdfs)
[08:28:41] <stashbot> Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[08:37:46] <joal> !log Deploying Airflow-analytics using scap (for artifacts synchronization)
[08:37:47] <stashbot> Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[08:56:35] <joal> !log Manually warm HDFS cache as deployment ofairflow-dags/analytics doesn't synchronize artifacts from aiflow-dags/main
[08:56:37] <stashbot> Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[11:55:31] <wikibugs> 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Remove .m. subdomain, serve mobile and desktop variants through the same URL - https://phabricator.wikimedia.org/T214998#10673685 (10toni.stoev) >>! In T214998#10552803, @toni.stoev wrote: > Shall a committee be formed? In orde...
[12:15:09] <wikibugs> 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Remove .m. subdomain, serve mobile and desktop variants through the same URL - https://phabricator.wikimedia.org/T214998#10673751 (10toni.stoev) >>! In T214998#9990688, @Jdforrester-WMF wrote: > Note for those interested that th...
[14:16:46] <wikibugs> 06Data-Engineering, 10Event-Platform: Bug: jsonschema-tools generates non deterministic examples for date format fields - https://phabricator.wikimedia.org/T389881#10674235 (10jsn.sherman) @Ottomata in T382147 I went back to just editing the 1.0.0 examples in place instead of incrementing and tests are still f...
[15:19:55] <wikibugs> 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Commons-Impact-Metrics, 10Commons-Impact-Metrics-Requests: Update Commons Impact Metrics allow-list February 2025 - https://phabricator.wikimedia.org/T387592#10674581 (10Ahoelzl)
[15:22:50] <wikibugs> 10Data-Engineering (Q3 2025 January 1st - March 31th), 10DPE-Mediawiki-Content: dumps_publish_wikitext_raw_to_xml DAG fails sporadically. Needs sensors to wait before running. - https://phabricator.wikimedia.org/T363941#10674619 (10Ahoelzl)
[15:46:08] <wikibugs> 06Data-Engineering, 10Event-Platform: Bug: jsonschema-tools generates non deterministic examples for date format fields - https://phabricator.wikimedia.org/T389881#10674724 (10Reedy) https://gitlab.wikimedia.org/repos/data-engineering/jsonschema-tools/-/tags/v1.5.0 was only created 40 minutes ago The version...
[16:00:09] <wikibugs> 06Data-Engineering, 10Event-Platform: Bug: jsonschema-tools generates non deterministic examples for date format fields - https://phabricator.wikimedia.org/T389881#10674755 (10Reedy) >>! In T389881#10674724, @Reedy wrote: > https://gitlab.wikimedia.org/repos/data-engineering/jsonschema-tools/-/tags/v1.5.0 was...
[16:09:35] <wikibugs> 06Data-Engineering, 10Event-Platform: Bug: jsonschema-tools generates non deterministic examples for date format fields - https://phabricator.wikimedia.org/T389881#10674836 (10Ottomata) > @Ottomata in T382147 I went back to just editing the 1.0.0 examples in place instead of incrementing and tests are still fa...
[16:14:52] <wikibugs> 06Data-Engineering, 10Event-Platform, 13Patch-For-Review: Bug: jsonschema-tools generates non deterministic examples for date format fields - https://phabricator.wikimedia.org/T389881#10674876 (10jsn.sherman) >>! In T389881#10674836, @Ottomata wrote: >> @Ottomata in T382147 I went back to just editing the 1....
[16:25:24] <wikibugs> 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Event-Platform, 13Patch-For-Review: Bug: jsonschema-tools generates non deterministic examples for date format fields - https://phabricator.wikimedia.org/T389881#10674949 (10Ottomata)
[16:25:26] <wikibugs> 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Event-Platform, 13Patch-For-Review: Bug: jsonschema-tools generates non deterministic examples for date format fields - https://phabricator.wikimedia.org/T389881#10674950 (10Ottomata) p:05Triage→03High
[16:25:32] <wikibugs> 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Event-Platform, 13Patch-For-Review: Bug: jsonschema-tools generates non deterministic examples for date format fields - https://phabricator.wikimedia.org/T389881#10674951 (10Ottomata) a:03Ottomata
[17:17:55] <wikibugs> 06Data-Engineering, 06Data-Engineering-Radar, 10BDC-Implementation, 06Data-Platform-SRE, and 2 others: [Trino] Develop procedure and scripting for Trino cluster maintenance. - https://phabricator.wikimedia.org/T386391#10675182 (10Jgreen) The script trino-top is deployed to fran2001, it uses the dagster vir...
[17:18:22] <wikibugs> 06Data-Engineering, 10Commons-Impact-Metrics: Update Commons Impact Metrics to account for new File table - https://phabricator.wikimedia.org/T389800#10675192 (10Ahoelzl)
[17:30:51] <wikibugs> 06Data-Engineering, 06Abstract Wikipedia team, 10Wikifunctions: Bug: Cassandra Unique Devices not loading Wikifunctions mobile data - https://phabricator.wikimedia.org/T370551#10675239 (10Milimetric) It's a data quality issue, with unknown impact, on the unique devices dataset. Someone would have to do a sp...
[17:37:55] <wikibugs> 06Data-Engineering, 06MW-Interfaces-Team, 06Traffic: Log Api-User-Agent header in Turnilo - https://phabricator.wikimedia.org/T373871#10675270 (10Milimetric) This should've been tagged Data-Engineering, fixing
[17:41:52] <wikibugs> 06Data-Engineering, 06MW-Interfaces-Team, 06Traffic, 07OKR-Work: Log Api-User-Agent header in Turnilo - https://phabricator.wikimedia.org/T373871#10675291 (10Ottomata) This should fit into work for [[ https://meta.wikimedia.org/wiki/Wikimedia_Foundation_Annual_Plan/2025-2026/Product_%26_Technology_OKRs#Res...
[17:45:05] <wikibugs> 06Data-Engineering, 06Abstract Wikipedia team, 10Wikifunctions: Bug: Cassandra Unique Devices not loading Wikifunctions mobile data - https://phabricator.wikimedia.org/T370551#10675311 (10Ottomata) Related: {T214998}
[17:46:05] <wikibugs> 06Data-Engineering, 06Abstract Wikipedia team, 06Movement-Insights, 10Wikifunctions: Bug: Cassandra Unique Devices not loading Wikifunctions mobile data - https://phabricator.wikimedia.org/T370551#10675315 (10Ottomata)
[17:50:28] <wikibugs> 10Data-Engineering (Q4 2025 April 1st - June 30th): Analytics Cluster Dataset Usage Discovery Task - https://phabricator.wikimedia.org/T389903#10675337 (10Ahoelzl)
[17:54:20] <wikibugs> 06Data-Engineering, 10MediaWiki-extensions-Campaigns, 10MediaWiki-extensions-EventLogging, 10Event-Platform, and 2 others: Remove deprecated parameters from ServerSideAccountCreation - https://phabricator.wikimedia.org/T389819#10675374 (10Ottomata) @Reedy how can we help?
[17:55:25] <wikibugs> 06Data-Engineering, 10Commons-Impact-Metrics: Update Commons Impact Metrics to account for new File table - https://phabricator.wikimedia.org/T389800#10675383 (10Ahoelzl) @mforns can you please clarify impact and urgency?
[17:55:37] <wikibugs> 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Commons-Impact-Metrics: Update Commons Impact Metrics to account for new File table - https://phabricator.wikimedia.org/T389800#10675385 (10Ahoelzl)
[17:57:31] <wikibugs> 06Data-Engineering, 06Abstract Wikipedia team, 06Movement-Insights, 10Wikifunctions: Bug: Cassandra Unique Devices data quality issue: mobile data - https://phabricator.wikimedia.org/T370551#10675409 (10Milimetric)
[17:57:38] <wikibugs> 06Data-Engineering, 10MediaWiki-extensions-Campaigns, 10MediaWiki-extensions-EventLogging, 10Event-Platform, and 2 others: Remove deprecated parameters from ServerSideAccountCreation - https://phabricator.wikimedia.org/T389819#10675411 (10Reedy) Just wondering what is the process for us to do this removal...
[18:00:38] <wikibugs> 06Data-Engineering, 06Abstract Wikipedia team, 06Movement-Insights, 10Wikifunctions: Bug: Cassandra Unique Devices data quality issue: mobile data - https://phabricator.wikimedia.org/T370551#10675430 (10Milimetric) ping to our friends in #movement-insights: if you have any questions, just let me know
[18:17:05] <wikibugs> 06Data-Engineering, 10MediaWiki-extensions-Campaigns, 10MediaWiki-extensions-EventLogging, 10Event-Platform, and 2 others: Remove deprecated parameters from ServerSideAccountCreation - https://phabricator.wikimedia.org/T389819#10675492 (10Reedy) Or, as the answer was in {T389822} from yourself... >>! In T...
[18:29:16] <icinga-wm> PROBLEM - statsv Varnishkafka log producer on cp3066 is CRITICAL: PROCS CRITICAL: 0 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka
[18:47:16] <icinga-wm> RECOVERY - statsv Varnishkafka log producer on cp3066 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka
[19:23:18] <wikibugs> 06Data-Engineering, 10MediaWiki-extensions-Campaigns, 10MediaWiki-extensions-EventLogging, 10Event-Platform, and 2 others: Remove deprecated parameters from ServerSideAccountCreation - https://phabricator.wikimedia.org/T389819#10675844 (10Ottomata) >> Or... you could make a brand new Metrics Platform based...
[19:34:53] <wikibugs> 06Data-Engineering, 06Experimentation Lab: NEW/CHANGE FEATURE REQUEST: make available the centralauth.globaluser table in Data Lake - https://phabricator.wikimedia.org/T389666#10675917 (10Arinaigu) **The work requested in this ticket would support the following:** - my current work: - Strategic events impac...
[20:32:41] <wikibugs> 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Remove .m. subdomain, serve mobile and desktop variants through the same URL - https://phabricator.wikimedia.org/T214998#10676078 (10bd808) @toni.stoev Please read https://www.mediawiki.org/wiki/Special:MyLanguage/Code_of_Conduc...
[20:43:38] <wikibugs> 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 13Patch-For-Review: Add ability to add tags to alerts - https://phabricator.wikimedia.org/T389162#10676097 (10tchin)
[20:46:53] <wikibugs> 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 13Patch-For-Review: Add ability to add tags to alerts - https://phabricator.wikimedia.org/T389162#10676113 (10tchin)
[21:10:12] <wikibugs> (03PS3) 10TChin: Support inserting ResultKey into DeequVerificationSuiteToDataQualityAlerts [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1127964 (https://phabricator.wikimedia.org/T384962)
[21:26:36] <wikibugs> 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 13Patch-For-Review: Add ability to add tags to alerts - https://phabricator.wikimedia.org/T389162#10676311 (10tchin)
[21:28:00] <wikibugs> 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 13Patch-For-Review: Add ability to add tags to alerts - https://phabricator.wikimedia.org/T389162#10676313 (10tchin)
[21:41:22] <wikibugs> 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 13Patch-For-Review: Add ability to add tags to alerts - https://phabricator.wikimedia.org/T389162#10676357 (10tchin)
[21:46:55] <wikibugs> 10Data-Engineering (Q3 2025 January 1st - March 31th), 10Data-Engineering-Roadmap, 10DPE-Mediawiki-Content, 13Patch-For-Review: Add ability to add tags to alerts - https://phabricator.wikimedia.org/T389162#10676364 (10tchin)
[23:48:01] <wikibugs> 06Data-Engineering, 06SRE, 06Traffic-Icebox, 10MobileFrontend (Tracking): RFC: Remove .m. subdomain, serve mobile and desktop variants through the same URL - https://phabricator.wikimedia.org/T214998#10676590 (102003problems) >>! In T214998#10676078, @bd808 wrote: > @toni.stoev Please read https://www.medi...