[08:35:26] <wikibugs>	 (03PS1) 10Joal: Update changelog.md retroactively [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1147707
[08:40:09] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Update changelog.md retroactively [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1147707 (owner: 10Joal)
[08:50:47] <wikibugs>	 (03CR) 10Aqu: [C:03+1] "Looks good." [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1147707 (owner: 10Joal)
[10:03:15] <wikibugs>	 10Data-Engineering (Q4 2025 April 1st - June 30th): Facilitate automatic artifact cache warming for airflow-dags artifacts - https://phabricator.wikimedia.org/T392244#10834056 (10BTullis) > ` > /usr/lib/airflow/bin/artifact-cache warm /path/to/artifact.yaml > ` > We would need to run this comment on the Airflow...
[10:18:14] <wikibugs>	 (03CR) 10Joal: "recheck" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1147707 (owner: 10Joal)
[10:37:27] <wikibugs>	 (03CR) 10Joal: [V:03+2 C:03+2] "Merging for later deploy" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1147707 (owner: 10Joal)
[10:38:50] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 10ConfirmEdit (CAPTCHA extension), 10MediaWiki-extensions-Campaigns, and 4 others: Send hCaptcha API response data to event platform - https://phabricator.wikimedia.org/T379179#10834183 (10kostajh)
[11:11:33] <wikibugs>	 06Data-Engineering, 06Infrastructure-Foundations, 06Traffic, 13Patch-For-Review: WMF-Last-Access-Global cookie set on wrong domain when accessing static assets - https://phabricator.wikimedia.org/T367346#10834307 (10Vgutierrez) @mforns I've submitted https://gerrit.wikimedia.org/r/c/operations/puppet/+/114...
[11:58:40] <wikibugs>	 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Investigate first set of DQ alerts - https://phabricator.wikimedia.org/T394310#10834427 (10gmodena) > Thus exposing suppression data (for private, internal use) would require a new page change schema version, and I am not even sure th...
[13:05:05] <wikibugs>	 06Data-Engineering, 06cloud-services-team, 10Data-Services: Create existencelinks table in production - https://phabricator.wikimedia.org/T394617#10834617 (10Ladsgroup) From the #DBA sign off, it had and has my sign off (I reviewed the schema, etc.). You can deploy it yourself. I added it to the table catalo...
[13:51:46] <wikibugs>	 06Data-Engineering, 06Data-Engineering-Radar, 10DPE-Mediawiki-Content, 10GitLab (Upstream pit of despair 🕳️): Gitlab bug makes us have spurious artifact mismatch errors on YARN when running mw_content_merge_events_to_mw_content_history_daily - https://phabricator.wikimedia.org/T391123#10834840 (10xcollazo)
[13:53:28] <wikibugs>	 10Data-Engineering (Q4 2025 April 1st - June 30th): Facilitate automatic artifact cache warming for airflow-dags artifacts - https://phabricator.wikimedia.org/T392244#10834848 (10xcollazo) >I think that would be enough to allow us to stop using scap for artifact sync and would mean that we can decommission all o...
[13:55:28] <wikibugs>	 (03PS1) 10Peter Fischer: Update parent POM [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1147771 (https://phabricator.wikimedia.org/T367405)
[14:38:18] <wikibugs>	 06Data-Engineering, 06Infrastructure-Foundations, 06Traffic, 13Patch-For-Review: WMF-Last-Access-Global cookie set on wrong domain when accessing static assets - https://phabricator.wikimedia.org/T367346#10835061 (10mforns) @Vgutierrez thanks a lot for that! This should fix the issue and honor the filter t...
[14:48:40] <wikibugs>	 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Experimentation Lab (Experiment Platform Sprint 6): FY 24-25 SDS 2.4.9 CDN Synthetic Beacon: EventGate & Varnish: update to receive events from beacon event v2 - https://phabricator.wikimedia.org/T391959#10835094 (10phuedx) Being **bold**.
[14:55:13] <wikibugs>	 06Data-Engineering, 06Infrastructure-Foundations, 06Traffic, 13Patch-For-Review: WMF-Last-Access-Global cookie set on wrong domain when accessing static assets - https://phabricator.wikimedia.org/T367346#10835125 (10mforns) Thinking about the potential effects of this change... I think it might add new cou...
[14:59:50] <wikibugs>	 10Data-Engineering (Q4 2025 April 1st - June 30th), 13Patch-For-Review: Enable Spark data lineage for all Airflow instances - https://phabricator.wikimedia.org/T386862#10835132 (10mforns) Thanks @brouberol and @tchin!  OK, I think we can indeed switch the desired URL depending on whether we are in Kubernetes o...
[15:09:18] <wikibugs>	 10Data-Engineering (Q4 2025 April 1st - June 30th), 10Experimentation Lab (Experiment Platform Sprint 6): FY 24-25 SDS 2.4.9 CDN Synthetic Beacon: EventGate & Varnish: update to receive events from beacon event v2 - https://phabricator.wikimedia.org/T391959#10835162 (10mpopov) 05Open→03Resolved
[15:16:44] <wikibugs>	 10Data-Engineering (Q4 2025 April 1st - June 30th): Facilitate automatic artifact cache warming for airflow-dags artifacts - https://phabricator.wikimedia.org/T392244#10835221 (10JAllemandou) >>! In T392244#10834848, @xcollazo wrote: >>I think that would be enough to allow us to stop using scap for artifact sync...
[15:16:55] <wikibugs>	 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Infrastructure-Foundations, 06Traffic, 13Patch-For-Review: WMF-Last-Access-Global cookie set on wrong domain when accessing static assets - https://phabricator.wikimedia.org/T367346#10835223 (10Ahoelzl) a:03mforns
[15:21:58] <wikibugs>	 06Data-Engineering, 06Data-Persistence, 10Data-Platform, 06Data-Platform-SRE, and 2 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10835264 (10dr0ptp4kt)
[15:23:46] <wikibugs>	 06Data-Engineering, 06Data-Persistence, 10Data-Platform, 06Data-Platform-SRE, and 2 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10835274 (10fnegri) @dr0ptp4kt thanks! We might even start doing this for `an-redacteddb1001`, before clouddbs. @BTullis wdyt?
[15:36:33] <wikibugs>	 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Infrastructure-Foundations, 06Traffic: WMF-Last-Access-Global cookie set on wrong domain when accessing static assets - https://phabricator.wikimedia.org/T367346#10835418 (10Vgutierrez) 05Open→03Resolved test request on cp7001 after applying the p...
[16:20:31] <wikibugs>	 10Data-Engineering (Q4 2025 April 1st - June 30th): Facilitate automatic artifact cache warming for airflow-dags artifacts - https://phabricator.wikimedia.org/T392244#10835645 (10xcollazo) > Would a simple/not perfect check of existence be enough for now?  Agreed. I think that was also @amastilovic suggestions i...
[16:35:03] <wikibugs>	 06Data-Engineering, 06Data-Persistence, 10Data-Platform, 06Data-Platform-SRE, and 2 others: Migrate clouddb* hosts to MariaDB 10.11 - https://phabricator.wikimedia.org/T394372#10835772 (10Marostegui) >>! In T394372#10835232, @fnegri wrote: > @Marostegui is there any page on wikitech with the procedure that...
[16:55:17] <xcollazo>	 !log Deployed latest Airflow DAGs for the main instance. T392494.
[16:55:20] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[16:55:20] <stashbot>	 T392494: Add data quality metrics to mediawiki_content_current_v1 - https://phabricator.wikimedia.org/T392494
[17:01:01] <wikibugs>	 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content, 13Patch-For-Review: Add data quality metrics to mediawiki_content_current_v1 - https://phabricator.wikimedia.org/T392494#10835951 (10xcollazo) Ok we now have DQ tests in production.  Will wait until a successful run before closing t...
[17:25:58] <wikibugs>	 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Document wmf_content.mediawiki_content_current_v1 - https://phabricator.wikimedia.org/T392948#10836130 (10xcollazo)
[17:26:35] <wikibugs>	 10Data-Engineering (Q4 2025 April 1st - June 30th), 10DPE-Mediawiki-Content: Document wmf_content.mediawiki_content_current_v1 - https://phabricator.wikimedia.org/T392948#10836131 (10xcollazo) Modified documentation to reflect we have DQ and are officially in production.
[18:31:05] <icinga-wm>	 PROBLEM - statsv Varnishkafka log producer on cp6013 is CRITICAL: PROCS CRITICAL: 2 processes with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka
[18:32:05] <icinga-wm>	 RECOVERY - statsv Varnishkafka log producer on cp6013 is OK: PROCS OK: 1 process with args /usr/bin/varnishkafka -S /etc/varnishkafka/statsv.conf https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka
[19:14:04] <wikibugs>	 06Data-Engineering-Radar, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Shutdown varnishkafka instances - https://phabricator.wikimedia.org/T393772#10836716 (10Fabfur)
[19:50:42] <wikibugs>	 10Data-Engineering (Q4 2025 April 1st - June 30th), 06Trust and Safety Product Team, 13Patch-For-Review, 10Product-Analytics (Kanban): Add mediawiki_product_metrics_incident_reporting_system_interaction to the sanitization allowlist - https://phabricator.wikimedia.org/T384650#10836844 (10kostajh) >>! In T3...