[05:35:12] 06Data-Engineering, 06Data Products, 06DBA, 07Schema-change-in-production: Cleanup revision table schema - https://phabricator.wikimedia.org/T367856#10030537 (10Marostegui) [08:33:38] 06Data-Engineering, 06Movement-Insights, 10Wikidata, 10Wikidata Analytics, 10Wmfdata-Python: Add linter and formatter to wmfdata-python (and link check) - https://phabricator.wikimedia.org/T348999#10030788 (10Manuel) [08:33:41] 06Data-Engineering, 10Wikidata, 10Wikidata Analytics, 10Wmfdata-Python: Add testing framework to wmfdata-python - https://phabricator.wikimedia.org/T349531#10030789 (10Manuel) [10:02:40] 06Data-Engineering, 06Traffic: Upgrade Benthos package on cp hosts - https://phabricator.wikimedia.org/T366031#10031080 (10fgiunchedi) untagging o11y since benthos is going to be removed from cp hosts [10:03:44] 06Data-Engineering, 06Traffic, 13Patch-For-Review: Install benthos on single esams host to check performances under higher load - https://phabricator.wikimedia.org/T365968#10031082 (10fgiunchedi) untagging o11y since benthos is going to be removed from cp hosts [10:09:50] 06Data-Engineering, 06Data Products, 06Traffic: New software: haproxykafka - https://phabricator.wikimedia.org/T370668#10031099 (10fgiunchedi) untagging o11y since this is unrelated [10:41:10] 06Data-Engineering, 10WMF-JobQueue, 10Event-Platform, 07Wikimedia-production-error: JobQueueError: Could not enqueue jobs - https://phabricator.wikimedia.org/T371433#10031211 (10Dreamy_Jazz) p:05Triage→03Unbreak! Prevents login for me to. Impacts abuse-mitigation that involves looking at the CheckUser... [10:52:32] 06Data-Engineering, 10WMF-JobQueue, 10Event-Platform, 07Wikimedia-production-error: Could not enqueue jobs for stream {stream} - https://phabricator.wikimedia.org/T371471#10031285 (10mszabo) [11:02:27] 06Data-Engineering, 10WMF-JobQueue, 10Event-Platform, 07Wikimedia-production-error: Could not enqueue jobs for stream {stream} - https://phabricator.wikimedia.org/T371471#10031361 (10mszabo) →14Duplicate dup:03T371433 [11:04:03] 06Data-Engineering, 10WMF-JobQueue, 10Event-Platform, 13Patch-For-Review, 07Wikimedia-production-error: JobQueueError: Could not enqueue jobs - https://phabricator.wikimedia.org/T371433#10031363 (10mszabo) [11:25:20] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10CirrusSearch, 03Discovery-Search (Current work), 10MW-1.43-notes (1.43.0-wmf.15; 2024-07-23): [Search Update Pipeline] Source streams for private wikis - https://phabricator.wikimedia.org/T346046#10031426 (10Urbanecm_WMF) FTR, this broke all jobs at... [11:25:24] 06Data-Engineering, 10WMF-JobQueue, 10Event-Platform, 13Patch-For-Review, 07Wikimedia-production-error: JobQueueError: Could not enqueue jobs - https://phabricator.wikimedia.org/T371433#10031423 (10Urbanecm_WMF) Okay, jobs should now be back. @Dreamy_Jazz can now login to checkuserwiki with no issues, an... [12:40:41] 06Data-Engineering, 06Data Products, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Remove Benthos from ulsfo hosts - https://phabricator.wikimedia.org/T370741#10031746 (10Fabfur) [12:56:45] 06Data-Engineering, 10WMF-JobQueue, 10Event-Platform, 07Wikimedia-production-error: JobQueueError: Could not enqueue jobs - https://phabricator.wikimedia.org/T371433#10031778 (10Urbanecm_WMF) 05Open→03Resolved a:03Urbanecm_WMF [13:15:32] ottomata: [13:15:34] ottomata: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1058582/1/wmf-config/ext-EventStreamConfig.php [13:15:57] ottomata: trying to see what that has done, but slightly concerned [13:22:15] i was just about to ping you! [13:22:24] ottomata: it's 6:20 am here :P [13:22:39] i forget where you live for some reason I thought you were east coast! [13:22:47] looking as well. [13:23:25] ottomata: i think it's producing events to public streams,i see office.wikimedia.org 1 time in the page-create stream (in kafka) [13:24:07] oh my [13:24:21] yes, because eventbus TYPE_EVENT is enabled for private now [13:24:46] ottomata: but oddly i created a new page in officewiki while watching the public streams (over http) and didn't get an event [13:25:23] hm. [13:25:53] well the right thing to do is to undo that revert, and override mediawiki.job.* streams mediawiki_eventbus.enable => true [13:25:55] making patch [13:27:50] hmm, yea i've found private events in the public mediawiki.recentchange stream now [13:32:57] 10Data-Engineering (Q1 2024 July 1st - September 30th), 10CirrusSearch, 03Discovery-Search (Current work), 10MW-1.43-notes (1.43.0-wmf.15; 2024-07-23): [Search Update Pipeline] Source streams for private wikis - https://phabricator.wikimedia.org/T346046#10031891 (10EBernhardson) Unfortunately reverting it... [13:36:12] ebernhardson: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1058603 [13:37:04] ottomata: lgtm, assuming regex's are auto-detected there (i never realized they did, but since you used them i expect they must ) [13:37:49] 06Data-Engineering, 10WMF-JobQueue, 10Event-Platform, 13Patch-For-Review, 07Wikimedia-production-error: JobQueueError: Could not enqueue jobs - https://phabricator.wikimedia.org/T371433#10031915 (10Ottomata) Thank you okay! So [EventStreamConfig: Re-enable mediawiki_eventbus on private wikis (105858... [13:38:03] regex streams were was a requirement when petr etc. did eventbus job queue implementation, and I really do not like it! [13:38:08] it makes many things bad. [13:38:15] but it will work! [13:43:44] i'm not sure how much followup is necessary, that shipped private wiki events for a few hours. Clearing them out of kafka is probably non-triial [13:43:46] trivial [13:57:55] indeed, i just want to track it to make https://phabricator.wikimedia.org/T370524 be a little more important [14:01:55] 06Data-Engineering, 10Image-Suggestions, 10Event-Platform: Refactor Image Suggestions Feedback > Cassandra Flink Job and Deploy to DSE k8s - https://phabricator.wikimedia.org/T329524#10032021 (10Ottomata) [14:12:55] 06Data-Engineering, 10WMF-JobQueue, 10Event-Platform, 13Patch-For-Review, 07Wikimedia-production-error: JobQueueError: Could not enqueue jobs - https://phabricator.wikimedia.org/T371433#10032045 (10Ottomata) [EventStreamConfig - fix for private wiki streams (1058603)](https://gerrit.wikimedia.org/r/c... [14:44:31] 06Data-Engineering, 10Data-Platform-SRE (2024.07.29 - 2024.08.16): Design a suitable DAG deployment method - https://phabricator.wikimedia.org/T368033#10032168 (10bking) [14:44:35] 06Data-Engineering, 10Data-Platform-SRE (2024.07.29 - 2024.08.16): Some wikibase tables not available in commonswiki_p - https://phabricator.wikimedia.org/T298452#10032169 (10bking) [14:47:14] 14Data-Engineering-Kanban, 10Cassandra, 06Data-Platform-SRE: Properly add aqsloader user (w/ secrets) - https://phabricator.wikimedia.org/T305600#10032191 (10bking) [14:48:16] 06Data-Engineering, 06Discovery-Search, 06Java-Scala-Standardization, 10Data-Platform-SRE (2024.07.29 - 2024.08.16): Update parent pom to disable fetching dependencies from Archiva and use Gitlab instead - https://phabricator.wikimedia.org/T367404#10032196 (10bking) [15:06:13] 06Data-Engineering, 06Data Products, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Remove Benthos from ulsfo hosts - https://phabricator.wikimedia.org/T370741#10032281 (10Fabfur) [15:07:39] 06Data-Engineering, 06Data Products, 10Observability-Logging, 06Traffic, 13Patch-For-Review: Remove Benthos from ulsfo hosts - https://phabricator.wikimedia.org/T370741#10032300 (10Fabfur) Puppet run on all cp-ulsfo hosts has been completed, HAProxy configuration now doesn't send logs anymore to Benthos.... [15:15:14] 06Data-Engineering, 10Cassandra, 10Data Pipelines, 10Data-Platform-SRE (2024.07.29 - 2024.08.16): Create puppet defined type for adding/updating/deleting secrets or other small files on HDFS - https://phabricator.wikimedia.org/T323692#10032316 (10bking) [15:20:20] 06Data-Engineering, 10Data Pipelines: Add support for repository artifacts in Airflow - https://phabricator.wikimedia.org/T322690#10032326 (10Ottomata) In [[ https://docs.google.com/document/d/1E55eW6txPbv1bxZDZtDOd2w9MT1t288bLDy4jEyWvQk/edit?pli=1#heading=h.lgwsqcdh977p | today's DPE sync meeting ]], @Milimet... [15:31:01] 06Data-Engineering, 06collaboration-services, 10Data Pipelines, 10Data-Platform-SRE (2024.07.29 - 2024.08.16), and 2 others: Upgrade Airflow to 2.9.3 - https://phabricator.wikimedia.org/T365449#10032411 (10bking) [15:31:15] 06Data-Engineering, 10Data-Platform-SRE (2024.07.29 - 2024.08.16): Design a suitable DAG deployment method - https://phabricator.wikimedia.org/T368033#10032420 (10Ottomata) Hm, thought: Currently, when airflow-dags is deployed, a scap hook is fired that synchronizes declared artifacts to HDFS. E.g. - https:/... [15:43:25] (03PS29) 10Aqu: Refactor Refine to be triggerd by Airflow [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/1016808 (https://phabricator.wikimedia.org/T356762) [16:44:34] 06Data-Engineering, 06Discovery-Search, 06Java-Scala-Standardization, 10Release-Engineering-Team (Radar): Java projects hosted on Gerrit should publish artifacts to Gitlab - https://phabricator.wikimedia.org/T370400#10032871 (10thcipriani) [16:55:18] 10Data-Engineering (Q1 2024 July 1st - September 30th), 06Data-Platform-SRE, 06Discovery-Search, 06Java-Scala-Standardization, and 2 others: Validate CI integration so that Ci can release Maven artifacts on user's demand - https://phabricator.wikimedia.org/T367403#10032971 (10thcipriani) [17:08:51] 06Data-Engineering, 10Event-Platform: Simplify EventBus code by deprecating EventBusFactory - https://phabricator.wikimedia.org/T370524#10033053 (10Ottomata) [18:04:31] 06Data-Engineering, 10Event-Platform, 07Wikimedia-production-error: EventBus::send incorrectly assumes that JSON response can be converted to an associative array - https://phabricator.wikimedia.org/T371432#10033201 (10Ottomata) I think this warning was recently introduced in work for {T363587}. I think t... [18:31:25] 06Data-Engineering, 10Event-Platform, 07Wikimedia-production-error: EventBus::send incorrectly assumes that JSON response can be converted to an associative array - https://phabricator.wikimedia.org/T371432#10033266 (10brennen) [18:31:58] 06Data-Engineering, 10Data Products (Data Products Sprint 17), 13Patch-For-Review: Add wikitech (labswiki) to the sqoop list - https://phabricator.wikimedia.org/T217792#10033257 (10Milimetric) @VirginiaPoundstone adding this to our sprint because it's basically a no-op and an easy resolution to an old task. [18:53:09] 06Data-Engineering, 10Event-Platform, 07Wikimedia-production-error: EventBus::send incorrectly assumes that JSON response can be converted to an associative array - https://phabricator.wikimedia.org/T371432#10033329 (10Dreamy_Jazz) >>! In T371432#10033199, @Ottomata wrote: > I think this is just a warning lo... [18:55:41] 07Analytics-Data-Problem, 06Data-Engineering, 06Data Products, 10Pageviews-API: Missed pageview data over API - https://phabricator.wikimedia.org/T370108#10033342 (10Mayakp.wiki) [19:12:31] !log deploy latest DAGS to analyics Airflow instance. [19:12:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:22:18] 06Data-Engineering, 03Discovery-Search (Current work), 10MW-1.43-notes (1.43.0-wmf.14; 2024-07-16), 07Wikimedia-production-error: '.event.pageViewId' should be string, '.event.subTest' should be string, '.event.searchSessionId' should be string - https://phabricator.wikimedia.org/T286814#10033415 (10EBernha... [19:25:25] !log Ran DDL for wmf_dumps.wikitext_inconsistent_rows_rc1 https://gitlab.wikimedia.org/repos/data-engineering/dumps/mediawiki-content-dump/-/blob/91a343657d8053858a6c086004c549b9aa3245c0/hql/create-wmf_dumps_wikitext_inconsistent_rows.hql T368756 [19:25:28] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:25:28] T368756: Airflow job to orchestrate the emission mechanism - https://phabricator.wikimedia.org/T368756