[05:05:15] (EventgateLoggingExternalLatency) firing: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [05:15:15] (EventgateLoggingExternalLatency) resolved: Elevated latency for POST events on eventgate-logging-external in eqiad. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org/?q=alertname%3DEventgateLoggingExternalLatency [08:31:34] 10Data-Engineering, 10DC-Ops, 10Infrastructure-Foundations: clouddb1021 missing firmware; debian installer cannot connect to network - https://phabricator.wikimedia.org/T306148 (10Marostegui) >>! In T306148#7854405, @razzi wrote: > @marostegui says I should tag @MoritzMuehlenhoff - hopefully we can all solve... [10:24:06] 10Data-Engineering: Error with refinery-drop-mediawiki-snapshots: table specs not matching partitions for wmf/wikidata/entity and wmf/wikidata/item_page_link - https://phabricator.wikimedia.org/T305591 (10Milimetric) p:05Triage→03High a:05razzi→03None removing Razzi, resetting to high and pinging @Ottoma... [10:26:49] 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban, 10Platform Engineering, 10Product-Analytics: AQS `edited-pages/new` metric does not make clear that the value is net of deletions - https://phabricator.wikimedia.org/T240860 (10Milimetric) a:05mforns→03None @EChetty I'm unassigning this and... [10:33:23] 10Data-Engineering, 10DC-Ops, 10Infrastructure-Foundations: clouddb1021 missing firmware; debian installer cannot connect to network - https://phabricator.wikimedia.org/T306148 (10Marostegui) Once clouddb1021 is reimaged please run: `maintain-views --database enwiki --table flaggedtemplates --replace` as th... [10:34:59] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Connect MVP to Hive metastore [Mile Stone 4] - https://phabricator.wikimedia.org/T299897 (10Milimetric) When I get back I'll write an airflow job that does the ingestion on a regular basis. For now I'd just like @EChetty and @odimitrijevic to ta... [13:22:49] (03CR) 10Vivian Rook: [C: 03+2] Expose history of query revisions [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/773578 (https://phabricator.wikimedia.org/T100982) (owner: 10Vivian Rook) [13:26:26] (03Merged) 10jenkins-bot: Expose history of query revisions [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/773578 (https://phabricator.wikimedia.org/T100982) (owner: 10Vivian Rook) [13:28:45] 10Quarry, 10Patch-For-Review: Add list of query executions to the query page side-bar - https://phabricator.wikimedia.org/T100982 (10rook) 05Open→03Resolved [13:30:26] 10Quarry: Make quarry history ascending - https://phabricator.wikimedia.org/T306340 (10rook) [13:32:31] 10Quarry: History shows blank for blank queries - https://phabricator.wikimedia.org/T306342 (10rook) [13:34:08] 10Quarry: Truncate text on quarry history links - https://phabricator.wikimedia.org/T306343 (10rook) [13:47:19] 10Quarry: Add list of query executions to the query page side-bar - https://phabricator.wikimedia.org/T100982 (10rook) [13:47:24] 10Quarry: Add links to past query runs - https://phabricator.wikimedia.org/T137650 (10rook) [13:48:07] 10Quarry: Add list of query executions to the query page side-bar - https://phabricator.wikimedia.org/T100982 (10rook) [13:48:10] 10Quarry: Show query code revisions and runs history - https://phabricator.wikimedia.org/T206482 (10rook) [13:50:09] 10Quarry: Make quarry history descending - https://phabricator.wikimedia.org/T306340 (10rook) [14:08:34] (03CR) 10AGueyte: Add new event action (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/774980 (https://phabricator.wikimedia.org/T296428) (owner: 10AGueyte) [14:47:52] PROBLEM - eventgate-analytics-external validation error rate too high on alert1001 is CRITICAL: 5.58 gt 2 https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-service=eventgate-analytics-external&var-stream=All&var-kafka_broker=All&var-kafka_producer_type=All&var-dc=thanos [15:03:28] RECOVERY - eventgate-analytics-external validation error rate too high on alert1001 is OK: (C)2 gt (W)1 gt 0.862 https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?orgId=1&refresh=1m&var-service=eventgate-analytics-external&var-stream=All&var-kafka_broker=All&var-kafka_producer_type=All&var-dc=thanos [17:38:58] (03CR) 10Vivian Rook: "nit, the ! marks make the text wrap to the next line. Also make it a little inconsistent with the other selections in the same area." [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/780899 (https://phabricator.wikimedia.org/T303028) (owner: 10David Caro) [17:39:06] (03PS3) 10Vivian Rook: Add links to feedback and bug pre-filled tasks [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/780899 (https://phabricator.wikimedia.org/T303028) (owner: 10David Caro) [18:22:56] (03CR) 10Vivian Rook: [C: 03+1] Add links to feedback and bug pre-filled tasks [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/780899 (https://phabricator.wikimedia.org/T303028) (owner: 10David Caro) [18:22:59] (03CR) 10Vivian Rook: [C: 03+2] Add links to feedback and bug pre-filled tasks [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/780899 (https://phabricator.wikimedia.org/T303028) (owner: 10David Caro) [18:26:37] (03Merged) 10jenkins-bot: Add links to feedback and bug pre-filled tasks [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/780899 (https://phabricator.wikimedia.org/T303028) (owner: 10David Caro) [18:37:12] (03PS11) 10Sharvaniharan: Android schemas migrated from legacy MobileWikiAppAppearanceSettings MobileWikiAppFindInPage MobileWikiAppLinkPreview MobileWikiAppPageScroll MobileWikiAppToCInteraction [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/778603 (https://phabricator.wikimedia.org/T306385) [18:39:56] (03CR) 10Sharvaniharan: "Please review when you get a chance :-)" [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/778603 (https://phabricator.wikimedia.org/T306385) (owner: 10Sharvaniharan) [18:59:54] (03PS5) 10Vivian Rook: Remember recent queries filter last used by a user. [analytics/quarry/web] - 10https://gerrit.wikimedia.org/r/176506 (https://phabricator.wikimedia.org/T76084) (owner: 10Rtnpro) [19:24:16] 10Data-Engineering, 10Generated Data Platform, 10SRE, 10SRE-Access-Requests: Request to grant cparle and mfossati login to an-airflow1003.eqiad.wmne - https://phabricator.wikimedia.org/T306057 (10Dzahn) @ottomata We can use the existing group `airflow-search-admins`. It will give them access to machines wi... [19:32:34] 10Data-Engineering, 10DC-Ops, 10Infrastructure-Foundations: clouddb1021 missing firmware; debian installer cannot connect to network - https://phabricator.wikimedia.org/T306148 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by razzi@cumin1001 for host clouddb1021.eqiad.wmnet with OS... [20:02:08] 10Data-Engineering, 10DC-Ops, 10Infrastructure-Foundations: clouddb1021 missing firmware; debian installer cannot connect to network - https://phabricator.wikimedia.org/T306148 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by razzi@cumin1001 for host clouddb1021.eqiad.wmnet with OS bust... [20:16:53] 10Data-Engineering, 10DC-Ops, 10Infrastructure-Foundations: clouddb1021 missing firmware; debian installer cannot connect to network - https://phabricator.wikimedia.org/T306148 (10razzi) I reimaged the host back to Buster for now, which went smoothly. Replication lag is a few days behind but is catching up g... [20:18:12] 10Data-Engineering, 10DC-Ops, 10Infrastructure-Foundations: clouddb1021 missing network firmware bnx2x/bnx2x-e2-7.13.21.0.fw in Debian 11 Bullseye - https://phabricator.wikimedia.org/T306148 (10razzi) [22:43:29] 10Data-Engineering, 10Equity-Landscape: Milestone: Dashboard Mockup Complete - https://phabricator.wikimedia.org/T305476 (10JAnstee_WMF) 05Open→03In progress a:03KCVelaga_WMF [22:43:31] 10Data-Engineering, 10Equity-Landscape, 10Epic: Deploy the GDI Equity Landscape Dashboard - https://phabricator.wikimedia.org/T305468 (10JAnstee_WMF) [22:43:57] 10Data-Engineering, 10Equity-Landscape: Milestone: Dashboard Interaction Map Complete - https://phabricator.wikimedia.org/T305477 (10JAnstee_WMF) a:03JAnstee_WMF