[00:07:16] (03PS37) 10AGueyte: Basic ipinfo instrument setup [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/753548 (https://phabricator.wikimedia.org/T296415) [00:08:14] (03CR) 10jerkins-bot: [V: 04-1] Basic ipinfo instrument setup [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/753548 (https://phabricator.wikimedia.org/T296415) (owner: 10AGueyte) [00:09:59] (03PS38) 10AGueyte: Basic ipinfo instrument setup [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/753548 (https://phabricator.wikimedia.org/T296415) [00:24:55] (03CR) 10AGueyte: Basic ipinfo instrument setup (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/753548 (https://phabricator.wikimedia.org/T296415) (owner: 10AGueyte) [05:42:13] 10Quarry, 10cloud-services-team (Kanban): quarry-nfs-1 went down; quarry is offline - https://phabricator.wikimedia.org/T302154 (10zhuyifei1999) p:05Unbreak!→03High [07:03:40] 10Data-Engineering, 10Data-Engineering-Kanban: Some varnishkafka instances dropped traffic for a long time due to the wrong version of the package installed - https://phabricator.wikimedia.org/T300164 (10elukey) The varnishkafka package version will be handled in T302301 by the Traffic team. [07:59:27] (HiveServerHeapUsage) firing: Hive Server JVM Heap usage is above 80% on an-test-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-test-coord1001:10100 - https://alerts.wikimedia.org [08:14:27] (HiveServerHeapUsage) resolved: Hive Server JVM Heap usage is above 80% on an-test-coord1001:10100 - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Cluster/Hive/Alerts#Hive_Server_Heap_Usage - https://grafana.wikimedia.org/d/000000379/hive?panelId=7&fullscreen&orgId=1&var-instance=an-test-coord1001:10100 - https://alerts.wikimedia.org [10:03:59] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: Define the Kubernetes Deployments for Datahub - https://phabricator.wikimedia.org/T301454 (10JMeybohm) >>! In T301454#7729037, @BTullis wrote: > 1) How should I go about specifying that these deployments should be able to ac... [12:18:05] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: Define the Kubernetes Deployments for Datahub - https://phabricator.wikimedia.org/T301454 (10BTullis) Thanks again @JMeybohm > I don't see helm lint failing with the current patch set. Maybe I'm missing something? Apologi... [13:40:30] hey folks. can i just check that https://phabricator.wikimedia.org/T302233 is on your radar? [13:40:43] razzi: i was told you might be the person to talk to :) [13:43:32] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog, 10Patch-For-Review: Define the Kubernetes Deployments for Datahub - https://phabricator.wikimedia.org/T301454 (10JMeybohm) >>! In T301454#7731707, @BTullis wrote: > I've now uploaded another patchset ([[https://gerrit.wikimedia.org/r/c/operatio... [14:10:37] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: [Airflow] Troubleshoot MySQL connection issues - https://phabricator.wikimedia.org/T298893 (10EChetty) a:03mforns [14:26:01] PROBLEM - Check unit status of produce_canary_events on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [14:36:15] RECOVERY - Check unit status of produce_canary_events on an-launcher1002 is OK: OK: Status of the systemd unit produce_canary_events https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [16:08:09] kormat: razzi is taking care of the views but I owed him a brainbounce yesterday, so you should blame me for the lag. It looks like we need to account for a recent schema change, shouldn't be too bad [16:08:32] but yes, it's on our radar [16:08:49] milimetric: ah ok :) there's a current _ongoing_ schema change (that i'm running). meaning that while some views have already broken, some are still yet to break (presumably) [16:09:23] I see, /me looks for a list of changes [16:09:45] https://phabricator.wikimedia.org/T300774 is the relevant schema change [16:09:57] it's currently running against s3. [16:10:25] got it, so just these three columns: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/FlaggedRevs/+/757949/4/backend/schema/mysql/patch-drop-fr_img.sql [16:10:39] yep! [16:11:28] whether they run on one or more wikis I don't think matters too much, the views should probably just pretend it ran on all the wikis, and deal with some disfunction until that's true, or do you have reason to expect that'll take longer than a week or so? [16:11:52] you know, I actually don't know what the expected service level is here, I'll see if razzi knows [16:16:22] the referenced columns are already unused, so I'd just remove them from all views. [16:18:08] the schema change should be done in the next week [16:29:14] 10Data-Engineering, 10DBA, 10Data-Services, 10MediaWiki-extensions-FlaggedRevs, 10cloud-services-team: Toolforge db: View 'fiwiki_p.flaggedrevs' references invalid table/column/rights to use them - https://phabricator.wikimedia.org/T302233 (10nskaggs) According to the responsibilities matrix, WMCS still... [16:29:42] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Data-Services, and 2 others: Toolforge db: View 'fiwiki_p.flaggedrevs' references invalid table/column/rights to use them - https://phabricator.wikimedia.org/T302233 (10Milimetric) a:05Kormat→03razzi [17:03:19] ping standup a-team [17:04:36] (folks can't make it to the SRE sync sorry) [17:18:25] 10Data-Engineering-Kanban, 10Data-Catalog: [[wikitech:Data Catalog Application Evaluation Rubric]] links to some non-public Google Doc "execution plan" - https://phabricator.wikimedia.org/T299900 (10Milimetric) 05Open→03Resolved [17:18:54] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog, 10Epic: Data Catalog Technical Evaluation - https://phabricator.wikimedia.org/T293643 (10Milimetric) [17:18:58] 10Data-Engineering, 10Data-Engineering-Kanban, 10Epic: Finish evaluation of "other" Data Governance Options - https://phabricator.wikimedia.org/T296672 (10Milimetric) 05Open→03Resolved [17:26:58] elukey: No worries. Catch you another time. [17:27:15] 10Data-Engineering, 10Data-Engineering-Kanban: Kerberos identity for bmansurov - https://phabricator.wikimedia.org/T300450 (10razzi) 05Open→03Resolved a:03razzi Sounds good! [17:32:18] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog, 10Release Pipeline, 10Patch-For-Review: Create DataHub containers with deployment pipeline - https://phabricator.wikimedia.org/T301453 (10BTullis) 05Open→03Resolved [17:32:20] 10Data-Engineering, 10Data-Catalog, 10Epic: Data Catalog MVP - https://phabricator.wikimedia.org/T299910 (10BTullis) [17:46:12] (03PS1) 10Nmaphophe: Initial [analytics/refinery] - 10https://gerrit.wikimedia.org/r/765314 [17:48:04] (03CR) 10Nmaphophe: "Hi Marcel," [analytics/refinery] - 10https://gerrit.wikimedia.org/r/765314 (owner: 10Nmaphophe) [19:09:35] (03CR) 10Mforns: "Queries look good!!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/765314 (owner: 10Nmaphophe) [19:17:38] (03PS2) 10Nmaphophe: Add mediawiki directories [analytics/refinery] - 10https://gerrit.wikimedia.org/r/765314 [19:29:59] hi milimetric - let's talk when you have a moment about the wivivi data :) [19:30:49] 10Analytics, 10Analytics-Wikistats, 10Data-Engineering, 10Data-Engineering-Kanban, and 5 others: Wikistats pageview data missing counts for Mobile App pageviews on Commons, going back to 2020-11 - https://phabricator.wikimedia.org/T299439 (10JAllemandou) Hi @SNowick_WMF - Can you confirm there is nothing e... [19:35:12] (03PS39) 10AGueyte: Basic ipinfo instrument setup [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/753548 (https://phabricator.wikimedia.org/T296415) [19:35:49] (03CR) 10jerkins-bot: [V: 04-1] Basic ipinfo instrument setup [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/753548 (https://phabricator.wikimedia.org/T296415) (owner: 10AGueyte) [19:45:06] (03CR) 10Mforns: "Oh, I meant to say to put the 3 create table files in:" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/765314 (owner: 10Nmaphophe) [19:55:02] (03PS3) 10Nmaphophe: Table creates and queries to be in one directory [analytics/refinery] - 10https://gerrit.wikimedia.org/r/765314 [20:00:46] (03PS4) 10Nmaphophe: Table creates and queries [analytics/refinery] - 10https://gerrit.wikimedia.org/r/765314 [20:02:31] (03CR) 10Mforns: [V: 03+2 C: 03+2] "Thank you!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/765314 (owner: 10Nmaphophe) [20:10:38] (03PS40) 10AGueyte: Basic ipinfo instrument setup [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/753548 (https://phabricator.wikimedia.org/T296415) [20:55:33] 10Analytics, 10Analytics-Wikistats, 10Data-Engineering, 10Data-Engineering-Kanban, and 5 others: Wikistats pageview data missing counts for Mobile App pageviews on Commons, going back to 2020-11 - https://phabricator.wikimedia.org/T299439 (10SNowick_WMF) Hi @JAllemandou yes, confirmed this needs to be inve... [22:57:39] 10Data-Engineering, 10Data-Engineering-Kanban, 10DBA, 10Data-Services, and 2 others: Toolforge db: View 'fiwiki_p.flaggedrevs' references invalid table/column/rights to use them - https://phabricator.wikimedia.org/T302233 (10razzi) I went ahead and ran the maintain_views for the single table you specified... [23:00:20] !log sudo maintain-views --table flaggedrevs --databases fiwiki on clouddb1014.eqiad.wmnet and clouddb1018.eqiad.wmnet for T302233 [23:00:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [23:00:23] T302233: Toolforge db: View 'fiwiki_p.flaggedrevs' references invalid table/column/rights to use them - https://phabricator.wikimedia.org/T302233 [23:05:27] sorry I missed your ping joal, hopefully you're asleep by now and I'll talk to you tomorrow. That's not urgent at all [23:05:48] (but I started a thread on slack about it, in the data-engineering-team channel, if you prefer [23:30:17] (03PS1) 10MewOphaswongse: Add an image: add confirm_reject_suggestion action [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/765351 (https://phabricator.wikimedia.org/T302429)