[00:33:53] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Connect MVP to Hive metastore [Mile Stone 4] - https://phabricator.wikimedia.org/T299897 (10Milimetric) I think it makes sense to look at the karapace logs. I tried it with 'console' as the sink and it worked fine, no failures. And I cleaned ou... [00:59:27] (VarnishkafkaNoMessages) firing: ... [00:59:27] varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp2027:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [04:59:27] (VarnishkafkaNoMessages) firing: ... [04:59:27] varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp2027:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [06:32:39] PROBLEM - Check unit status of mediawiki-history-drop-snapshot on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit mediawiki-history-drop-snapshot https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [08:59:27] (VarnishkafkaNoMessages) firing: ... [08:59:27] varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp2027:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [09:47:31] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Services, 10Documentation, 10cloud-services-team (Kanban): Document on wikitech the general process of getting a table/column exposed to Wiki Replica users - https://phabricator.wikimedia.org/T209992 (10EChetty) >>! In T209992#7855983, @bd808 wrote: >... [12:59:27] (VarnishkafkaNoMessages) firing: ... [12:59:27] varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp2027:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [14:06:28] (03PS6) 10Luke Bowmaker: Image Suggestions feature schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/779052 [14:31:20] RECOVERY - Check unit status of mediawiki-history-drop-snapshot on an-launcher1002 is OK: OK: Status of the systemd unit mediawiki-history-drop-snapshot https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [15:33:55] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Connect MVP to Hive metastore [Mile Stone 4] - https://phabricator.wikimedia.org/T299897 (10Milimetric) On their slack they said this looked like we had mismatching client/server versions. So maybe it's possible 0.8.32 is not fully rolled out so... [16:05:04] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Connect MVP to Hive metastore [Mile Stone 4] - https://phabricator.wikimedia.org/T299897 (10Milimetric) Aha! I was wrong, server must still be on 0.8.28, I rolled back the datahub client to 0.8.28 and ingestion started working. All good then, I... [16:40:20] 10Data-Engineering-Kanban, 10Airflow: Migrate the Clickstream jobs to Airflow - https://phabricator.wikimedia.org/T305843 (10Antoine_Quhen) a:03Antoine_Quhen [16:59:27] (VarnishkafkaNoMessages) firing: ... [16:59:27] varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp2027:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [17:04:12] (VarnishkafkaNoMessages) resolved: ... [17:04:12] varnishkafka for instance cp2027:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp2027:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [17:07:59] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Services, 10Documentation, 10cloud-services-team (Kanban): Document on wikitech the general process of getting a table/column exposed to Wiki Replica users - https://phabricator.wikimedia.org/T209992 (10bd808) 05Declined→03Open [17:08:42] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Services, 10Documentation, 10cloud-services-team (Kanban): Document on wikitech the general process of getting a table/column exposed to Wiki Replica users - https://phabricator.wikimedia.org/T209992 (10bd808) >>! In T209992#7857432, @EChetty wrote: >... [18:36:10] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Services, 10Documentation, 10cloud-services-team (Kanban): Document on wikitech the general process of getting a table/column exposed to Wiki Replica users - https://phabricator.wikimedia.org/T209992 (10EChetty) Hey! Thank you for the context. As exp... [19:08:46] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Connect MVP to Hive metastore [Mile Stone 4] - https://phabricator.wikimedia.org/T299897 (10BTullis) >>! In T299897#7857881, @Milimetric wrote: > Aha! I was wrong, server must still be on 0.8.28, I rolled back the datahub client to 0.8.28 and in... [19:25:51] (03PS4) 10Sharvaniharan: Android schemas migrated from legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/778603 [19:26:22] (03CR) 10jerkins-bot: [V: 04-1] Android schemas migrated from legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/778603 (owner: 10Sharvaniharan) [19:26:54] (03PS5) 10Sharvaniharan: Android schemas migrated from legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/778603 [19:37:19] 10Data-Engineering, 10Data-Engineering-Kanban, 10Product-Analytics, 10Superset, 10Patch-For-Review: Upgrade Superset to 1.4.2 - https://phabricator.wikimedia.org/T304972 (10mpopov) We're currently blocked on this by not being able to SSH to an-tool1005 and are being asked for a password. [19:41:57] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Services, 10Documentation, 10cloud-services-team (Kanban): Document on wikitech the general process of getting a table/column exposed to Wiki Replica users - https://phabricator.wikimedia.org/T209992 (10bd808) >>! In T209992#7858398, @EChetty wrote: >... [19:50:55] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Connect MVP to Hive metastore [Mile Stone 4] - https://phabricator.wikimedia.org/T299897 (10Milimetric) It's all yours after today, so you can definitely upgrade on Tuesday. I'm going to leave some ingestion running at the end of the day, but th... [20:01:17] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Services, 10Documentation, 10cloud-services-team (Kanban): Document on wikitech the general process of getting a table/column exposed to Wiki Replica users - https://phabricator.wikimedia.org/T209992 (10EChetty) > Seems fine to me. It honestly also s... [20:03:46] 10Data-Engineering, 10Data-Services, 10Documentation, 10cloud-services-team (Kanban): Provide documentation for toolforge users to request access to unexposed data through WikiReplicas - https://phabricator.wikimedia.org/T209992 (10EChetty) [20:04:18] 10Data-Engineering, 10Data-Services, 10Documentation, 10cloud-services-team (Kanban): Provide documentation for toolforge users to request access to unexposed data through WikiReplicas - https://phabricator.wikimedia.org/T209992 (10EChetty) [20:07:47] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Connect MVP to Hive metastore [Mile Stone 4] - https://phabricator.wikimedia.org/T299897 (10Milimetric) ====Notes from the Field, Ingestion edition==== * event_sanitized, no profiling: 75 minutes, each table takes about 15-20 seconds once it get... [20:51:25] (03PS6) 10Sharvaniharan: Android schemas migrated from legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/778603 [21:05:34] (03PS7) 10Sharvaniharan: Android schemas migrated from legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/778603 [21:06:53] (03PS8) 10Sharvaniharan: Android schemas migrated from legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/778603 [21:25:11] (03PS9) 10Sharvaniharan: Android schemas migrated from legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/778603 [21:26:15] (03CR) 10jerkins-bot: [V: 04-1] Android schemas migrated from legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/778603 (owner: 10Sharvaniharan) [21:31:53] (03PS10) 10Sharvaniharan: Android schemas migrated from legacy [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/778603