[08:38:26] 10Quarry: Kill all queries stuck in running or queued state - https://phabricator.wikimedia.org/T307263 (10Aklapper) [08:43:18] 10Data-Engineering, 10Data-Engineering-Kanban, 10Cassandra, 10Patch-For-Review: Enable Cassandra encryption (inter-node & client) - https://phabricator.wikimedia.org/T307798 (10BTullis) a:03BTullis [09:05:17] 10Data-Engineering-Kanban, 10Data-Catalog: User Experience: Authentication - https://phabricator.wikimedia.org/T307711 (10BTullis) [10:57:10] !log upgrading datahub to version 0.8.34 [10:57:13] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:06:22] 10Data-Engineering-Kanban, 10Data-Catalog: Spike: Evaluate datahub schema versioning support - https://phabricator.wikimedia.org/T307716 (10BTullis) I have upgraded DataHub to version 0.8.34 and now the //Blame View// is accessible, which is the first way to visualize schema changes over time. {F35151186,width... [11:14:56] 10Quarry: Kill all queries stuck in running or queued state - https://phabricator.wikimedia.org/T307263 (10rook) Agreed. Though I believe this is a duplicate of T278583, please re-open if this is not the case. [11:15:27] 10Quarry: Kill all queries stuck in running or queued state - https://phabricator.wikimedia.org/T307263 (10rook) [11:16:01] 10Quarry, 10cloud-services-team (Kanban): Quarry should detect a dead worker and report something better than "running" forever - https://phabricator.wikimedia.org/T278583 (10rook) [11:18:14] 10Quarry, 10cloud-services-team (Kanban): Quarry should detect a dead worker and report something better than "running" forever - https://phabricator.wikimedia.org/T278583 (10rook) Workers sometimes die unexpectedly, and leave the db in the state described "Running". One method to clean this up may be to inspe... [11:20:17] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Upgrade Datahub - https://phabricator.wikimedia.org/T308052 (10BTullis) I have successfully upgraded DataHub to version 0.8.34. Documentation work has been started on here: https://wikitech.wikimedia.org/wiki/Analytics/Systems/DataHub/Upgrading... [11:20:34] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Upgrade Datahub - https://phabricator.wikimedia.org/T308052 (10BTullis) p:05Triage→03Medium [12:08:09] 10Quarry, 10Data-Services: Unable to use `force index` on replicas due to view layer intervention (Key 'PRIMARY' doesn't exist in table 'page') - https://phabricator.wikimedia.org/T251980 (10Aklapper) 05Open→03Declined Does not look like. [12:24:11] 10Data-Engineering, 10Data-Engineering-Kanban, 10Data-Catalog: Upgrade Datahub - https://phabricator.wikimedia.org/T308052 (10BTullis) Ah, there is an issue with the mae and mce consumer jobs - They are affected by the bug mentioned here: https://datahubspace.slack.com/archives/C029A3M079U/p1652561206592629... [12:24:15] (03PS1) 10Btullis: Make a small change to trigger the build process [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/793039 (https://phabricator.wikimedia.org/T308052) [12:30:34] (03PS11) 10Joal: Update to spark-3 and scala-2.12 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/656897 [12:51:13] (03CR) 10Btullis: [C: 03+2] Make a small change to trigger the build process [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/793039 (https://phabricator.wikimedia.org/T308052) (owner: 10Btullis) [13:05:41] 10Data-Engineering, 10Data-Engineering-Kanban, 10Cassandra, 10Patch-For-Review: Enable Cassandra encryption (inter-node & client) - https://phabricator.wikimedia.org/T307798 (10BTullis) As mentioned on T288470 I'd rather use the cfssl classes for the certificate and key generation, if it's not going to be... [13:06:16] (03Merged) 10jenkins-bot: Make a small change to trigger the build process [analytics/datahub] (wmf) - 10https://gerrit.wikimedia.org/r/793039 (https://phabricator.wikimedia.org/T308052) (owner: 10Btullis) [14:05:17] 10Data-Engineering, 10Data-Engineering-Kanban, 10Cassandra, 10Patch-For-Review: Enable Cassandra encryption (inter-node & client) - https://phabricator.wikimedia.org/T307798 (10Eevans) >>! In T307798#7938274, @BTullis wrote: > As mentioned on T288470 I'd rather use the cfssl classes for the certificate an... [14:05:41] aqu, joal: standup? [14:32:13] (VarnishkafkaNoMessages) firing: ... [14:32:19] varnishkafka for instance cp3056:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=esams%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp3056:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [14:42:13] (VarnishkafkaNoMessages) resolved: ... [14:42:13] varnishkafka for instance cp3056:9132 is not logging cache_text requests from eventlogging - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=esams%20prometheus/ops&var-source=eventlogging&var-cp_cluster=cache_text&var-instance=cp3056:9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages [16:33:37] 10Data-Engineering, 10Data-Engineering-Kanban, 10Cassandra, 10Patch-For-Review: Enable Cassandra encryption (inter-node & client) - https://phabricator.wikimedia.org/T307798 (10BTullis) OK, understood. [[https://cfssl.org/|cfssl]] Is the CloudFlare PKI toolkit. It's what we've started using recently for mo... [16:51:12] 10Data-Engineering, 10Data-Engineering-Kanban, 10Cassandra, 10Patch-For-Review: Enable Cassandra encryption (inter-node & client) - https://phabricator.wikimedia.org/T307798 (10Eevans) >>! In T307798#7939140, @BTullis wrote: > OK, understood. [[https://cfssl.org/|cfssl]] Is the CloudFlare PKI toolkit. It's... [17:00:21] 10Data-Engineering, 10Data-Engineering-Kanban, 10Cassandra, 10Patch-For-Review: Enable Cassandra encryption (inter-node & client) - https://phabricator.wikimedia.org/T307798 (10Eevans) One tiny nit to consider before committing key material (regardless of tooling): In [[ https://gerrit.wikimedia.org/r/c/op... [17:16:02] 10Analytics, 10Data-Engineering, 10Event-Platform, 10Product-Analytics, 10MW-1.36-notes (1.36.0-wmf.18; 2020-11-17): Migrate legacy metawiki schemas to Event Platform - https://phabricator.wikimedia.org/T259163 (10cooltey) [17:19:47] 10Data-Engineering, 10Data-Engineering-Kanban, 10Cassandra, 10Patch-For-Review: Enable Cassandra encryption (inter-node & client) - https://phabricator.wikimedia.org/T307798 (10BTullis) >>! In T307798#7939194, @Eevans wrote: >>>! In T307798#7939140, @BTullis wrote: >> OK, understood. [[https://cfssl.org/|c... [17:21:31] 10Data-Engineering, 10Data-Engineering-Kanban, 10Cassandra, 10Patch-For-Review: Enable Cassandra encryption (inter-node & client) - https://phabricator.wikimedia.org/T307798 (10BTullis) >>! In T307798#7939224, @Eevans wrote: > One tiny nit to consider before committing key material (regardless of tooling):... [18:05:08] (03PS1) 10MewOphaswongse: Add mentorship-optin module to homepagemodule schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/793105 (https://phabricator.wikimedia.org/T308690) [18:05:17] (03PS1) 10Razzi: Make an-tool1011.eqiad.wmnet the new turnilo staging [analytics/turnilo/deploy] - 10https://gerrit.wikimedia.org/r/793106 (https://phabricator.wikimedia.org/T308597) [18:28:09] (03PS12) 10Joal: Update to spark-3 and scala-2.12 [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/656897 [18:47:42] (03CR) 10Razzi: [V: 03+2 C: 03+2] Make an-tool1011.eqiad.wmnet the new turnilo staging [analytics/turnilo/deploy] - 10https://gerrit.wikimedia.org/r/793106 (https://phabricator.wikimedia.org/T308597) (owner: 10Razzi) [20:59:00] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: Improvements of artifacts cache - https://phabricator.wikimedia.org/T307115 (10Ottomata) [21:31:14] (03CR) 10Gergő Tisza: [C: 03+2] Add mentorship-optin module to homepagemodule schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/793105 (https://phabricator.wikimedia.org/T308690) (owner: 10MewOphaswongse) [21:32:19] (03Merged) 10jenkins-bot: Add mentorship-optin module to homepagemodule schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/793105 (https://phabricator.wikimedia.org/T308690) (owner: 10MewOphaswongse) [21:37:31] (03CR) 10Jdlrobson: [C: 03+2] Add is-anon, is-bot properties to schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791422 (https://phabricator.wikimedia.org/T307381) (owner: 10Clare Ming) [21:38:06] (03Merged) 10jenkins-bot: Add is-anon, is-bot properties to schema [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/791422 (https://phabricator.wikimedia.org/T307381) (owner: 10Clare Ming)