[00:14:33] <wikibugs>	 10Data-Engineering, 10Product-Analytics, 10wmfdata-python: Cannot query string data from MariaDB using Wmfdata-Python - https://phabricator.wikimedia.org/T319360 (10nshahquinn-wmf) I thought that maybe I could work around this by installing a newer version of mysql-connector-python (`conda install -c conda-f...
[05:51:14] <icinga-wm>	 RECOVERY - SSH on analytics1076.mgmt is OK: SSH OK - OpenSSH_7.4 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook
[07:11:46] <wikibugs>	 10Analytics-Radar, 10Data-Engineering-Radar, 10Event-Platform Value Stream, 10Patch-For-Review: Move Kafka Jumbo's TLS clients to the new bundle - https://phabricator.wikimedia.org/T296064 (10elukey) I added a detailed plan for kafka-main in T319372 :)
[07:47:28] <awight>	 joal: I don't see an obvious "name" for the schema, though.  https://schema.wikimedia.org/repositories//secondary/jsonschema/analytics/mediawiki/maps/interaction/current.yaml
[07:50:09] <awight>	 The only other place a name might come from seems to be the event stream name set in $wgEventStreams, but that isn't simple either: "mediawiki.maps_interaction"
[08:00:09] <joal>	 awight: Data is present on HDFS under /wmf/data/event/mediawiki_maps_interaction, and in hive under event.mediawiki_maps_interaction - Looking at the schema, it is inded not obvious how to extract the name
[08:26:35] <awight>	 Well that's fantastic, I guess I just needed to be patient about the first data landing!
[11:03:58] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: varnishkafka on cp2036 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp2036%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[11:07:12] <jinxer-wm>	 (VarnishkafkaNoMessages) resolved: varnishkafka on cp2036 is not sending enough cache_upload requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=codfw%20prometheus/ops&var-cp_cluster=cache_upload&var-instance=cp2036%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[13:06:12] <jinxer-wm>	 (VarnishkafkaNoMessages) firing: varnishkafka on cp4032 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp4032%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[13:09:16] <mforns>	 heya teammm!
[13:11:12] <jinxer-wm>	 (VarnishkafkaNoMessages) resolved: varnishkafka on cp4032 is not sending enough cache_text requests - https://wikitech.wikimedia.org/wiki/Analytics/Systems/Varnishkafka - https://grafana.wikimedia.org/d/000000253/varnishkafka?orgId=1&var-datasource=ulsfo%20prometheus/ops&var-cp_cluster=cache_text&var-instance=cp4032%3A9132&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DVarnishkafkaNoMessages
[13:13:40] <wikibugs>	 (03CR) 10Aqu: [V: 03+2 C: 03+2] "LGTM!" [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/838238 (owner: 10Snwachukwu)
[13:21:48] <SandraEbele>	 !log deploying fix for projective tags on airflow.
[13:21:50] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:22:01] <wikibugs>	 (03Merged) 10jenkins-bot: Bump changelog.md to v0.2.8 before release [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/838238 (owner: 10Snwachukwu)
[13:33:10] <wikibugs>	 10Data-Engineering, 10Equity-Landscape: Readership Output Rank Metrics - https://phabricator.wikimedia.org/T306617 (10KCVelaga_WMF) @JAnstee_WMF    - Missing countries should be fixed now.   - Northern America is present in both continent and sub-continent level aggregations.  As my calculations are based on N...
[13:42:00] <wikibugs>	 10Data-Engineering, 10Equity-Landscape: Readership input metrics - https://phabricator.wikimedia.org/T309273 (10KCVelaga_WMF) > Does the unique devices data actually come from the same wmf.pageview_hourly or is this a copy/paste error?  I think this might have been a typo, as [[https://wikitech.wikimedia.org/w...
[13:48:01] <SandraEbele>	 !log killed Oozie projectview-geo-coord job
[13:48:03] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[13:49:12] <SandraEbele>	 !log Started Airflow projectview_geo job
[13:49:13] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[14:00:07] <wikibugs>	 10Data-Engineering, 10Event-Platform Value Stream (Sprint 02), 10Patch-For-Review: Design Schema for page state and page state with content (enriched) streams - https://phabricator.wikimedia.org/T308017 (10Ottomata) > listen to your stream and compact it with all visibility change events You mean join with?...
[14:05:30] <mforns>	 !log starting refinery deploy - regular weekly train
[14:05:31] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[14:24:43] <mforns>	 btullis: hi! :] One question please, I deployed refinery to the regular hosts, and went well, but then when deploying with `-e thin` both labstore hosts failed. I've seen there were some problems with labstore hosts, related to jupyter, could this be caused by the same issue?
[14:26:25] <wmf-insecte>	 Starting build #113 for job analytics-refinery-maven-release-docker
[14:33:55] <mforns>	 !log finished refinery deploy - regular weekly train
[14:33:56] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[14:38:48] <wmf-insecte>	 Project analytics-refinery-maven-release-docker build #113: 09SUCCESS in 12 min: https://integration.wikimedia.org/ci/job/analytics-refinery-maven-release-docker/113/
[16:31:16] <joal>	 mforns: I left you a message on the chan of the meeting - not sure if you've seen it
[16:31:31] <mforns>	 yes, responded joal :]
[16:32:39] <joal>	 mforns: now that you know m concern, can you confirm it's a valid one?
[16:33:03] * joal needs to bu a nerw keboard with a working Y ke
[16:40:04] <wikibugs>	 (03PS1) 10Btullis: Update the targets for thin deployments [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/838856 (https://phabricator.wikimedia.org/T309346)
[16:40:58] <wikibugs>	 (03CR) 10Joal: [C: 03+1] "Thanks for catching this Ben" [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/838856 (https://phabricator.wikimedia.org/T309346) (owner: 10Btullis)
[16:42:21] <wikibugs>	 (03CR) 10Btullis: [V: 03+2 C: 03+2] Update the targets for thin deployments [analytics/refinery/scap] - 10https://gerrit.wikimedia.org/r/838856 (https://phabricator.wikimedia.org/T309346) (owner: 10Btullis)
[16:42:58] <btullis>	 mforns: That change is merged, so the thin refinery deploy should work now.
[16:43:12] <mforns>	 thanks btullis, will do now!
[16:48:32] <btullis>	 !log forcibly and lazily unmounted legacy labstore hosts from an-launcher1002 and removed their /etc/fstab entries
[16:48:33] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[16:52:00] <icinga-wm>	 RECOVERY - Check unit status of drop-features-actor-rollup-hourly on an-launcher1002 is OK: OK: Status of the systemd unit drop-features-actor-rollup-hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[17:21:26] <mforns>	 btullis: At first deployment to "thin" nodes failed, because I forgot to git pull on analytics/refinery/scap, but then after pulling, it's failing with this error: Permission denied: '/srv/deployment/analytics/refinery-cache'
[17:30:01] <icinga-wm>	 RECOVERY - Host an-master1001.mgmt is UP: PING OK - Packet loss = 0%, RTA = 56.04 ms
[17:30:07] <icinga-wm>	 RECOVERY - Host an-worker1082.mgmt is UP: PING OK - Packet loss = 0%, RTA = 1.40 ms
[17:30:07] <icinga-wm>	 RECOVERY - Host an-worker1081.mgmt is UP: PING OK - Packet loss = 0%, RTA = 1.93 ms
[17:30:07] <icinga-wm>	 RECOVERY - Host an-worker1103.mgmt is UP: PING OK - Packet loss = 0%, RTA = 4.25 ms
[17:30:08] <icinga-wm>	 RECOVERY - Host an-worker1122.mgmt is UP: PING OK - Packet loss = 0%, RTA = 3.49 ms
[17:30:09] <icinga-wm>	 RECOVERY - Host an-worker1123.mgmt is UP: PING OK - Packet loss = 0%, RTA = 5.32 ms
[17:30:25] <icinga-wm>	 RECOVERY - Host aqs1007.mgmt is UP: PING OK - Packet loss = 0%, RTA = 1.53 ms
[17:33:01] <icinga-wm>	 RECOVERY - Host an-worker1139.mgmt is UP: PING OK - Packet loss = 0%, RTA = 6.60 ms
[17:34:55] <icinga-wm>	 RECOVERY - Host druid1004.mgmt is UP: PING OK - Packet loss = 0%, RTA = 1.35 ms
[17:35:44] <btullis>	 mforns: Thanks. Can it wait until the morning UK time?
[17:36:06] <mforns>	 btullis: yes, sure, there's no changes that I'm aware that need to be there so fa
[17:36:08] <mforns>	 far
[17:52:13] <icinga-wm>	 RECOVERY - Host an-tool1010.mgmt is UP: PING OK - Packet loss = 0%, RTA = 4.62 ms
[18:13:33] <wikibugs>	 10Data-Engineering, 10Equity-Landscape: Editorship Output Rank Metrics - https://phabricator.wikimedia.org/T306618 (10JAnstee_WMF) @KCVelaga - I completed a second pass following th re-run and found we are still not aligning on outputs:  * Editor presence and Editor growth outputs seem to be scaled 0 to 1 rath...
[18:50:13] <wikibugs>	 10Data-Engineering, 10Equity-Landscape: Readership Output Rank Metrics - https://phabricator.wikimedia.org/T306617 (10JAnstee_WMF) @KCVelaga Completed second pass, we are still not fully aligned. We have a bit rounding error to contend with but most discrepancies seem minor and could relate somewhat to the dif...
[19:20:20] <wikibugs>	 10Data-Engineering-Operations, 10Data Engineering Planning, 10Mail, 10SRE: Add xcollazo@wikimedia.org to the analytics-alerts mailing list - https://phabricator.wikimedia.org/T315486 (10Dzahn) team membership confirmed per https://www.mediawiki.org/wiki/Platform_Engineering_Team/Data_Value_Stream  ---  @xc...
[19:25:08] <wikibugs>	 10Data-Engineering-Operations, 10Data Engineering Planning, 10Mail, 10SRE: Add xcollazo@wikimedia.org to the analytics-alerts mailing list - https://phabricator.wikimedia.org/T315486 (10xcollazo) Thank you @Dzahn!  ( Side note: I have confirmed that we can make the list public if we choose to move it to Go...
[19:28:20] <wikibugs>	 10Data-Engineering-Operations, 10Data Engineering Planning, 10Mail, 10SRE: Add xcollazo@wikimedia.org to the analytics-alerts mailing list - https://phabricator.wikimedia.org/T315486 (10Dzahn) @xcollazo ITS can create the group and then give admin ship to your team so that you can self-manage it.
[19:36:00] <wikibugs>	 10Data-Engineering-Operations, 10Data Engineering Planning, 10Mail, 10SRE: Add xcollazo@wikimedia.org to the analytics-alerts mailing list - https://phabricator.wikimedia.org/T315486 (10xcollazo) @Dzahn: we discussed moving the list today and there was concern on whether we could make the content of the li...
[19:39:46] <wikibugs>	 10Data-Engineering-Operations, 10Data Engineering Planning, 10Mail, 10SRE: Add xcollazo@wikimedia.org to the analytics-alerts mailing list - https://phabricator.wikimedia.org/T315486 (10Dzahn) @xcollazo There are 2 possible routes you can go. Both result in your team being able to self-manage the list.  a)...
[19:43:30] <wikibugs>	 10Data-Engineering-Operations, 10Data Engineering Planning, 10Mail, 10SRE: Add xcollazo@wikimedia.org to the analytics-alerts mailing list - https://phabricator.wikimedia.org/T315486 (10xcollazo) Ack @Dzahn, thank you for the context and options! Will discuss with team and get back to you.
[21:21:53] <wikibugs>	 (03PS2) 10Milimetric: [WIP] Collaborate on a new editors dataset [analytics/refinery] - 10https://gerrit.wikimedia.org/r/838256
[23:42:02] <wikibugs>	 10Data-Engineering, 10Equity-Landscape: Population input metrics - https://phabricator.wikimedia.org/T309279 (10JAnstee_WMF) We have decided to supplement World Bank Population Gaps first with UN, then with Penn World Table, and then with IMF sources where there are missing nations across agencies.   I am cont...