[04:37:37] <wikibugs>	 10Analytics-Radar, 10Event-Platform, 10WMF-JobQueue, 10Wikibase change dispatching scripts to jobs, and 2 others: Queuing jobs is extremely slow - https://phabricator.wikimedia.org/T292048 (10Ladsgroup) Yup. The full results are in:  - 23% of the jobrunners: https://performance.wikimedia.org/arclamp/svgs/d...
[07:34:45] <wikibugs>	 10Analytics-Data-Quality, 10WMDE-TechWish, 10WMDE-Templates-FocusArea, 10WMDE-TechWish-Sprint-2021-09-29: Check whether VE template dialog and Template Wizard metrics are healthy - https://phabricator.wikimedia.org/T292045 (10awight) a:05awight→03None
[07:51:38] <wikibugs>	 10Analytics-Radar, 10Event-Platform, 10WMF-JobQueue, 10Wikibase change dispatching scripts to jobs, and 2 others: Queuing jobs is extremely slow - https://phabricator.wikimedia.org/T292048 (10Joe) >>! In T292048#7407318, @Ladsgroup wrote: > Yup. The full results are in: >  - 23% of the jobrunners: https://...
[08:54:51] <wikibugs>	 (03PS1) 10Joal: Add BaseDataPublisher copy [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727156
[08:55:06] <joal>	 gehel: --^
[08:56:04] <wikibugs>	 (03CR) 10Gehel: [V: 03+2 C: 03+2] Add BaseDataPublisher copy [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727156 (owner: 10Joal)
[08:56:47] <wikibugs>	 (03PS1) 10Joal: Update dependencies for hadoop and kafka [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727163
[09:24:06] <wikibugs>	 10Analytics: Conda's CPPFLAGS may not be correct when pip installing a package that needs c/cpp compilation - https://phabricator.wikimedia.org/T292699 (10elukey)
[09:27:23] <wikibugs>	 10Analytics: Conda's CPPFLAGS may not be correct when pip installing a package that needs c/cpp compilation - https://phabricator.wikimedia.org/T292699 (10elukey)
[09:42:50] <wikibugs>	 (03Abandoned) 10Joal: Update dependencies for hadoop and kafka [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727163 (owner: 10Joal)
[11:11:12] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Test Alluxio as cache layer for Presto - https://phabricator.wikimedia.org/T266641 (10BTullis) Instead of Alluxio as acaching layer, we might like to look at the caching features of the hive connector that...
[11:20:11] <wikibugs>	 (03PS1) 10Gehel: Removed Hadoop dependencies. [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727277
[11:38:35] <wikibugs>	 (03CR) 10Joal: [V: 03+2 C: 03+2] "LGTM! Thanks Guillaume" [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727277 (owner: 10Gehel)
[11:38:42] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Test Alluxio as cache layer for Presto - https://phabricator.wikimedia.org/T266641 (10BTullis) Unfortunately, that's a no on all three counts. https://trino.io/docs/current/connector/hive-caching.html#limi...
[11:39:08] <gehel>	 joal: ^^ note that the provided dependencies still end up in the fat jar
[11:39:23] <joal>	 I've seen the message gehel 
[11:39:29] <gehel>	 cool!
[11:39:41] <joal>	 gehel: shall we try shaded?
[11:39:56] <gehel>	 by default, shaded seems to do the same
[11:40:40] <joal>	 hm
[11:42:23] <joal>	 Will try the jar
[11:45:28] <gehel>	 at least it should have the right version of hadoop
[12:19:05] <wikibugs>	 10Analytics: Conda's CPPFLAGS may not be correct when pip installing a package that needs c/cpp compilation - https://phabricator.wikimedia.org/T292699 (10Ottomata) Your suggestion makes sense to me.  `CPPFLAGS` gets set when you activate your stacked environment, is that right?  If so, we could certainly do som...
[12:32:16] <wikibugs>	 10Analytics: Conda's CPPFLAGS may not be correct when pip installing a package that needs c/cpp compilation - https://phabricator.wikimedia.org/T292699 (10elukey) Yes exactly afaics the `CPPFLAGS` are set when I activate my stacked conda env. We could try to add:  ` export CPPFLAGS="${CPPFLAGS} -isystem /home/$(...
[12:47:53] <wikibugs>	 (03PS1) 10Gehel: test commit [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727329
[12:49:26] <wikibugs>	 (03CR) 10Hashar: "recheck" [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727329 (owner: 10Gehel)
[12:53:30] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review: Test Alluxio as cache layer for Presto - https://phabricator.wikimedia.org/T266641 (10BTullis) Do we want to revisit the idea to {T256108}?  @JAllemandou spoke of an alternative solution, which was to crea...
[12:58:40] <wikibugs>	 (03Abandoned) 10Gehel: test commit [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727329 (owner: 10Gehel)
[13:02:33] <wikibugs>	 10Analytics: Conda's CPPFLAGS may not be correct when pip installing a package that needs c/cpp compilation - https://phabricator.wikimedia.org/T292699 (10Ottomata) I think the active env path will be available as the `CONDA_PREFIX` env var.
[13:04:28] <wikibugs>	 (03PS1) 10Joal: Add TimePartitionedDataPublisher copy [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727336
[13:14:07] <wikibugs>	 (03PS1) 10Gehel: Re-activate package cycle checks. [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727339
[13:14:21] <gehel>	 joal: ^
[13:15:14] <wikibugs>	 10Analytics: Conda's CPPFLAGS may not be correct when pip installing a package that needs c/cpp compilation - https://phabricator.wikimedia.org/T292699 (10elukey) >>! In T292699#7408475, @Ottomata wrote: > I think the active env path will be available as the `CONDA_PREFIX` env var.   Yep way better!
[13:15:48] <wikibugs>	 (03CR) 10Joal: [V: 03+2 C: 03+2] "LGTM! Merging" [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727339 (owner: 10Gehel)
[13:16:59] <gehel>	 joal: now that we have CI in place, you **should** wait for jenkins to complete :)
[13:17:21] <joal>	 woops - I didn't know CI was already in place - May bad gehel 
[13:32:23] <gehel>	 joal: event-utilities is messing up with our tests: https://gerrit.wikimedia.org/r/c/wikimedia-event-utilities/+/727344
[13:32:46] <gehel>	 joal: do you know if we can just release a new version of event-utilities?
[13:34:33] <joal>	 We could gehel - It's not complicated :)
[13:34:57] <gehel>	 it shoudl be just merging that CR and running the release job?
[13:39:01] <gehel>	 https://integration.wikimedia.org/ci/job/wikimedia-event-utilities-maven-release-docker/
[13:39:44] <wikibugs>	 (03PS1) 10Ladsgroup: Add getting number of rows of wb_changes [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/727346
[13:44:25] <wikibugs>	 (03CR) 10Michael Große: [C: 03+2] Add getting number of rows of wb_changes [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/727346 (owner: 10Ladsgroup)
[13:44:34] <wikibugs>	 (03PS1) 10Gehel: Remove junit from the test classpath. [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727347
[13:45:13] <wikibugs>	 (03PS1) 10Ladsgroup: Add getting number of rows of wb_changes [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/727197
[13:45:22] <wikibugs>	 (03CR) 10Ladsgroup: [C: 03+2] Add getting number of rows of wb_changes [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/727197 (owner: 10Ladsgroup)
[13:45:43] <wikibugs>	 (03Merged) 10jenkins-bot: Add getting number of rows of wb_changes [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/727346 (owner: 10Ladsgroup)
[13:46:33] <wikibugs>	 (03Merged) 10jenkins-bot: Add getting number of rows of wb_changes [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/727197 (owner: 10Ladsgroup)
[13:50:18] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Remove junit from the test classpath. [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727347 (owner: 10Gehel)
[13:58:22] <wikibugs>	 (03PS2) 10Gehel: Remove junit from the test classpath. [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727347
[14:04:06] <joal>	 gehel: sorry I got pulled in another meeting, and now kids - will be back in 2h :S
[14:04:37] <gehel>	 joal: take your time!
[14:13:19] <ottomata>	 elukey:  joal or btullis, i'm trying to use flink with hive, and am encountering a kerberos barrier!  (flink with yarn ok so far).  got a sec for a brain bounce?  maybe one of you can help me figure out the right incantation
[14:15:06] <elukey>	 ottomata: I am in the daily sync, should be free in 10/15 mins, would it be ok?
[14:15:36] <elukey>	 do you mean flink on k8s?
[14:15:45] <ottomata>	 no, just flink local or in yarn
[14:15:50] <elukey>	 ahh okok
[14:15:58] <ottomata>	 ya 15 is good, i'll join daily sync too
[14:20:22] <ottomata>	 elukey:  am here if you like http://meet.google.com/mkb-aigp-xpc
[14:30:56] <wikibugs>	 10Analytics-Radar, 10Fundraising-Backlog, 10Product-Analytics, 10Wikipedia-iOS-App-Backlog, and 2 others: Understand impact of Apple's Relay Service - https://phabricator.wikimedia.org/T289795 (10sgrabarczuk) We (the Product department, Wikimedia Foundation) are working on an announcement. Within a week, w...
[14:32:27] <gehel>	 ottomata: would you have a minute to review and merge: https://gerrit.wikimedia.org/r/c/wikimedia-event-utilities/+/727344
[15:04:26] <wikibugs>	 10Analytics-Radar, 10Event-Platform, 10WMF-JobQueue, 10Wikibase change dispatching scripts to jobs, and 2 others: Queuing jobs is extremely slow - https://phabricator.wikimedia.org/T292048 (10Ladsgroup) Increasing the replicas also reduced the save time p75: https://grafana.wikimedia.org/d/000000085/save-t...
[15:37:54] <ottomata>	 gehel:  done thank you
[15:57:27] <joal>	 Here!
[15:59:13] <joal>	 ottomata: have ou figured out your thing with Flink?
[16:03:06] <wikibugs>	 (03CR) 10Joal: [C: 03+2] "LGTM! Thanks for having figured out the missing dep Guillaume :)" [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727347 (owner: 10Gehel)
[16:05:09] <wikibugs>	 (03CR) 10Joal: [C: 03+2] "Self-Merging code change to match previous version" [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727336 (owner: 10Joal)
[16:09:01] <ottomata>	 joal:  elukey helped and we think flink and hivecatalog might not work with kerberos
[16:09:09] <ottomata>	 i'm going to email the flink mailing list
[16:09:21] <joal>	 meh - we're facing a similar issue with alluxio :S
[16:10:05] <btullis>	 Sorry I missed the brainbounce opportunity. Would be good to catch up on what you found.
[16:10:54] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Remove junit from the test classpath. [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727347 (owner: 10Gehel)
[16:12:53] <wikibugs>	 (03CR) 10Joal: [C: 03+2] "recheck" [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727347 (owner: 10Gehel)
[16:14:16] <ottomata>	 the 'kerberos barrier' is everywhere
[16:14:20] <gehel>	 ottomata / joal : any objections to me releasing event-utilities ?
[16:14:28] <ottomata>	 gehel: not at all please do
[16:14:32] <joal>	 Thanks gehel :)
[16:18:12] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add TimePartitionedDataPublisher copy [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727336 (owner: 10Joal)
[16:18:51] <joal>	 hm - gehel, may I safely assume that the errors I experience at build for gobblin come from the eventutilities not yet released?
[16:20:11] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Remove junit from the test classpath. [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727347 (owner: 10Gehel)
[16:20:19] <joal>	 ottomata: would you have a minute to pair with me on an ops-week ticket that needs ops-power?
[16:20:22] <joal>	 please
[16:21:08] <btullis>	 joal: I'm happy to help if you like.
[16:21:14] <joal>	 Hey btullis :)
[16:21:16] <joal>	 thank you!
[16:21:20] <joal>	 btullis: batcave?
[16:21:25] <btullis>	 See you there.
[16:26:42] <wikibugs>	 (03Merged) 10jenkins-bot: Remove junit from the test classpath. [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727347 (owner: 10Gehel)
[16:28:46] <wikibugs>	 10Analytics, 10Analytics-Kanban: Request Kerberos credentials - https://phabricator.wikimedia.org/T292532 (10BTullis) Created the principal as per [[https://wikitech.wikimedia.org/wiki/Analytics/Systems/Kerberos#Create_a_principal_for_a_real_user|the procedure here]]. ` btullis@krb1001:~$ sudo manage_principal...
[16:56:39] <wikibugs>	 (03CR) 10Joal: [C: 03+2] "recheck" [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727336 (owner: 10Joal)
[16:59:02] <wikibugs>	 (03Merged) 10jenkins-bot: Add TimePartitionedDataPublisher copy [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727336 (owner: 10Joal)
[16:59:47] <ottomata>	 joal:  just saw!  am here lemme know if you still need help
[17:00:16] <joal>	 heya ottomata - got help from btullis thank you :)
[17:01:39] <btullis>	 ottomata: Any chance you could spend a few minutes explaining the flink progress/issue to me? Strugglling to follow via Slack.
[17:13:36] <btullis>	 I see. The list of Flink connectors and services that support Kerberos: https://ci.apache.org/projects/flink/flink-docs-release-1.13/docs/deployment/security/security-kerberos/#how-flink-security-works
[17:13:56] <btullis>	 Specifically does not mention the Hive connector.
[17:14:35] <ottomata>	 btullis: sure, bat cave?
[17:14:45] <btullis>	 Yep.
[17:15:02] <elukey>	 joal, gehel - after a brainstorm with Miriam and Erik we were able to run the empty tensorflow package hack with magenta!
[17:15:15] <joal>	 \o/
[17:15:21] <joal>	 you folks rock :)
[17:15:39] <elukey>	 I'll have to test more but this is a major game changer, thanks a lot for the idea!!
[17:15:43] <gehel>	 Cool!
[17:16:03] <miriam>	 yes joal and gehel thanks a looot
[17:17:51] <elukey>	 I'll try to add some docs on wikitech tomorrow
[17:23:45] <wikibugs>	 10Analytics, 10Event-Platform, 10Observability-Logging, 10SRE, and 2 others: Integrate Event Platform and ECS logs - https://phabricator.wikimedia.org/T291645 (10herron)
[18:27:04] <wmf-insecte>	 Starting build #11 for job wikimedia-event-utilities-maven-release-docker
[18:28:39] <wmf-insecte>	 Project wikimedia-event-utilities-maven-release-docker build #11: 09SUCCESS in 1 min 35 sec: https://integration.wikimedia.org/ci/job/wikimedia-event-utilities-maven-release-docker/11/
[18:53:47] <wikibugs>	 (03PS1) 10Gehel: Upgraded to wikimedia-eventutilities 1.0.9 [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727463
[19:10:24] <wikibugs>	 (03PS1) 10Gehel: Removed properties already defined in parent pom.xml [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727469
[19:59:55] <wikibugs>	 (03CR) 10ODimitrijevic: [C: 03+1] "lgtm" [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727469 (owner: 10Gehel)
[20:00:39] <wikibugs>	 (03CR) 10ODimitrijevic: [C: 03+1] "lgtm" [analytics/gobblin-wmf] - 10https://gerrit.wikimedia.org/r/727463 (owner: 10Gehel)
[22:14:36] <wikibugs>	 10Analytics-Radar, 10Product-Analytics, 10Wikipedia-Android-App-Backlog (Android Release FY2021-22): What percentage of app editors are IP editors? - https://phabricator.wikimedia.org/T291866 (10SNowick_WMF) Using data for the previous quarter in `druid.edits_hourly`the breakdown of combined iOS and Android...