[02:33:24] <wikibugs>	 10Analytics, 10Analytics-EventLogging, 10Performance-Team, 10Patch-For-Review: EventLogging background queue beforeunload event handler blocks Back-Forward cache - https://phabricator.wikimedia.org/T285220 (10DLynch) @Krinkle I think we're talking in circles here? Anyway, it won't break anything, because a...
[06:52:45] <wikibugs>	 10Analytics, 10WMDE-Templates-FocusArea, 10Patch-For-Review, 10WMDE-TechWish (Sprint-2021-02-03), 10WMDE-TechWish-Sprint-2021-06-09: Add missing normalization to CodeMirror Grafana board - https://phabricator.wikimedia.org/T273748 (10awight) @mforns Thanks for adjusting the start date.  Can I ask if you...
[08:16:39] <wikibugs>	 10Analytics, 10WMDE-Templates-FocusArea, 10Patch-For-Review, 10WMDE-TechWish (Sprint-2021-02-03), and 2 others: Add missing normalization to CodeMirror Grafana board - https://phabricator.wikimedia.org/T273748 (10WMDE-Fisch)
[13:07:07] <wikibugs>	 10Analytics, 10WMDE-Templates-FocusArea, 10Patch-For-Review, 10WMDE-TechWish (Sprint-2021-02-03), and 2 others: Add missing normalization to CodeMirror Grafana board - https://phabricator.wikimedia.org/T273748 (10mforns) @awight, I haven't yet deleted the output files, because the code change, even if merg...
[13:08:05] <mforns>	 hi alll!
[13:08:08] <wikibugs>	 (03CR) 10Ottomata: [WIP] Metrics Platform schema (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/676392 (https://phabricator.wikimedia.org/T276379) (owner: 10Jason Linehan)
[13:09:21] <wikibugs>	 (03CR) 10Ottomata: [C: 03+1] "Cool, will merge this after Petr +1s the mw core change." [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/700253 (https://phabricator.wikimedia.org/T285113) (owner: 10Urbanecm)
[13:20:01] <ottomata>	 morning!
[13:20:24] <ottomata>	 mforns: revieweeyyyeyyeyeye???? https://github.com/wikimedia/jsonschema-tools/pull/31 :)
[13:20:44] <mforns>	 ottomata: yes, doing now, sorry
[13:21:01] <ottomata>	 no worried, wasn't sure if you saw my pings beforeeeee sorry for pestering!
[13:21:12] <ottomata>	 take your time it isn't that high priority
[13:28:57] <wikibugs>	 10Analytics, 10WMDE-Templates-FocusArea, 10Patch-For-Review, 10WMDE-TechWish (Sprint-2021-02-03), and 2 others: Add missing normalization to CodeMirror Grafana board - https://phabricator.wikimedia.org/T273748 (10awight) >>! In T273748#7172339, @mforns wrote: > @awight, I haven't yet deleted the output fil...
[13:33:30] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Fundraising-Backlog, and 2 others: CentralNoticeBannerHistory and CentralNoticeImpression Event Platform Migration - https://phabricator.wikimedia.org/T271168 (10Ottomata)
[13:42:14] <ottomata>	 mforns:  FYI am merging https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikimediaEvents/+/701101
[13:42:28] <ottomata>	 we missed this weeks train cut...i could backport
[13:42:43] <ottomata>	 hmm maybe i will, would lbe annoying to wait another week to finish those
[13:42:45] <mforns>	 ottomata: ok
[13:43:11] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Patch-For-Review: WMDEBanner* Event Platform Migration - https://phabricator.wikimedia.org/T282562 (10Ottomata)
[13:54:13] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Platform Engineering, 10Research, 10User-razzi: Create airflow instances for Platform Engineering and Research - https://phabricator.wikimedia.org/T284225 (10Ottomata) Oh right, there is more to do than just https://wikitech.wikimedia.org/wiki/Analytics/Systems/Airflow#...
[13:54:46] <wikibugs>	 (03CR) 10Mholloway: [WIP] Metrics Platform schema (031 comment) [schemas/event/secondary] - 10https://gerrit.wikimedia.org/r/676392 (https://phabricator.wikimedia.org/T276379) (owner: 10Jason Linehan)
[14:13:13] <milimetric>	 mforns: got a sec to brainbounce this encoding issue?
[14:13:28] <mforns>	 milimetric: sure!
[14:13:42] <mforns>	 milimetric: could it be in 10 mins?
[14:13:47] <mforns>	 I'm finishing a CR
[14:13:55] <milimetric>	 mforns: ofc, anytime you're ready over the next hour
[14:14:01] <mforns>	 k
[14:23:10] <wikibugs>	 (03CR) 10Nikerabbit: "FYI you got the task number wrong ;)" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/683485 (https://phabricator.wikimedia.org/T281688) (owner: 10Joal)
[14:26:17] <wikibugs>	 (03CR) 10Ppchelko: [C: 03+2] mediawiki/api/request: Add performer field and bump to 1.0.0 [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/700253 (https://phabricator.wikimedia.org/T285113) (owner: 10Urbanecm)
[14:26:54] <wikibugs>	 (03Merged) 10jenkins-bot: mediawiki/api/request: Add performer field and bump to 1.0.0 [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/700253 (https://phabricator.wikimedia.org/T285113) (owner: 10Urbanecm)
[14:27:08] <mforns>	 hey milimetric should we pair?
[14:28:18] <wikibugs>	 (03CR) 10Ppchelko: "> Patch Set 4:" [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/700253 (https://phabricator.wikimedia.org/T285113) (owner: 10Urbanecm)
[14:29:45] <wikibugs>	 (03CR) 10Ottomata: "> Patch Set 4:" [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/700253 (https://phabricator.wikimedia.org/T285113) (owner: 10Urbanecm)
[14:30:30] <milimetric>	 mforns: omw cave
[14:30:40] <mforns>	 k
[14:38:01] <wikibugs>	 (03CR) 10Urbanecm: "Thanks both 😊" [schemas/event/primary] - 10https://gerrit.wikimedia.org/r/700253 (https://phabricator.wikimedia.org/T285113) (owner: 10Urbanecm)
[14:42:08] <ottomata>	 elukey:  sorry i merged too soon!
[14:42:21] <ottomata>	 didn't realize that class was going to go away once we could  declare in admin module
[14:43:02] <elukey>	 ottomata: nono the change looks really good, I didn't have anything agaist it, added a note to get your feedback :)
[14:44:15] <ottomata>	 we only have 1 node left to get to all buster, right?
[14:44:16] <ottomata>	 an-master1001?
[14:44:23] <ottomata>	 and razzi is going to do that soon...maybe this week?
[14:44:53] <ottomata>	 elukey:  while i have your attention... i have a user naming problem
[14:44:54] <ottomata>	 :)
[14:44:57] <ottomata>	 https://phabricator.wikimedia.org/T284225
[14:45:07] <ottomata>	 so in admin, platform-engineering is already a user group
[14:45:23] <ottomata>	 to match conventions, it probably should have been called platform-engineering-users
[14:45:27] <ottomata>	 but it wasn't :(
[14:45:36] <ottomata>	 also, i'm not sure if we want to make these new groups prefixed with 'analytics-' or not.
[14:45:56] <ottomata>	 it is possible they could/will be used for other things
[14:46:18] <ottomata>	 the users in platform-engineering seem like the like right folks for their airflow instance
[14:46:25] <ottomata>	 but i dunno what to call the system user
[14:47:27] <ottomata>	 i could just call it 'platform-engineering', but then there wouldn't be a distinct posix group just for the system user
[14:47:33] <elukey>	 I would use analytics- to keep the nomenclature uniform, just to avoid to have to remember exceptions
[14:47:58] <elukey>	 lemme check admin
[14:48:02] <ottomata>	 yeah, but what happens when they start using airflow to schedule things outside of 'analyttcs' ?
[14:48:25] <ottomata>	 its just a name so it doesn't really matter...but it will be confusing
[14:48:47] <elukey>	 ah ok this is an interesting use case
[14:49:00] <elukey>	 should they be able to schedule things outside analytics' realm?
[14:49:10] <ottomata>	 probably eventually, yeah
[14:49:31] <ottomata>	 we could always revisit later, but it'd be nice to avoid that work if we can get a good name/convention now
[14:50:01] <ottomata>	 e.g. creating new keytabs, etc.
[14:50:26] <elukey>	 I completely get your point, I just feel that expanding airflow to non analytics use cases seems to be something that shouldn't fall under our radar
[14:50:32] <elukey>	 not to say that we don't want to support
[14:51:07] <elukey>	 airflow without kerberos could run in kubernetes in theory
[14:51:19] <elukey>	 but then they'd have two things etc..
[14:51:47] <elukey>	 keeping the scope of their keytab to analytics-only things feels better in my head
[14:52:02] <elukey>	 but I am always the paranoid one so my opinion is biased :D
[14:53:05] <razzi>	 Hi all, good morning
[14:53:12] <razzi>	 an-master1002 is doing great!
[14:53:36] * razzi waves to an-master1002
[14:53:52] * razzi pats an-master1002 on the lid
[14:54:53] <ottomata>	 haha
[14:55:23] <ottomata>	 elukey:  yah maybe we wouldn't want it to be under our radar...maybe
[14:55:31] <ottomata>	 but, it will probably happen
[14:55:45] <ottomata>	 and the group naming is kinda separate from what we'd necessarily manage
[14:55:49] <ottomata>	 although....
[14:55:59] <ottomata>	 i guess this system user is specifically for analytics cluster...
[14:56:16] <ottomata>	 just annoying that we have to have that be the same user that owns all the airflow instance stuff
[14:57:28] <elukey>	 but if we create analytics-platform-eng-users with the & trick so that people can update only one place?
[14:57:37] <elukey>	 plus the related system user
[14:57:48] <ottomata>	 hm, yeah...
[14:58:03] <elukey>	 it looks a compromise, not great but somethign
[14:58:06] <ottomata>	 well, we don't really need to create that -users group, do we?
[14:58:08] <ottomata>	 i mean we could
[14:58:20] <ottomata>	 so
[14:58:30] <ottomata>	 analytics-platform-eng system user
[14:58:39] <ottomata>	 ?
[14:58:47] <elukey>	 what we care about is that the system user gets the fixed uid/gid
[14:58:50] <elukey>	 plz :)
[14:59:00] <ottomata>	 yeah eaiser to reason about
[14:59:06] <ottomata>	 will consider making the -users group... lemme see
[14:59:12] <ottomata>	 could just use platform-engineering as is
[14:59:14] <ottomata>	 maybe.
[14:59:16] <ottomata>	 hmm
[14:59:31] <elukey>	 exactly yes I think it would be fine, and that group will be able to sudo as analytics-blabla
[14:59:34] <elukey>	 could be an option
[15:00:08] <ottomata>	 we don't have docs on creating a new analytics system user, right?
[15:00:11] <ottomata>	 if not i'll make some
[15:00:23] <ottomata>	 i think there are a lot of steps, including capacity scheduler queues, declaring in all the right hiera files, etc.
[15:00:55] <ottomata>	 haha, i'd make this user just be 'analytics-platform' if that wasn't  soo generic
[15:06:57] <mforns>	 elukey: qq: did we change anything in the way reportupdater-queries gets deployed? before I just merged changes, and they were deployed by puppet, but yesterday I merged some changes and the old code is still in an-launcher1002, under /srv/reportupdater/jobs/reportupdater-queries/codemirror
[15:07:07] <mforns>	 hi! :] BTW
[15:08:05] <elukey>	 ottomata: nope no docs I am afraid, but I can help if needed!
[15:08:23] <ottomata>	 k i'll write up as I do this and you can help me fix mistakes :)
[15:08:26] <elukey>	 mforns: hi! Not that I recall, maybe the timer is stuck??
[15:08:32] <elukey>	 ottomata: +1 :)
[15:09:04] <mforns>	 there's a timer that deploys code? looking at puppet
[15:09:53] <elukey>	 mforns: nono sorry I meant the timer that uses the code, didn't see that you checked the repo on disk, so not that for sure
[15:09:57] <elukey>	 the cr is https://gerrit.wikimedia.org/r/c/analytics/reportupdater-queries/+/700990 right?
[15:10:17] <mforns>	 elukey: yes!
[15:10:55] <elukey>	 mforns: so on launcher's puppet.log I see
[15:10:55] <elukey>	 Jun 22 21:14:13 an-launcher1002 puppet-agent[17322]: (/Stage[main]/Profile::Reportupdater::Jobs/Reportupdater::Job[browser]/Git::Clone[analytics/reportupdater-queries]/Exec[git_pull_analytics/reportupdater-queries]/returns) executed successfully (corrective)
[15:11:10] <mforns>	 o.O
[15:14:35] <elukey>	 mforns: if I go in /srv/reportupdater/jobs/reportupdater-queries and do git log I see that the last commit was on May 3rd, but https://gerrit.wikimedia.org/r/q/project:analytics%252Freportupdater-queries looks different
[15:16:33] <elukey>	 mforns: so I am in reportupdater::job
[15:16:34] <elukey>	     # Path at which this reportupdater job repository will be cloned.
[15:16:37] <elukey>	     $path            = "${::reportupdater::job_repositories_path}/${repository}"
[15:16:43] <mforns>	 looking at that as well
[15:16:54] <elukey>	 but the job_repositories_path is not declared in reportupdater::init
[15:17:00] <mforns>	 uou!
[15:17:19] <elukey>	 do you recall that there was a code change for RU that I fixed some weeks ago
[15:17:28] <elukey>	 I am wondering if I missed to fix that bit as well
[15:17:39] <mforns>	 I don't recall
[15:18:36] <elukey>	 https://phabricator.wikimedia.org/T274880#7114360
[15:19:01] <elukey>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/692909/5/modules/reportupdater/manifests/init.pp
[15:19:18] <elukey>	 job_repositories_path was changed into jobs_path
[15:19:18] <mforns>	 oh!!! elukey yes, that was me! ofc for sure I messed that up
[15:19:27] <mforns>	 there you go
[15:21:05] <elukey>	 created https://gerrit.wikimedia.org/r/c/operations/puppet/+/701130, lemme run pcc to see what chances
[15:21:08] <elukey>	 *changes
[15:22:05] <mforns>	 yea, that looks good
[15:22:09] <elukey>	 ah yes the /srv/reportupdater is a little messed up
[15:22:23] <mforns>	 I didn't think to look for all external usages of that variable
[15:23:15] <elukey>	 mforns: https://puppet-compiler.wmflabs.org/compiler1002/29981/an-launcher1002.eqiad.wmnet/index.html
[15:24:18] <elukey>	 elukey@an-launcher1002:~$ ls -ld /reportupdater-queries/
[15:24:19] <elukey>	 drwxr-xr-x 19 analytics root 4096 May 26 06:02 /reportupdater-queries/
[15:24:28] <elukey>	  /o\ /o\ /o\
[15:24:50] <mforns>	 elukey: that makes sense, BYW, there's now a reportupdater-queries folder directly on /
[15:24:59] <elukey>	 yeah see above :D
[15:25:04] <mforns>	 hehe, yes
[15:25:04] <elukey>	 now it makes sense :D
[15:25:10] <elukey>	 going to merge! Green light?
[15:25:19] <mforns>	 yes!
[15:26:55] <ottomata>	 elukey:  there are ~10 hiera files that need  to be edited when adding a new system user and/or user group for analyitcs
[15:27:03] <wikibugs>	 10Analytics-Radar, 10EventStreams, 10MediaWiki-API, 10MW-1.37-notes (1.37.0-wmf.12; 2021-06-28), 10User-Urbanecm: Add user field to mediawiki/api/request - https://phabricator.wikimedia.org/T285113 (10Urbanecm) @ottomata Is there anything else to do before resolving this task? :)
[15:27:15] <ottomata>	 there's gotta be a way to make that better
[15:27:34] <elukey>	 it seems a lot, I recalled less
[15:27:59] <elukey>	 ahh because of how admin deploys stuff
[15:28:00] <elukey>	 yes yes 
[15:28:02] <ottomata>	 master, standby, worker, client for both clusters (prod and test)
[15:28:12] <ottomata>	 stat exploerer
[15:28:24] <ottomata>	 and for this case, airflow
[15:28:38] <elukey>	 mforns: /srv/reportupdater/jobs/reportupdater-queries looks good now!
[15:28:42] <mforns>	 elukey: thanks a lot, code has been updated in the right place!
[15:28:50] <mforns>	 yes, <3
[15:29:48] <elukey>	 mforns: I don't see open file handles for the /reportupdater-queries, I'd just drop it if you are ok
[15:30:18] <mforns>	 elukey: sure, please go ahead
[15:30:35] <wikibugs>	 (03PS5) 10Fdans: Adapt wiki selector to allow more than one wiki [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/700098 (https://phabricator.wikimedia.org/T285050)
[15:30:41] <elukey>	 !log drop /reportupdater-queries on an-launcher1002 after https://gerrit.wikimedia.org/r/c/operations/puppet/+/701130
[15:30:43] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[15:30:54] <elukey>	 done :)
[15:31:11] <elukey>	 now if everything explodes it is not my fault :P
[15:33:08] <wikibugs>	 10Analytics, 10WMDE-Templates-FocusArea, 10Patch-For-Review, 10WMDE-TechWish (Sprint-2021-02-03), and 2 others: Add missing normalization to CodeMirror Grafana board - https://phabricator.wikimedia.org/T273748 (10mforns) @awight, OK @elukey helped me find the problem. The code was not reaching the producti...
[16:29:12] <joal>	 razzi: I have a 1-1 during the talk - will you make a summary for me after?
[16:31:09] <joal>	 Ah razzi - Dan told me it's recorded :)
[16:59:32] <mforns>	 ottomata: I was looking at prefect, and I can still see the server and UI are under the prefect-community-license, no? https://www.prefect.io/legal/prefect-community-license/
[17:39:08] <joal>	 milimetric: Hi :) Do you have time to talk collaction?
[17:39:33] <milimetric>	 omw bc joal, 30 sec
[17:39:43] <joal>	 ack - joining as well milimetric
[17:46:45] <ottomata>	 mforns:  i didn't go deep, it seems so!
[17:46:49] <ottomata>	 so not realy open source after all
[17:46:50] <ottomata>	 :/
[17:47:00] <ottomata>	 oh well, good for us easier to feel great about airflow :)
[17:47:05] <mforns>	 xDDD
[17:48:52] <mforns>	 ottomata, I will try to bring out all (maybe few) reasons that we weighted when choosing airflow, and put them on wiki
[17:49:17] <ottomata>	 mforns:  pfct
[17:56:31] <ottomata>	 !log enable canary events for NavigationTiming extension streams - https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/699789
[17:56:33] <stashbot>	 Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log
[18:01:36] <joal>	 ottomata: helloooOOOOOoooo :) time to talk?
[18:06:59] <ottomata>	 joal:  ya! having a thing deployed atm but i think it should be no brainer
[18:07:00] <ottomata>	 bc?
[18:07:10] <joal>	 sure - joining ottomata 
[18:11:09] <joal>	 milimetric! I forgot - could you revert the pageview change and webrequest ticket
[18:12:34] <joal>	 please --^ :)
[18:14:08] <milimetric>	 joal: I think I already edited the task to investigate the similar issue if it existed, so I'll close that, and abandon the change, yes
[18:14:30] <milimetric>	 oh, I never submitted the change, so we're good
[18:14:40] <joal>	 ack milimetric, please don't forget the pageview wikitech page :)
[18:15:37] <milimetric>	 ah
[18:17:55] <milimetric>	 erased from history.  bonus: now the known issues table actually makes some sense
[18:20:51] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Top edited pages list on enwiktionary contains nonexistent pages with titles made up of question marks - https://phabricator.wikimedia.org/T284623 (10Milimetric)
[18:21:07] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: page titles in the webrequest pipeline - https://phabricator.wikimedia.org/T285324 (10Milimetric) 05Open→03Invalid  the root cause appears to be collation, so does not affect the pageview pipeline.  There are some open discussions around whether we...
[18:36:00] <joal>	 thanks a lot milimetric 
[18:58:49] <wikibugs>	 10Analytics-Radar, 10EventStreams, 10MediaWiki-API, 10MW-1.37-notes (1.37.0-wmf.12; 2021-06-28), 10User-Urbanecm: Add user field to mediawiki/api/request - https://phabricator.wikimedia.org/T285113 (10Ottomata) I don't think so; once your MW core change is out the new events should start coming in.  Mayb...
[20:09:42] <ottomata>	 joal:  https://gerrit.wikimedia.org/r/c/wikimedia-event-utilities/+/701177
[20:09:42] <ottomata>	 :)
[20:36:07] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Replace Camus by Gobblin - https://phabricator.wikimedia.org/T271232 (10Ottomata) @JAllemandou, I wonder if Gobblin will just be better than Camus in general, and we don't need more than one job to import events after all.   The highest volume event stream...