[18:43:05] James_F: if you have a moment https://gerrit.wikimedia.org/r/c/mediawiki/extensions/timeline/+/724812/ [18:46:59] (or anyone else) :) [18:52:46] legoktm: +2'd [18:53:30] ty :) [19:21:00] Pchelolo: OMG :D [19:21:15] Amir1: yup. [19:21:31] can it be because we were migrating them from the old system to eventgate? [19:21:37] doesn't matter [19:21:41] let's fix it [19:24:18] I'll make a patch. there's some interesting complications there - we can't just send ALL jobs in a single batch, cause they might be bigger then POST size limit :) [19:25:04] but splitting them by type and sending types one-by-one is certainly no bueno [19:25:31] hmm, noted [19:26:12] I go eat dinner, might try to take a stab at it [19:47:57] made a patch. added you as reviewer. the patch should really help a lot. [19:49:04] plus, I don't quite understand the purpose of JobQueueGroup::push. What could possibly be the reason to not defer the job pushing? [19:50:07] back [19:50:16] Pchelolo: probably some jobs that have high priority [19:51:34] but it doesn't matter, by not deferring them you're saving negligeable amount of time in the whole job lifecycle. [19:51:50] and you're not making it any more reliable [19:52:07] I assume it's for third parties [19:52:10] the ordering should also not matter, cause order of execution is not guaranteed [19:52:15] but I'm just guessing [19:53:17] Pchelolo: I'm going to test this in mwdebug1001, and if it looks good, I deploy it [19:53:28] codewise it looks good [19:54:03] I haven't tested it locally yet [19:54:26] but I guess if you test it on real thing, I don't need to test locally [19:54:48] yeah, I test it with verbose and xhgui [19:54:55] that's going to give out a lot of info [19:55:43] ok. ping me when you have something, I'll go eat in the meantime [19:56:00] bon appetit! [20:06:55] Pchelolo: it doesn't fix the issue: https://performance.wikimedia.org/xhgui/run/symbol?id=6154c6056326f0021890b8de&symbol=MediaWiki%5CExtension%5CEventBus%5CAdapters%5CJobQueue%5CJobQueueEventBus%3A%3AdoBatchPush [20:07:12] still makes 7 requests [20:08:19] Amir1: 7 batch pushes is due to use of JQ::push not lazyPush probably [20:08:50] it makes as many sends and it makes batch pushes - that's what I was going for [20:09:30] so together with replacing push with lazyPush it will work [20:10:13] I see [20:12:00] Pchelolo: let's push the lazypush ones first to make sure it works properly. Would that work for you? [20:12:12] yeah, definitely better idea. [20:12:25] I mean yes. definitely. good idea. [20:19:20] hmm, there is also five other direct pushes https://performance.wikimedia.org/xhgui/run/symbol?id=6153427113e6381f84c80315&symbol=MediaWiki%5CExtension%5CEventBus%5CEventBus%3A%3Asend [20:19:43] these come directly to eventbus::send [20:19:53] but they might be important [20:20:25] Pchelolo: do you think we could batch these too ^ [20:20:34] looking [20:24:38] hm... so all 5 parents there are different use-cases of EventBus. I think I can make each use-case batch everything together. [20:25:49] batching between use-cases could be possible, but needs a bit more thinking on how to structure it. Also these 5 use-cases have 2 different destinations - main eventgate and analytics one [20:26:06] obviously we can't batch between these 2. [20:27:22] interesting, the logs all send to :6005 though [20:27:40] hm. maybe I'm mistaken.. lemme check [20:28:27] https://logstash.wikimedia.org/goto/a288ef574056abcd7bea7ad72eb2c3a4 [20:30:14] Amir1: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Echo/+/724835/ [20:30:43] zabe: how does it even work? [20:30:52] has been caused by https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/724077 [20:31:32] col [20:31:34] *cool [20:31:44] I'm merge it once jenkins is green [20:33:23] hm.. https://performance.wikimedia.org/xhgui/run/symbol?id=6153427113e6381f84c80315&symbol=MediaWiki%5CExtension%5CEventBus%5CAdapters%5CMonolog%5CEventBusMonologHandler%3A%3AMediaWiki%5CExtension%5CEventBus%5CAdapters%5CMonolog%5C%7Bclosure%7D should be going to 6004 [20:35:05] Pchelolo: you learn something about your infra every day [20:36:05] In the mean time, I think there is an extra push for stupid reasons [20:53:09] Amir1: ok, I know how to batch even more and do it nicely, will need to do a little refactoring beforehand. But figuring out Jobqueue push vs lazyPush is probably the lowest hanging and the largest fruit [20:54:22] Pchelolo: yeah, I think there is another issue there too, I'm trying to debug [20:54:34] not in eventbus, in mw core [20:54:41] with recent refactors [20:58:45] yup [21:03:06] filed a couple tasks for myself. Will implement them tomorrow [21:09:01] Pchelolo: before implementing those, we need to get this in: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/724843/ [21:09:46] without this, any push will just get ignored here :( [21:25:59] my brain is fried. I will look into this more tomorrow [21:28:38] I have one more trivial timeline patch if someone can +2 it, just bumping CACHE_VERSION: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/timeline/+/724853/ [21:29:10] thanks Amir1 :) [21:29:16] ^^ [21:44:36] okay, one more timeline fix :/ https://gerrit.wikimedia.org/r/c/mediawiki/extensions/timeline/+/724855/ [21:49:52] legoktm: {{done}} [21:49:57] ty :))