[11:26:04] lunch [13:19:28] o/ [14:08:56] dcausse I deployed staging rdf-streaming-updater Friday...would you be OK w/deploying the prod envs today? [14:09:22] I think I can do it myself but happy to pair if you'd like [14:28:48] inflatador: oops sorry forgot to answer T349147#9269373 [14:28:48] T349147: Follow up on rdf-streaming-updater failure 2023-10-17 - https://phabricator.wikimedia.org/T349147 [14:30:19] inflatador: note that you have a "redeploy" command that should do the "stop" and "deploy" command at once [14:30:27] * dcausse checking staging [14:38:14] staging looks good, feel free to deploy eqiad & codfw when you want [14:43:04] dcausse excellent, thanks! Will start deploying prod shortly [14:44:00] and thanks for the redeploy flag too! [14:46:17] \o [14:47:17] o/ [14:49:16] workout/errands, back in ~40 [16:08:37] dcausse: i anticipate something for code review in next 48-72 hours for the wdqs scholarly article split into HDFS [16:09:11] dr0ptp4kt: nice! [16:10:33] oh, that reminds me, i'll set a little time with you for next() instance of our wdqs talking [16:12:19] sure! [16:13:15] relatedly if you think we'll discuss a lot about spark perhaps we could have Erik if he's around [16:37:48] dinner [16:41:06] back [17:16:18] meh...producer is encoding timestamps as '{'nanoseconds': 0, 'epochSecond': 123456789}, which elastic doesn't understand. Poking around, but looks extra fun because the serializers are all magic'd up from the schemas [17:22:45] lunch, back in ~40 [18:05:10] back [18:23:56] * ebernhardson spends an hour figuring out how to get the schemas and everything into place to decode raw json string into the raw fields to then feed through the elastic emitter and...it formats the output string as i would expect and not the broken way [18:25:38] LOL...I guess that's an overall good? [18:27:29] well, it means I can't manage to reproduce the production error which is kinda annoying. It means things should work, but something somewhere is wrong and I avoided it when writing my test [18:30:34] Ah, I was thinking more like, "hooray, I don't have to write my own parser to deal with this" [18:39:38] this is all quite mysterious, i wonder if somehow prod is seeing a different schema [18:52:42] doesn't look like it...tried using an event stream factory that uses prod endpoints for schemas and event stream config, but output is same :( [18:55:11] Back in ~90m [19:39:57] * ebernhardson fleshes out more of the test and passes a full cirrusdoc response as plain json through the endpoint to simulate a fetch...still no reproduction :P [19:53:57] hmm, somehow the consumer IT manages to reproduce the error at least. getting somewhere :) [19:54:26] i suppose since we don't load a mapping into elastic it's not catching the same problem [20:22:58] back [20:37:51] ebernhardson: I had some questions about some of the search slo metrics, are you around at 2pm when I pair with inflatador? (at lunch now) [20:51:41] ryankemper: sure [20:54:00] cool, added you to today's invite [20:57:14] * ebernhardson still can't explain why this things emits the wrong updates :P I can clearly see it following a completely different code path...but still can't decide why [20:57:25] something about schema loading [21:00:02] 3’ late [21:48:03] * ebernhardson sees UpdateEventTypeInfo.create, which returns a PojoTypeInfo instead of the event schemas one [21:49:25] * ebernhardson has no clue if related :P [21:55:04] yes thats the differnece. If we serialize the fields with type info from UpdateEventTypeInfo.create() we get bad serialization, when we source it from the deserialization schema it works [22:19:28] * ebernhardson is less sure, back to not being able to replicate :P Should have made a temporary commit [22:24:10] * ebernhardson is a little sad ideavim doesn't implement :earlier [22:25:00] and the undo buffer is extremely short [22:49:04] huh, this is unrelated but the wmf flink eventutilities is comparing types with `if (simpleTypeInfo == Types.STRING)`, and that returns false. but Types.STRING.equals(simpleTypeInfo) returns true (and returns false for other arbitrary types) [22:49:19] but i wonder if that is also related, because the instant conversion is the same comparing with == Types.INSTANT [22:58:29] i dunno...class loader issues? I'm a bit lost...both objects are of type BasicTypeInfo and have a clazz of java.lang.String. But if i put a conditional breakpoint in that constructor only 1 is ever constructed...could it have a second version of the same class somehow? [22:59:15] at least in general know what's going on .... but no solution :P