[03:21:36] inflatador: We've got a `wcqs.jnl` of `413G` following the xfer onto `wcqs1003`, so it looks like the xfer worked [09:05:17] gehel, pfischer if you have thoughts on: https://gerrit.wikimedia.org/r/c/wikimedia-event-utilities/+/857779/1/pom.xml#74 [09:44:00] looking [10:00:07] I added a comment. I don't have great ideas :/ [10:00:38] I think the root of the problem is refinery. But fixing that isn't going to be easy. [10:03:46] yes... [10:03:48] Ouch, our version of Hadoop relies on Guava 11.0.2 ! We're already forcing an upgrade to Guava 16 [10:04:27] How much do we want to hold everything back, just because we're using an outdated version of Hadoop? [10:06:22] wdqs spark tools use spark and guava 22 it seems to "work", but this spark project is nowhere near as complex as refinery-source [10:07:24] refeniry-source also lacks some modularity imo [10:07:51] I'm also not sure what level of isolation does Hadoop provides? What happens if we package a recent version of guava in the code that we deploy on hadoop? Are there conflicts with the guava version used inside of hadoop? [10:08:49] I think it all depends on the runtime you use, generally you rarely run a bare java application there, you rely on a runtime like spark [10:09:27] I don't have a good mental model of how all those pieces fit together :( [10:10:17] me neither [10:10:52] I'm very much tempted to push for a major clean up of refinery, rather than holding back everything else. But that's data engineering job, not mine :) [10:11:15] yes... :) [10:11:19] maybe jump in a call with you, me and joal if he is around? [10:11:27] sure [10:11:53] he seems ooo [10:12:24] next week should be good [10:12:42] damn, for next week then. Should we have Andrew as well? [10:12:55] sure [10:13:32] Andrew is off part of next week. Damn, scheduling is hard [10:14:03] I'm surprised that Google does not yet provide an AI driven scheduling assistant as part of Calendar ! [10:15:13] :) [10:59:21] Lunch [12:30:45] there was a surge of Status code 503; upstream connect error or disconnect/reset before headers. reset reason: connection termination [12:30:55] elastic nodes seem healthy so unsure how to debug this [12:31:27] lunch [13:47:23] o/ [16:15:23] \o [16:17:03] will not make gaming unmtg, I'm at the coffee shop [16:19:52] I've just published a weekly update. Thanks all for filling up our etherpad! That helps a ton! https://app.asana.com/0/0/1203400665178060 [17:56:25] heading home for lunch, back in ~1h [19:00:58] back [19:03:42] * ebernhardson finds this coding style (from a vendor lib) entirely awkward. Wsa the ternary really necessary? isset($args[1]) ? $args[1] |= \JSON_PRESERVE_ZERO_FRACTION : $args[1] = \JSON_PRESERVE_ZERO_FRACTION; [19:06:07] you just don't understand the subtle differences between \JSON_PRESERVE_ZERO_FRACTION and \JSON_PRESERVE_ZERO_FRACTION [19:06:35] it's more that they are doing an assignment inside the ternary, while valid it's rarely done [19:07:10] oh yeah, I missed that [19:07:20] more typical might be: $args[1] = ($args[1] ?? 0) & \JSON_PRESERVE_ZERO_FRACTION [19:07:37] basically just appending to existing args, right? [19:08:34] $args[1] is a bitfield, they are trying to say if the bitfield was provided set this additonal bit, otherwise if the value is unset provide only the desired bit [19:09:16] * inflatador runs to google for 'bitfield' [19:09:42] it's where you treat an integer (or smaller real number) not as the number, but where each bit in the number has a specific meaning [19:10:21] so JSON_PRESERVE_ZERO_FRACTION is 1024, or 2^10, so to preserve the zero fraction you set the 10th bit to 1 [19:10:55] ah OK, sounds a lot like binary code itself if I'm understanding it right [19:11:35] significantly less used these days, but was super common when machines had less memory. These days someone might simply pass an array with string flags or some such (and this is why modern software is only slightly faster despite machines being 1000x more powerful :P) [19:12:45] yea it's treating the number as a collection of individual binary bits, instead of a number [19:12:55] each bit has a specific meaning [19:13:23] Also sounds kinda like IP addressing [19:14:13] yea tcp flags are handled with bitfields as well [19:14:29] all 0= net ID, all 1's = subnet mask, etc [19:14:43] ACK/SYN/RST/FIN are each an individual bit [19:14:58] ahh, yea similar [19:15:07] OK yeah, I'm with you now [19:15:32] Kinda getting a vague urge to write some C all of a sudden ;) [19:16:10] lol, yea C is where i learned bitfields. But at the time memory was measured in megabytes :) [19:16:31] i probably had 16 or 32MB at the time [19:17:13] maybe less...i'd have to go looking what was typical in a 486dx2-66 :) [19:17:33] yeah, I went off to college with a whopping 64 MB of RAM, no one else I new had more than 32 [19:17:46] and the finest Cyrix 6x86 available [19:21:49] I might just blow my paycheck on "Abandon Wear" https://www.redbubble.com/people/Abandon-Wear/shop [19:25:11] lol, that's actually some interesting stuff [20:12:04] * ebernhardson tries to remember why i went map for coordinates instead of struct [20:14:54] also, of course 'Template talk:Coord' is the page with a bunch of coordinates at 1, 1. Should have expected that :P [21:18:00] maybe the cirrus import to hadoop should be flagging pages it fixes..some of these have been reasonably easy to track down but finding example pages for things where we simple delete the column(file_text == false, labels == [], etc.) is proving annoying [21:21:22] quick break, back in ~20 [21:34:44] hmm, cindy bugged out and keeps voting on the citolytics thing :S looking [21:44:15] back [21:44:59] and the answer is...i dunno :P but it's back to working now. Had to manually checkout a recent branch, citolytics patch was so old it broke the test runner