[08:00:47] dcausse, ejoseph: we're welcoming Peter in https://meet.google.com/kfb-dfxb-wmo [08:44:06] Errand, back in a bit [10:05:46] lunch [10:09:36] lunch too! [12:43:48] greetings [12:43:54] o/ [13:07:07] https://discovery.wmflabs.org/wdqs/#wdqs_usage is dead? D: [13:11:41] addshore: yes, these dashboards are all dead :( [13:12:04] If I wanted a number for how many sparql queries you serve per hour / day or whatever, do you have one? [13:12:15] (That is where I would have looked before) [13:15:07] addshore: this should be in hive [13:15:26] in a meeting, will try to point you at the right table soon [13:16:21] I'll have a gaze around [13:19:14] addshore: https://gerrit.wikimedia.org/r/plugins/gitiles/wikidata/query/rdf/+/refs/heads/master/rdf-spark-tools/docs/schema/spark/metrics/general_query_metrics.hql [13:27:11] cant find which databse it is in xD [13:27:31] aaah discovery! [13:29:18] :) [14:02:41] addshore: there is a graph on https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&refresh=1m&viewPanel=18 but it also contains updates [14:37:35] \o [14:43:10] o/ [16:14:30] random curiosity, in the discovery-analytics repo `find . -name '*.whl' -o -name '*.jar' | xargs -L1 echo | wc -l` gives 50, but `find . -name '*.whl' -o -name '*.jar' -print0 | xargs -0 -L1 echo | wc -l` only gives 6 and skips all the *.whl files [16:14:50] * ebernhardson realizes the xargs isn't needed in the first one, but its in there for consistency :P [16:16:02] oh, the -o (or) does much more than we would expect. it wants `find . -name '*.whl' -print0 -o -name '*.jar' -print0` [16:29:01] works too: `find . \( -name '*.whl' -o -name '*.jar' \) -print0 [16:31:28] we should consider this syntax for cirrussearch it's so obvious and fluent :P [16:31:34] :) [16:50:23] are maven sections equivalent to -D... from the java cli? [16:50:52] I'm trying to understand how pom.xml is supposed to provide -Dscalafix.config=.scalafix.test.conf [16:51:00] (but with a reference to a resource in the parent pom, i imagine) [16:51:09] s/pom/jar/ [16:58:03] -D is a java property not sure they override maven [16:58:42] really i want the other way, scalafix only documents providing the path to the scalafix.config via -D, so how to provide that from the discovery-parent-pom? [16:59:12] ah [16:59:16] it's also not clear how to reference a resource from a path [16:59:19] * ebernhardson isn't a java person :P [17:00:00] i suppose, i'm assuming scalafix goes into discovery-parent-pom, the generic config shared between projects goes into the resources folder, and then the pom.xml should be configured to use that config by default [17:00:58] it seems part of the config of the plugin but yes if this plugin is not able to load a resource from an artifact it's not going to work :( [17:02:34] I think some of these resources are in the "discovery-maven-tool-configs" project [17:02:36] this is expecting org.apache.maven.plugins.annotations.Parameter to do all the magic of transforming a property into a File object, i'm guessing maven would support it? [17:03:03] then it should be similar to checkstyle I suppose? [17:04:00] yes a plugin can have a dependency [17:04:27] and then org/wikimedia/discovery/build/tools/checkstyle/checkstyle.xml [17:04:39] if all the magic is handled by maven this might work [17:04:59] ebernhardson: merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/828630 for ya [17:05:31] hmm, totally guessing from looking at the code, in the section of a plugin we provide whatever is the property name? Checkstyle's mojo uses @Parameter annotation on a variable called configLocation, and it seems that gets magic'd in from the section of in the plugin [17:06:26] scalafix seems to have a similar "config" param (https://github.com/evis/scalafix-maven-plugin) [17:07:18] ryankemper: thanks! [17:08:58] dcausse: will try it out and see how things go. I've never understood how these pom's are actually constructed :P Lots of magic when trying to trace what actually happens [17:09:16] me neither :P [17:24:16] * ebernhardson thinks pom.xml should have a better way to specify java version. The only way seems to be setting the compiler version, so any checks that run before the compiler simply fail with random error messages like Parsing signatures failed: Method not found while parsing signature: java.util.ResourceBundle#getBundle(java.lang.String,java.lang.Module) [17:24:50] i suppose the maven answer is to write a custom mojo :P [17:35:21] lunch/errands, back in an hr or so [18:29:54] back [19:09:12] the standup etherpad has gotten quite slow for me, anyone else? Wondering if we should archive the bottom 90% of it or some such [19:09:37] I suppose i don't know for sure, but i'm guessing having 2 years worth of standups and ~10.5k lines is whats making ti slow [19:14:03] i feel like this should be illegal, it fails the build with: [Artima SuperSafe] 2 errors found in /home/ebernhardson/Jobs/mediawiki/wikidata/query/rdf/streaming-updater-producer/src/main/scala/org/wikidata/query/rdf/updater/IncomingStreams.scala that are not reported by the Community Edition. To get full information about the errors, please install your license file in the .supersafe [19:14:05] subdirectory of your home directory on this machine. If you don't have an activated license, you can get a 30 day free trial at: http://www.artima.com/shop/supersafe. [19:14:33] it's one thing to run your checks and put in a bit of an advertisement that you could tell us more. But to fail the build and say pay us to fix it? [19:15:01] * ebernhardson knows nothing about supersafe, but would be tempted to simply chuck it in the bin :P [19:15:12] :) [19:16:04] we could also stop using this I suppose [19:16:32] it's failing now that i re-enabled -Xfatal-warnings in streaming-updater-producer. There were some other failures as well but i've fixed all those and this is the last bit, commenting it out lets the build pass [19:16:52] commenting out the supersafe plugin i mean [19:18:01] I'm fine dropping supersafe I think I copied this pom without much considering all this [19:18:30] ok will do [19:18:58] the file it's complaining about is only 150 lines, but darned if i can guess which part it doesn't like without better details :P [19:19:32] the -Xfatal-warnings was hard to drop tho, IIRC there was unused imports detection that did not play well with ._ imports for implicits [19:20:18] ebernhardson: do you remember what "expected_nodes" is in the elastic.yaml? [19:20:29] and what is the impact of getting it wrong? [19:20:35] hmm, for some reason i haven't seen any unused imports complaints when dropping it in streaming-updater-producer, the complaints were only for incomplete or overcomplete case match's [19:21:13] gehel: When first starting up the cluster will wait up to a certain time limit for `expected_nodes` number of nodes to exist. If the time limit passes and it doesn't have enough nodes it will start the cluster anyways [19:21:26] gehel: it's supposed to prevent the cluster from getting overloaded by traffic before there are enough nodes to handle it [19:21:59] gehel: i suspect it does almost nothing in our case [19:22:45] Oh, so if expected_nodes isn't reached for some time, it will still boot if recover_after_nodes are present? [19:23:00] Too much complexity... [19:23:00] gehel: yea i think thats it [19:24:32] we could probably set it to 0, aparently thats the default in 7.10. That config has been around since at least 1.6 [19:29:31] ebernhardson: I think archiving etherpad is a good idea, let's maybe keep the last ~month or maybe two and shove the rest somewhere else [19:33:56] ryankemper: the ultimate question though, where? :) [19:34:52] ebernhardson: perhaps another etherpad that we link to in the current etherpad? [19:38:14] ryankemper: hmm, seems plausible [19:38:38] ebernhardson: I'll take a swing at it [19:40:59] (done) [19:41:17] We're down to ~620 lines now [19:41:53] I imagine stuff should be snappy again but if not we can chop more, since I left stuff as far back as `2022-07-19` [19:48:55] being silly and doing a bit of a binary-search with replacing methods with null, apparently artima thinks there is a problem with EventWithMetadataHostFilter.filter. uris.getHost.equals(r.meta().domain()) and uris.getHost.equals(c.domain()) both trigger an error. But it's not clear why, one is a java String and one is a scala String, but a scala String is `type String = java.lang.String` [19:48:57] so...? [19:54:41] ryankemper: thanks! [20:51:32] back [23:23:33] heh, summer is officialy over with the holiday weekend, and yet its going to be 42C next week. how fun :P