[10:30:40] errand+lunch [14:14:40] gehel I opted out of our 1x1...I think we're OK to go back to 1 mtg/wk...let me know what you think [14:16:21] forced update...back in ~30 [14:30:49] inflatador: ok [14:51:52] back [15:41:45] dcausse OK, so the swiftly command is `swiftly get rdf-streaming-updater-codfw -p wikidata` ... the p is for prefix [15:42:18] ah thanks! same as the main swift command then [15:42:51] I also just realized that I have Cyberduck (https://en.wikipedia.org/wiki/Cyberduck) , so I can probably open a tunnel and browse with the GUI [16:46:55] Small CR to fix Puppet on cloudelastic1010 if anyone has time to look: https://gerrit.wikimedia.org/r/c/operations/puppet/+/994763 [16:58:36] workout, back in ~40 [17:43:10] back [18:47:16] lunch, back in ~40 [20:27:36] still trying to figure out why this value https://gerrit.wikimedia.org/r/c/operations/puppet/+/994800/9/hieradata/role/common/elasticsearch/cloudelastic.yaml#108 comes out of the Puppet wash without its domain (ref https://puppet-compiler.wmflabs.org/output/994800/3063/cloudelastic1002.wikimedia.org/index.html ) [20:41:15] inflatador: https://gerrit.wikimedia.org/r/c/operations/puppet/+/994800/9/hieradata/role/common/elasticsearch/cloudelastic.yaml#70 is why [20:43:09] ryankemper Awesome, thanks for the hint [20:45:52] might need a better solution, but for the moment i've setup a bash script on mwmaint1002 i can run that will run compare-cluster.py between eqiad and cloudelastic and write the results to a timestamped directory. Have to manually invoke, but hopefully can compare a few days (in addition to the saneitizer stats) [20:47:01] for all the wikis currently being written to [21:05:51] ryankemper OK, based on your info this is ready to review https://gerrit.wikimedia.org/r/c/operations/puppet/+/994800 [21:06:50] ebernhardson cool, if you wanna do something fancier as a kubernetes batch job or something LMK...would be a good excuse to get familiar w such things [21:07:28] inflatador: one small formatting nit but otherwise +1 [21:07:33] heading to gym, back in time for pairing [21:19:12] inflatador: would be more tedious :P There is probably a better way, but currently i find out which cluster to query for each wiki by running a mw debug script [21:20:13] * ebernhardson is also mildly surprised at the rate at which wikidata is emitting inconsistencies between the two clusters..should have run a baseline prior to switchover [21:23:51] well, the puppet change blew up on CE1010. Puppet is off on all CE hosts, so not a huge deal [21:26:20] looks like the cloudelastic value at https://gerrit.wikimedia.org/r/c/operations/puppet/+/994800/14/hieradata/role/common/elasticsearch/cloudelastic.yaml#b71 actually resolves to a directory on the host where the letsencrypt data lives...not directly related to the cert data itself [22:15:24] * ebernhardson wonders if we should prune parts of wikitech:Search. Do we really need a section on estimating the size of a wiki prior to adding it to a cluster? They are all new wikis these days [22:55:21] * ebernhardson was bold and removed it :) [22:55:50] {◕ ◡ ◕} [23:02:00] had to roll back again...it still doesn't like creating a directory that's an FQDN (at least I think that's why it's complaining). Looking at acme-chief, which is basically a letsencrypt proxy, none of the directories that hold certs are FQDNs [23:02:52] but that's a problem for tomorrow ;) [23:15:20] i'm looking to run IGUANA again on the full and main graphs on the graph split hosts. this time it'll be two 5-hour runs in a row, roughly [23:22:44] kicked it off...