[00:13:26] Looks like we're still running into elasticsearch dependency issues: https://phabricator.wikimedia.org/P19226 [00:14:19] ryankemper: hmm, unable to locate. That means the component hasn't been added to apt yet? [00:21:36] ebernhardson: which would be weird, these are on stretch so the same OS as the existing elastics [00:22:17] Naively I might think it's something to do with apt-fetching but I ran puppet twice before reverting the patch and got the same error [00:22:26] so I don't think it's a "first run fails" type thing [00:22:36] ryankemper: yea i agree, but from the puppet output it seems like /etc/apt/sources.list.d/ doesn't have a file for the component [00:22:55] looking at the puppet i don't see how though :( [00:24:02] ebernhardson: https://phabricator.wikimedia.org/P19227 yeah it's def missing the files [00:25:34] Yeah how bizarre, the puppet output seems to indicate it did have those files [00:26:01] I'll try to dig into it more tomorrow morning, it's gotta be something fairly obvious (at least I hope :P) [00:26:08] :) gl [00:37:48] heh, so the `eqiad.cqs-external.sparql-query` topics aren't created by us, those are being populated by the thing that verifies live streams work. (submitted a patch to fix it) [00:37:56] but then that means wcqs isn't sending anything to eventgate :( [08:05:44] Wat!?! Do we really already have 3B triples in WCQS? https://grafana.wikimedia.org/d/000000489/wikidata-query-service?viewPanel=7&orgId=1&from=now-7d&to=now&var-cluster_name=wcqs [08:06:04] That's already 1/4 of WDQS [08:07:18] and already 310G on disk [08:09:30] If this continues growing at the same rate, we'll be trouble really soon! [08:10:47] It's probably reasonable to expect that there is a pretty high initial growth that slows down significantly over time. [10:17:23] ryankemper, inflatador: in case this wasn't clear from the standup notes, this patch is for you, to be included in your plan to migrate to Elasticsearch 6.8.23. [10:17:38] There is pretty good documentation on how to review and merge on this repo: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/software/elasticsearch/plugins/+/refs/heads/master/README.txt [10:18:13] There are additional steps needed to upload this to our APT repo. [10:19:01] We probably want to do the merge + upload fairly close to the actual migration to ensure we don't deploy the new plugins on an old ES version [10:51:52] Lunch + errand [12:45:06] break [13:14:31] gehel: the exercise took me an hour to do [13:14:35] Break [13:14:54] ejoseph: not too bad! [13:15:06] How many times did you have to touch your mouse? [13:15:26] I tried not to but i had to at some point [13:16:59] Next time: disable the touchpad and unplug the mouse :) [13:23:50] gehel: There was a message from the GLAM-Wiki telegram as well that someone is mass-adding references to all DPLA SDC statements, which will add over 20M references. Maybe that's also part of the spike? [13:26:51] mpham: 20M in 3B would be barely noticeable. There isn't much history in that graph yet, so not sure when the main growth occurred. We could have a look at the size of dumps over time, which would be a good indication [13:30:29] Looks like the oldest dumps we have are from November 21, already 25G (https://dumps.wikimedia.org/other/wikibase/commonswiki/20211121) , and the latest from January 23 only 27G (https://dumps.wikimedia.org/other/wikibase/commonswiki/20220123). [13:30:41] So most of the growth is from before Nov [14:00:24] Greetings! I got my mixer last night, so I should be able to use the good mic with the new Mac now. [14:01:29] Also , nice to see WCQS getting some action [14:04:23] o/ [14:12:59] godog thanks for the invite, see you in ~20 ;) [14:13:27] inflatador: cheers, the invite is for tomorrow though :) [14:13:37] LOL [14:13:48] I plead that it's still early in my time zone ;P [14:14:16] And....see you in ~24:20, then! [14:15:30] inflatador: haha! yeah it is early alright, I wouldn't want to wake up to a meeting I didn't know about [14:16:41] Yeah, that was good etiquette on your part for sure. /me waits for the coffee to kick in... [14:58:34] * gehel just learned that France is working this night (no details, but I presume that whoever was suppose to work this night is in quarantine or isolation or sick in some way) [14:59:04] I'll be alone with the kids. They should be sleeping before any late meetings, so there should be no impact, but who knows... [15:01:27] ryankemper, inflatador: I think this one needs your attention and is required for WCQS go-live: T299222 [15:01:27] T299222: Properly configure logback for W[CD]QS streaming updater - https://phabricator.wikimedia.org/T299222 [15:16:35] :eyes [15:44:17] addshore: is T264642 still valid? [15:44:17] T264642: Wikibase docker images, Log Formatting for WDQS could include a date in the timestamp - https://phabricator.wikimedia.org/T264642 [16:13:56] gehel: I believe so! [16:14:11] addshore: ok, then I'll keep ignoring it! [16:14:17] :) [17:01:01] quick workout, back in ~30 [18:32:41] dinner [18:33:38] sorry, was back for awhile, but now eating lunch [18:36:12] do we still need the wcqs deployment meeting later? [18:36:36] i think we covered most of that already this morning, but can have it if needed [19:45:28] ebernhardson: inflatador: I think we're good cancelling that. when brian and I pair later today we'll go over the deparameterize PR and ping you if we have any questions [19:56:30] ryankemper ebernhardson I'm OK With that [19:58:46] ryankemper , gehel and I are meeting in ~30m to work thru that deployment-prep TLS stuff, sending you the invite now. Let us know if you can make it [20:00:00] alright sounds good, i'll still be around if ya need anything. [20:00:14] inflatador: sure i can [20:01:09] totally unrelated, but the incoming_links tests on cindy are intermittently failing, and not some timeout issue but the incoming_links count never makes it to the expected value :( [20:27:18] inflatador: cool yeah i'll bethere [20:27:19] be there* [20:27:31] (☞ﾟヮﾟ)☞ ＤＯ　ＩＴ！ [20:31:13] looks like my unicode support isn't good enough :/ [20:33:37] lunch [20:34:07] ebernhardson: I've invited you to a weekly SRE pairing session starting a few weeks from now, as discussed. Let's see how that work. [21:11:12] inflatador: gotta go grab lunch and stuff but how's pairing in an hour or so sound on that wcqs patch / etc? [21:11:48] sorry gehel and ryankemper , brt [21:11:57] or at least ryankemper , unless you want to pick it up again later [21:12:37] inflatador: yeah we ended the meeting. I need to run but free to pair in an hour, altho we need to work on the wcqs stuff so I don't foresee us making headway on the deployment-prep stuff during pairing [21:12:56] (in any case you likely need to reach out to j.bond or someone with similar knowledge to get unblocked anyway) [21:13:14] ryankemper understood, we'll do WCQS when you get back, will reach out in SRE for the dp stuff [21:13:22] perfect [21:14:03] FWiW, I think it may be crappy wifi instead of crappy USB (at least I hope). We'll see in an hr or so [21:14:26] that could definitely explain the semi-robot voice...altho jury's still out on the echo [21:19:34] oh yeah, we'll have to look into that too. anyway, see ya soon [21:27:23] back [22:58:23] ebernhardson: https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/757514 is good to merge, looks like I don't have merge permissions on that repo though [22:59:45] maybe I'm supposed to be in https://gerrit.wikimedia.org/r/admin/groups/896bb182e55868ea25eec329ed1142e9f756f254,members? could have sworn I was added to that [22:59:54] not sure if that group is what's needed for the rdf merge perms tho [23:00:19] ryankemper: jenkins/zuul will merge [23:00:58] That "Starting gate-and-submit jobs." comment from jenkins-bot is it saying that it is retesting before merging [23:01:47] bd808: doh! thanks [23:02:06] manual merges are only needed/desirable in repos that do not have CI at all or places like ops/puppet where it was decided that humans should do all the work [23:03:10] ack, thanks for the explainer [23:03:31] and you are not even close to the first SRE to be so used to the ops/puppet workflow that you get a bit confused out in application land :) [23:03:55] guilty as charged :D [23:04:01] it's scary out there [23:04:24] so much...*shudders* business logic [23:09:35] * ebernhardson has the opposite problem, +2's patches and realizing the next day there is no merge-bot [23:23:20] haha [23:37:25] build of https://gerrit.wikimedia.org/r/c/wikidata/query/rdf/+/757514 failed, looks like due to https://integration.wikimedia.org/ci/job/publish-to-doc/6697/console [23:38:32] there are a bunch of "interesting" failures happening in CI. This seems likely related [23:38:34] T300214 [23:38:34] T300214: 'No such file or directory' CI failures in multiple repos - https://phabricator.wikimedia.org/T300214 [23:39:09] ah yeah that must be it [23:39:40] the failure on that publish job is "bash: line 1: rsync: command not found" which seems wacky [23:41:47] so... yeah. the ssh target there that failed with a missing rsync binary is integration-agent-docker-1028.integration.eqiad1.wikimedia.cloud and that instance was built today -- https://openstack-browser.toolforge.org/server/integration-agent-docker-1028.integration.eqiad1.wikimedia.cloud [23:42:43] ryankemper: ^ that's worth a bug in phab linked to T252071 if you have the energy to write it up [23:42:44] T252071: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster/bullseye - https://phabricator.wikimedia.org/T252071