[09:51:26] There is a Product & Tech leadership off-site this week. I'll be unavailable most os the afternoon & evening. [09:51:44] I'll decline meetings that conflict. [09:54:19] I've pulled 2 tickets about migration to Gitlab for our Java projects into our backlog. No emergency for those, but if someone is ready to take them on, ping me and we can discuss the details. There is some context in the tickets, but probably a few more ideas that are only in my brain at this time. [09:55:00] T367405 & T367403 [09:55:00] T367405: Migrate existing Java packages to deploying to Gitlab, including new version of parent pom, validation that all dependencies are available, and validation that deployment to production still works - https://phabricator.wikimedia.org/T367405 [09:55:01] T367403: Validate CI integration so that Ci can release Maven artifacts on user's demand - https://phabricator.wikimedia.org/T367403 [13:14:50] one of the houses on my street caught fire. It's out now and everyone's OK, but it's hazy out [14:07:58] \o [14:08:00] oh no : [14:15:36] yeah, it must have happened very quickly in the early morning. I didn't even hear a siren, although we did see lots of fire trucks [15:20:36] we spent about 20 minutes doing quick backlog click-and-drag from the discovery backlog and wdqs backlog. we'll postpone full board grooming to tomorrow, though. i'll create a new one-off meeting for tomorrow during the same hour [15:28:44] actually...pfischer would you be okay if we started tomorrow morning's meeting 1 hour later? i'm thinking that way both inflatador can also make it and there's a _chance_ that r.yankemper could make it (he would need to reschedule). they both have a different triage tomorrow during the existing hour. [15:39:22] dr0ptp4kt ryan and I are officially optional on Search triage (per g-ehel) so don't feel like you have to schedule around us. I know I can't make it tomorrow at 11:00 CDT either [15:40:27] oh 11 cdt no good tomorrow, good to know. it's always good to have you for triage, but noted. i'll book for 10 cdt in that case as originally conceived [15:49:03] starting to see logstash crashloop alerts on elastic CODFW hosts [15:49:31] elastic2064:9100 and elastic2055:9100 so far...investigating [15:56:33] `RESOLVED: [15x] SystemdUnitCrashLoop: logstash.service crashloop on elastic2055:9100 `...hmmm [16:13:56] human-readable timestamps for logstash would help, but looks like a UDP connection issue between 1538-1544 UTC, at least on the host I checked. Maybe related to backpressure? Anyway, it's cleared [16:35:19] hmm, i wonder what the right unit is for the `missing revisions in search results` dashboard. Errors per minute? Errors over some longer timespan? [16:40:26] uh oh verizon is down all over the place [16:54:21] hmm, the codfw->eqiad mirrormaker seems to have picked up our privae topic over the weekend. eqiad->codfw still hasn't started yet though :S [16:54:49] also eqiad-main -> eqiad-jumbo has only gotten the codfw topic, but not the eqiad topic [16:55:54] i guess it's enough to deploy, since codfw->eqiad is what we need right now. but worry it will simply be broken later :P [17:08:29] sent some new messages, magically it seems to be working in all directions now. [17:36:26] ┐('～`)┌ [18:00:02] looks to all be working now, private events go out over a separate private topic, ingested in the consumer and update the index [18:02:47] lunch, back in ~1h [18:43:00] * ebernhardson tries to remember under which conditions page view popups show on special:search [20:55:27] i wonder what the difference is between normalized_host.project_class and normalized_host.project_family. For every row in our webrequest_metrics table for august they are exactly the same [21:04:36] and the answer is...they are the same :P `result[IDX_PROJECT_CLASS] = normHost.getProjectFamily()` and `result[IDX_PROJECT_FAMILY] = normHost.getProjectFamily()` [21:30:41] Patch up for configuring a wdqs-categories VM if anyone has time to look: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1076841 [22:09:22] is wdqs1001 a placeholder? [22:09:59] otherwise, i have no clue :P seems plausible [23:29:30] ebernhardson good catch, that should be 'wdqs-categories`. fixing [23:33:24] ^^ ryankemper if you have time to look at that categories CR LMK, would be good to get it going before pairing tomorrow