[07:51:03] good morning all [07:51:26] I am back, thanks for checking in on me [07:56:17] ejoseph: hey, glad to see you feel better! [08:00:01] thanks, David [10:00:23] lunch [10:05:19] Lunch [10:09:06] Hello everyone, I have a copy of the feedback I am submitting to the ERC [10:09:09] https://docs.google.com/document/d/1C5V0b8DKqx8dlQ9u3A5bsfYLmQ1sBeZqIUEx4DTnpSo/edit?usp=sharing [10:09:53] Kindly review to see if I am misrepresenting or omitted anything [12:34:14] ejoseph (and all): I've added a few more comments to that doc [12:35:00] @all: we're already past the deadline to review this doc. Could you make sure you review it and let Emmanuel and myself know when it's done? Before today - end of day? [12:41:29] ejoseph: all good from me, thanks for working on this! [13:07:47] greetings [13:33:42] o/ [13:34:20] inflatador: do you know much time it would take to upgrade codfw to elastic7? [13:35:21] I haven't been following recent cluster rolling restarts so can't judge how long it will last [14:05:27] dcausse Let me check my notes, but with the current tooling I think it's something like 5-6 hours [14:05:49] oh wow that's fast [14:11:04] inflatador: we want to reimage to bullseye all nodes before upgrading to es7 right? [14:13:55] dcausse yes, we should be done with the bullseye upgrades within the next few wks [14:14:07] cool, thanks! [14:19:37] can we close T301959 ? [14:19:38] T301959: Upgrade Search elasticsearch cluster / eqiad to elasticsearch 6.8.23 - https://phabricator.wikimedia.org/T301959 [14:22:33] we should probably close T218995 too I think [14:22:33] T218995: re-enable deprecation warning logger on elasticsearch once issues are solved - https://phabricator.wikimedia.org/T218995 [14:23:36] closing those, feel free to reopen them if I'm wrong [14:23:40] For T301959 , have we addressed the question at the bottom of the ticket? [14:23:51] Other than that I'm totally fine with whatever [14:24:15] looking [14:26:40] ok done [14:28:03] filed T308676, I tried to list all known prerequisites and draft a plan, please fix/amend anything you think is wrong or missing [14:28:04] T308676: Elasticsearch 7.10.2 rollout plan - https://phabricator.wikimedia.org/T308676 [14:29:01] we could pack Week 2 and Week 3 if we're really fast rolling out elastic on codfw but not sure we should pressure ourselves here [14:29:20] I mean pack Week 1 with Week 2 [14:29:53] errand, back later today [14:36:43] OK, will take a look [15:01:50] ryankemper, mpham, Trey314159 : Wednesday meeting, and Tyler is here to talk with us! [15:20:52] i'm in a product meeting at the same time [15:43:41] ebernhardson was that from the Stretch decoms last wk? Let's follow up offline if so [15:57:58] workout, back in ~30 [16:39:00] back [17:02:07] ryankemper, inflatador, ebernhardson, Trey314159: can you have a look at the ERC feedback doc and confirm you are ok with it ? See email for details. [17:04:27] gehel thought I did that already? Will look again [17:08:22] I added a small note, but confirming again that I have read/understood [17:38:44] lunch, back in 30-45 [18:13:53] back [18:28:27] inflatador: there is another document that Emmanuel shared today, with notes from our discussion. Those notes will be shared with the ERC Core team [18:29:12] gehel thanks, I did miss that one, reviewing now... [19:15:43] quick errand, back in 30 [19:35:37] dcausse, ebernhardson: a question you might be able to answer: https://lists.wikimedia.org/hyperkitty/list/wiki-research-l@lists.wikimedia.org/thread/VXTSXKSG6WGEPHWJRTQQOHFV53H3LPUD/ [19:47:15] \o [19:48:25] inflatador: there's a couple tickets, https://phabricator.wikimedia.org/T308339 and https://phabricator.wikimedia.org/T308331, that will require us to ban/depool a couple elastic hosts so that they can be reracked. It sounds like it won't happen for several months (https://phabricator.wikimedia.org/T308339#7937642) so there's no action required for us now, but I went ahead and added you as a subscriber to those two tasks as well [19:55:51] gehel: i dunno, noone ever asked for the models. They could easily be made public [19:56:11] well, "easily" as a one time thing, some process to auto-publish would of course be more work [19:57:16] Any chance of leaking pii ? My limited understanding is that it is unlikely, but... Limited understanding [19:57:53] gehel: i guess we could double check with research, but the trees amount to if(title_match > 0.5) { if (near_match < 0.4) { if ... [19:58:29] i think the chances are pretty low [19:59:09] as a one time thing, it's mostly copying the right file out of hdfs [19:59:28] back [20:01:41] inflatador: re: beta breakage, it wasn't the search clusters. changeprop had old decomissioned kafka hostnames, and mediawiki had decomissioned eventgate hostnames [20:06:48] apparently i can respond to the list from the mailman interface, nifty. Will try and write something [20:10:08] ebernhardson gotcha, I think I remember seeing htat one now that you mention it [20:24:03] sorry, not feeling so good. Going to pick up some meds and hopefully will see you tomorrow [20:26:21] inflatador: feel better!