[09:18:17] <gehel>	 ejoseph: hey! can you ping me when you have a few minutes to chat about ERC?
[09:40:55] <ejoseph>	 gehel: i think we could do it over open hangout
[09:48:37] <gehel>	 ejoseph: I have another meeting in 2', I'll ping you again just after
[10:24:07] <gehel>	 ejoseph: meet.google.com/vsn-cwii-jqa
[11:16:12] <gehel>	 lunch
[14:04:25] <inflatador>	 greetings
[15:06:16] <ebernhardson>	 \o
[15:07:48] <ebernhardson>	 ejoseph: with the variances between linux and macos i suspect my local mods that made mediawiki-cirrus-docker  work wont be relevant to you, but perhaps i'll at least have enough context now we can walk through yours and get it going too
[15:09:26] <ejoseph>	 Thanks so much for helping out
[15:10:39] <ejoseph>	 I feel we should be able to help get it work now
[15:10:45] <ejoseph>	  can we look into that tomorrow though?
[15:10:50] <ebernhardson>	 ejoseph: certainly :)
[15:11:08] <ejoseph>	 I am looking into some of the  elastic slides 
[15:11:45] <ejoseph>	 gehel: I might not be able to attend retrospective next week
[15:12:24] <ejoseph>	 It clashes with my elasticsearch training
[15:12:40] <gehel>	 ejoseph: let me know. If we want to talk about ERC, we need you to be there.
[15:13:16] <ejoseph>	 And i also wont be able to make ERC meeting too
[15:22:08] <gehel>	 Do you need me (or someone else) to replace you for the ERC meeting ?
[15:22:56] <gehel>	 ejoseph: unrelated: maybe inflatador can help with you Mac slowdowns. He is using an M1
[15:22:56] <stashbot>	 M1: MediaWiki Userpage - https://phabricator.wikimedia.org/M1
[15:23:58] <inflatador>	 ejoseph you are using an Intel-based Mac, right? I think we might be discussing this stuff tomorrow?
[15:25:09] <gehel>	 ejoseph: I have my first interview for our SWE next week. I forwarded the meeting to you. If you're available.
[15:26:22] <ejoseph>	 I would still be in class 7pm on Modnay ☹️
[15:26:31] <ejoseph>	 Monday*
[15:26:37] <gehel>	 no problem, we'll have others as well
[15:26:43] <ejoseph>	 Ok
[16:09:58] <gehel>	 inflatador: is there anything left to do on T276198 ? Did you restart at least one node on each cluster to ensure the change works as expected
[16:09:58] <stashbot>	 T276198: /var/run/elasticsearch deleted by elasticsearch - https://phabricator.wikimedia.org/T276198
[16:11:14] <ebernhardson>	 great investigative work there, but also more proof that ideas that seem not that difficult (just run two instances on the same hardware, how hard could it be) often "work" but have painful cases we don't recognize :S
[16:12:15] <inflatador>	 gehel we restarted in eqiad yesterday, but we probably should try a few more nodes today
[16:13:27] <ejoseph>	 infladator: yes i am using intel based mac
[16:13:50] <gehel>	 I'm not sure a full cluster restart is required in this context, there should be no changes to the state of elasticsearch after this change.
[16:13:51] <inflatador>	 ebernhardson thanks! I found that phab ticket where you found the madvise syscall, you are my troubleshooting hero ;P
[16:14:00] <gehel>	 Unless I'm missing something
[16:15:47] <inflatador>	 gehel it's probably safer to restart the entire cluster, "minimize time that running state differs from configured state" but I agree it's not necessary in this specific case
[16:16:22] <ebernhardson>	 inflatador: lol, that one took more than a week, and that was pretty much the only thing i was doing since the rest of the org was off at wikimania
[16:16:23] <gehel>	 Your call !
[16:18:11] <inflatador>	 gehel Let's plan on restarting a single node in the other clusters, I don't think a full restart is necessary
[17:00:14] <inflatador>	 quick break, back in ~30
[17:21:32] <ebernhardson>	 for puppet deploy window today i have https://gerrit.wikimedia.org/r/c/operations/puppet/+/765667 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/767259
[17:22:00] <ebernhardson>	 (it's not started yet, but gcal reminded me so putting patches here)
[17:30:49] <ryankemper>	 ^ thx
[17:30:58] <ryankemper>	 I’ll be about 5 mins late to the deploy window
[17:51:22] <inflatador>	 ebernhardson I can't make the window today, sorry for late notice. ryankemper should have you covered, if not I am at your service after 1 PM your time
[18:00:07] <ebernhardson>	 inflatador: i think we got it handled, thanks!
[18:45:52] <inflatador>	 lunch, back in ~30-45
[19:27:28] <inflatador>	 aaand back
[19:31:10] <gehel>	 ryankemper: SRE pairing session: https://meet.google.com/ckm-dmmh-opt
[20:56:54] <ebernhardson>	 lunch
[21:28:43] <inflatador>	 ebernhardson we're trying to build the plugin deb from https://gerrit.wikimedia.org/r/c/operations/software/elasticsearch/plugins/+/763485 and we're getting "Error: Component 'component/elastic71' as given to --component is not know." ... any ideas?
[21:38:22] <ebernhardson>	 inflatador: should it be elastic710 perhaps? checking
[21:39:01] <ebernhardson>	 hmm, may not matter in this case since it doesn't exist yet (looking at https://apt.wikimedia.org/wikimedia/pool/component/)
[21:40:20] <inflatador>	 ebernhardson will give it a shot. It looks like elastic710 exists in the repo at /modules/aptrepo/files/updates
[21:40:44] <ebernhardson>	 inflatador: i suspect it has to be defined here: 
[21:40:46] <ebernhardson>	 https://phabricator.wikimedia.org/source/operations-puppet/browse/production/modules/aptrepo/files/distributions-wikimedia
[21:41:03] <ebernhardson>	 but i don't actually know, just poking things :)
[21:41:53] <inflatador>	 yeah, that actually makes sense
[21:42:46] <inflatador>	 modules/aptrepo/files/updates has it but that distributions-wikimedia file does not
[21:44:18] <ebernhardson>	 yea that would make sense then, and agrees with https://gerrit.wikimedia.org/r/c/operations/puppet/+/765253 which added a component recently
[23:12:14] <ebernhardson>	 taking off a little early today