[10:33:39] <ejoseph>	 dcausse: my Mac went off after installing virtualbox
[10:33:46] <ejoseph>	 And won’t come on
[10:33:56] <dcausse>	 :(
[10:37:56] <ejoseph>	 It's on now
[10:38:09] <ejoseph>	 😤
[10:38:39] <dcausse>	 ejoseph: I feel your pain... not sure what we can do about that
[10:39:04] <dcausse>	 seems like broken hardware but I'm not knowledge on the mac eco-system to tell for sure
[10:39:38] <dcausse>	 gehel: can our ITS help on this kind of issues?
[10:39:53] <gehel>	 maybe, not sure
[10:39:57] <gehel>	 it's worth contacting them
[11:03:49] <dcausse>	 lunch
[13:07:17] <inflatador>	 Greetings
[13:19:18] <dcausse>	 o/
[13:30:51] <inflatador>	 <o/
[14:39:45] <ebernhardson>	 \o
[14:41:21] <inflatador>	 having weird SSH issues, rebooting to see if it helps
[14:41:34] <dcausse>	 o/
[14:41:55] <ebernhardson>	 not finding myself super happy with Elastica, i suppose i had assumed if we wait long enough they would have a good transition story from 6->7, but for the most part they don't. Examples like $index->getMapping() don't have a place in the api for us to tell it to use the new version which either means ignore. Only the underlying elasticsearch-php library has good support for this :S
[14:42:15] <ebernhardson>	 err, means ignore or replace with calls to the underlying library
[14:42:29] <dcausse>	 :/
[14:43:09] <ebernhardson>	 i wonder what Elastica is really buying us in most parts, but then if we dropped elastica surely we would just end up with a new class called Index that does the same things :)
[14:43:45] <dcausse>	 won't the dc-switch-with-the-train-hack help on this?
[14:44:13] <ebernhardson>	 i think that's what i've been considering the ignore part, we just let 6.x spew a bunch of errors and hope we fix it all in the 7.x branch
[14:44:34] <ebernhardson>	 seems less ideal, but probably works.
[14:45:23] <ebernhardson>	 i suppose i liked transitioning whats there so it happily runs against 6.x and then 7.x was to get confidence by running the code in prod, rather than dropping a large branch at once. But we could
[14:46:20] <dcausse>	 if it's only getMapping() maybe that's manageable because it's only called on main scripts IIRC?
[14:46:53] <ebernhardson>	 it's not just getMapping, although it might be limited to the kinds of things that deal with include_type_name (mappings, settings, index creation, etc.)
[14:47:06] <ebernhardson>	 i suppose that's the only part i've started looking into so far, and i'm worried the rest will be the same :P
[14:47:23] <inflatador>	 back
[14:47:36] <dcausse>	 ah yes might well be :/
[14:47:53] <dcausse>	 btw regarding deprecation warnings
[14:47:56] <ebernhardson>	 it's more like, i don't see any obvious attempts of building a transition from 6->7 here
[14:48:02] <ebernhardson>	 from upstrea
[14:48:04] <ebernhardson>	 m
[14:48:21] <dcausse>	 isn't there a log file managed by elastic for this?
[14:48:41] <dcausse>	 (just saw your patch to pull that from the elastica transport)
[14:49:15] <ebernhardson>	 dcausse: there is, but they had a 30% ingestion reduction by adding warnings. So now there are thread-local variables that supress duplicate log messages related to deprecation
[14:49:58] <ebernhardson>	 my reading of the tickets at least was that they had a huge regression in speed from adding these deprecation warnings everywhere, so they refactored things to greatly reduce how many are emitted
[14:50:33] <ebernhardson>	 i didn't follow it in too deeply, but elastic doesn't want to emit the log message if it's seen that message in the last few (not sure how many) minutes
[14:50:46] <dcausse>	 ok I see
[14:51:37] <ebernhardson>	 the Warning headers seem consistent though, and that also gives stack traces on our side which is convenient
[14:51:46] <ebernhardson>	 not sure that can go into prod though, would be super-spammy
[14:52:10] <dcausse>	 maybe enabled on low traffic wiki?
[14:52:17] <dcausse>	 or "test" wiki
[14:52:18] <ebernhardson>	 hmm, yea that would work
[14:53:35] <ebernhardson>	 quasi related, also have to rework Metastore. It still references Index Type's (and trys to update the during a minor upgrade, wonder if those even work :)
[14:53:54] <ebernhardson>	 intending to drop minor upgrades and let everything go through the reindexing there
[14:56:17] <dcausse>	 automatic upgrade/migration based on versioning? these things rarely works :P
[14:56:55] <ebernhardson>	 :)
[14:59:51] <dcausse>	 inflatador: when you have a some time I would need your help for switching wdqs traffic to codfw (context is https://phabricator.wikimedia.org/T302494)
[14:59:58] <inflatador>	 We are meeting in https://meet.google.com/yau-mkip-tqg  , right?
[15:00:10] <dcausse>	 yes I think so
[15:00:21] <ebernhardson>	 inflatador: yea we unified all the rooms to the same name, but now the room has a generic name
[15:00:33] <ebernhardson>	 maybe overly specific and not generic enough
[15:00:45] <inflatador>	 dcausse sure , should be able to do that by EoD today
[15:01:19] <dcausse>	 inflatador: thanks!
[15:25:43] <razzi>	 Thor's hammer Mjolnir... I can see it now https://usercontent.irccloud-cdn.com/file/Xc62wFK0/thors_hammer.png
[15:53:07] <dcausse>	 lol
[15:54:57] <mpham>	 I'm not worthy to lift that
[15:56:43] <ebernhardson>	 lol
[16:02:25] <inflatador>	 razzi ROFL
[16:02:49] <razzi>	 :)
[16:07:29] <inflatador>	 quick break, back in ~30
[16:33:48] <inflatador>	   back
[16:34:55] <dcausse>	 inflatador: do you have a couple minutes for chat about some context on wdqs puppet changes?
[16:35:39] <inflatador>	 yes, I just started meet.google.com/pbq-rfnn-vov
[16:35:51] <inflatador>	 dcausse ^^
[16:35:55] <dcausse>	 sure joining
[17:02:23] <gehel>	 dinner
[17:04:20] <dcausse>	 errand
[17:11:06] <inflatador>	 ryankemper LMK if/when you have time to help out on https://gerrit.wikimedia.org/r/c/operations/puppet/+/770508 . Per dcausse we'll need to flip the WDQS prod service from eqiad to codfw
[17:12:07] <inflatador>	 also, looks like https://discovery.wmflabs.org/wdqs/ is down, anyone else seeing a problem?
[17:15:08] <inflatador>	 Is that just a link to https://grafana.wikimedia.org/d/000000489/wikidata-query-service?orgId=1&refresh=1m ?
[17:16:55] <ebernhardson>	 inflatador: i'm not sure what that would be, i'm not remembering us having discovery.wmflabs.org previously setup
[17:17:05] <ebernhardson>	 (doesn't mean we didn't, i forget things regularly :P)
[17:19:31] <ebernhardson>	 i suppose that might have been the old dashboards from ~5 years ago? they were certainly decommed. They would have been custom data analysis that released csv files from analytics and the frontend would have parsed and rendered those data files
[17:23:00] <inflatador>	 Interesting. It's still listed as the dashboard at https://wikitech.wikimedia.org/wiki/Wikidata_Query_Service and I have it bookmarked , I wonder 
[17:25:04] <ebernhardson>	 inflatador: poking at my email history, it suggests it was still kinda/sorta running a year ago. TBH I'm not sure when exactly we killed that, or even which instance it was actually running from
[17:29:35] <inflatador>	 ebernhardson interesting, I'll check around and see if it needs to be decommed/updated docs etc
[17:37:43] <inflatador>	 lunch, back in ~45
[18:25:37] <inflatador>	 back
[18:36:21] <ryankemper>	 inflatador: I can pair @ 1pm my time if that works for you?
[18:37:12] <inflatador>	 ryankemper I've just been informed that I have to pick up my son, so 2 PM is better if that's OK w/you
[18:49:58] <ryankemper>	 inflatador: works for me
[18:51:10] <inflatador>	 ryankemper ACK, will hit you up around that time
[19:09:58] * ebernhardson keeps deleting more code from metastore. Sooner or later will start actually fixing it :P
[19:13:50] <dcausse>	 inflatador: I think you can remove any references to https://discovery.wmflabs.org/wdqs/, these dashboards were no longer maintained and were dropped
[19:16:02] <inflatador>	 dcausse thanks, will remove from docs and figure out where DNS lives in all that stuff, if you have any other info let me know
[19:16:37] <dcausse>	 I think "discovery.wmflabs.org" is managed under horizon as a web proxy
[19:21:38] <dcausse>	 hm... can't seem to find it in horizon, tried "search", "shiny-r" and "wikidata-query" projects...
[19:22:10] <inflatador>	 yeah, I couldn't find it either. Was gonna check netbox for DNS records or something
[19:23:30] <volans>	 wmflabs is entirely managed in the realm of WMCS, so will not be in netbox or the prod dns repo ;)
[19:23:55] <inflatador>	 ah yeah, I see it's got 'openstack' in its NS record
[19:25:11] <gehel>	 inflatador: I think that bearloga_ did some work on decomissioning discover.wmflabs.org semi-recently. Those dashboards were unmaintained for a long time.
[19:26:06] <dcausse>	 seems like a wildcard record *.wmflabs.org so nothing left to clean up (other than refs from various docs)
[19:28:25] <inflatador>	 Thanks dcausse , I can close at least one ticket today ;)
[19:28:44] <dcausse>	 yw! :)
[20:10:57] * ebernhardson wonders if opensearch is going to finish the index types deprecations on the same schedule as elasticsearch did, since 7.x still retains the old formats toggleable by query flag
[20:19:54] <inflatador>	 picking up my son, back in ~45
[20:54:15] <inflatador>	 and back
[20:57:34] <mpham>	 Is there a phab ticket or anywhere that we are centrally tracking fixing our update pipeline?
[21:03:04] <inflatador>	 ryankemper OK I'm up at https://meet.google.com/qjs-gfeg-waa whenever
[21:12:06] <ryankemper>	 inflatador: cool, i'll be there in a few
[21:52:38] * ebernhardson tries to remember why we maybeCreateMetastore() is only allowed to create an index that doesn't exist, but not upgrade the index if it already exists
[22:20:49] <inflatador>	 d-causse we moved wdqs and wdqs-updater to codfw 
[22:34:52] <ryankemper>	 ^ wdqs and wdqs-internal*
[22:37:46] <gehel>	 mpham: I don't think we have a phab ticket yet. It's too vague of a project at this time