[09:43:18] inflatador: banning a node will shuffle shards around, and there's a good chance that the result is a more balanced cluster... [09:49:53] errand+lunch [12:47:41] pfischer: what's the procedure to make a new release of the wmf-jvm-utils project? [12:55:22] sigh... opensearch CI is something, 1h build time and extremely unstable... [13:12:03] dcausse: looking [13:14:19] o/ [13:21:00] dcausse: So this is built via jenkins and it uses parent pom 1.92, but with an outdated `gitlab.projectId`, so if you need it quickly, you can create a release locally by exporting CI_RELEASE_TOKEN and overriding -Dgitlab.projectId=2745 (https://gitlab.wikimedia.org/repos/wmf-packages, see settings > CI/CD for the token) [13:21:51] I can do that for you, if you want. [13:24:44] pfischer: no worries, I can run it I guess, should we just update the pom to target 2745? or is there a reason not to? [13:26:20] and do we need a jenkins build that runs maven-release, it does not seem to have one at the moment [13:26:51] dcausse: I just didn’t manage yet. While working on a catch-all package-registry-only GitLab project, I learned about Erik’s wmf-packages repo, whose token is already available via Jenkins. Hence we should use Eriks (2745) instead of mine. [13:28:17] pfischer: ok, so the plan could be: update the pom to target 2745, add a maven-release job to that project in jenkins jjb config, run the build from the jenkins UI, does this sound correct? [13:29:08] Yes, I just started to focus on the gerrit-hosted JVM projects today, so I’ll do that. [13:29:32] pfischer: thanks! [13:36:48] does anyone have any strong feelings about whether or not we should failover to CODFW before I bring some net-new EQIAD cirrus hosts into production? I feel like it's safe to do it without a failover but LMK if y'all disagree. ref T391118 [13:36:49] T391118: Bring cirrussearch1111-1122 into production - https://phabricator.wikimedia.org/T391118 [13:37:14] I also pinged ServiceOps to review https://gerrit.wikimedia.org/r/c/operations/puppet/+/838182 again [13:39:14] inflatador: I think it's fine, worst case I could think of would be new indices (like completion suggester) failing to go green if they're built there? [13:41:09] dcausse ah yes, let me get that patch up to disable the completion indices as well. Although I guess we shouldn't merge until we have failed over to CODFW [13:41:50] inflatador: it could go out now if we're confident we'll be failing over pretty soon in the next 2days [13:57:22] \o [13:58:41] .o/ [14:00:08] o/ [14:01:16] I'm confident we will be...I guess we need to get a mwconfig patch up though since we haven't gotten the review from service ops [14:06:58] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1141903 [14:26:24] FYI: the research brief about Reader Discovery is published: https://meta.wikimedia.org/wiki/Research:Reader_Discovery [14:50:43] quick errand, might be a bit late for triage [16:00:23] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1141914 CR for bringing on our first EQIAD hosts if anyone has time to look [16:09:47] dcausse: figure out the wmf-jvm-utils release? Poking around, it seems like maybe that hasn't been setup yet [16:11:30] ebernhardson: Yes, according to integration/config, non of the search projects does use maven-release job yet. [16:13:57] I already pinged hashar to get access to jenkins-manged config files (e.g. holding the global maven settings with credentials for archiva) so we could leverage that for GitLab as a package registry too. [16:15:27] ebernhardson: IIRC you published glent to GitLab through your wmf-packages repo. Did that run through jenkins/zuul? According to integration/config that can’t be, but maybe I am mistaken. [16:15:52] pfischer: i was trying to remember similar :) I certainly added the credentials for something...but didn't put a ticket number in the patch. Looking [16:17:04] Yes, CI_RELEASE_TOKEN is there, but that would only be picked up (via parent POM) to get the right SCM URL. It would not be used to authenticate towards the package registry (GitLab API, that requires `Job-Token` header) [16:17:55] pfischer: the patches imply search/extra and search/glent were both configured to release through gitlab, i only see glent in the registry though [16:18:48] ebernhardson: https://gerrit.wikimedia.org/r/plugins/gitiles/integration/config/+/fd9eaf18d047a619d5144a38b1ce30fca787e4d5/jjb/search.yaml [16:21:31] By default the java8/11 maven job only runs clean install (I do not understand why an install though) [16:21:49] So it would not even deploy snapshots to archiva/gitlab [16:22:24] pfischer: hmm, so i configured the properties in https://gerrit.wikimedia.org/r/c/search/glent/+/1092307/1/pom.xml ... i have to imagine i must have done the release from my machine, since jenkins isn't configured to do it [16:23:25] yea, in my bash history i have manual release:prepare release:perform runs at that same time [16:23:35] ebernhardson: I guess so, and those config properties are relvant but not sufficient (authentication is required) [16:24:03] You probably have sth. set up in you ~/.m2/settings.xml for authentication [16:24:18] i just have CI_RELEASE_TOKEN on the cmdline [16:25:02] Hm, that only effects the SCM url (relevant for tagging and pushing via git) [16:25:48] So that would still not explain how the deploy to wmf-packages got through [16:26:05] hmm, i guess i do have a Private-Token configured in ~/.m2/settings.xml, has the same value as CI_RELEASE_TOKEN [16:26:20] Jup, that’s it [16:26:32] Job-Token or Private-Token, both work as header [16:28:51] Hopefully zuul exposes some unique environment variable so I can move all those jenkins-to-gitlab-specifics in a profile in parent pom, activated by that environment variable. Alternatively, an explicit `-P jenkins` might do the trick to [16:34:23] zuul provides a few zuul specific vars, like ZUUL_URL, ZUUL_PROJECT, etc. [17:40:48] lunch/eye exam, back in ~90m or so [18:52:37] ebernhardson: do you have anything urgent for our 1:1? Or can we cancel / reschedule? [18:55:34] gehel: we can cancel [18:56:10] thanks! [19:03:24] pondering, pretty sure wmf-jvm-utils has to be published to central now, to be referenced from the highlighter which also goes to central iirc [19:05:44] inflatador: got some patches for ya https://gerrit.wikimedia.org/r/c/operations/puppet/+/1141956 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1141957 [19:05:49] (shifting over old wdqs-internal hosts to wdqs-main) [19:24:21] back [19:25:35] but I'm gonna have to leave again, eyes still too dilated. ryankemper I'll take a look at those once my eyes go back to normal [19:47:40] nw [20:41:05] OK, my eyes are a bit better. Gonna give those CRs a shot [21:15:25] ebernhardson: I pinged you on a paste on Phab. I can't get reindexing to run, and I'd appreciate any pointers in the right direction. I'm done for the day, so no rush.