[09:00:20] errand [10:45:43] lunch [13:24:19] dcausse pfischer for the search update pipeline work, what tickets are y'all using? I only see https://phabricator.wikimedia.org/T347075 listed as in progress, LMK If there are others [13:24:22] o/ [13:27:04] inflatador: Peter is out today and he should know better than me but I think I would use T317045 as starting point [13:27:04] T317045: [Epic] Re-architect the Search Update Pipeline - https://phabricator.wikimedia.org/T317045 [13:27:29] it depends on what kind of progress you're looking for [13:28:33] T347075 is definitely a big milestone in all this work [13:28:33] T347075: Deploy test instance of cirrus updater in k8s - https://phabricator.wikimedia.org/T347075 [13:31:18] dcausse ACK, just wanted to make sure...looking at https://gitlab.wikimedia.org/repos/search-platform/cirrus-streaming-updater/-/merge_requests?scope=all&state=merged for progress as well [13:31:27] anyway, mandated reboot! Back in ~30 [14:13:00] back [15:17:40] \o [15:18:27] o/ [15:27:36] workout, back in time for SRE mtg [15:27:41] .o/ [15:34:10] * ebernhardson finally realizes that `source env` is being incredibly dumb and resolving env using $PATH instead of the file named env in the current dir :P [15:53:50] back [16:41:09] ebernhardson LMK if you feel up to reviewing this bad boy https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/967229/ d-causse is out all wk ;( [16:41:29] inflatador: i have to actually look at that one :P gimme a few [16:43:05] ebernhardson no hurry, it is quite a large change. Lots of reconciliation between what's in k8s currently and what's managed by https://gerrit.wikimedia.org/r/plugins/gitiles/wikidata/query/deploy/+/refs/heads/master/flink/flink-job.py [17:03:24] inflatador: sorry I could not get to review this patch before going off :/ will try to adapt flink-job.py today to at least dump the values per env*job to ease the verification [17:05:21] dcausse np, have a great vacation [17:20:16] lunch, back in ~40 [17:48:24] back [18:01:35] updated https://phabricator.wikimedia.org/T347075 to strike through completed steps...if I messed that up, feel free to undo [18:07:15] there are always these random weird edge cases...you can't just take the redirect from Content::getRedirectTarget(), because WikiPage::getRedirectTarget changes NS_MEDIA into NS_FILE :P [18:13:55] :/ [18:19:52] at least have good reproduction now, it's actually fairly easy to get lagged mariadb replica in mwcli [18:34:34] inflatador: https://gerrit.wikimedia.org/r/c/wikidata/query/deploy/+/970428 should help to double-check the job options set in https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/967229/ [18:34:53] going offline [18:47:23] heh, in some mw source: In an ideal world, the DB would be cleaned up after a namespace change, but nobody could be bothered to do that. [18:50:47] thanks again d-causse, have a great vacation [19:33:48] ebernhardson gehel we're in pairing if y'all want to join [19:41:18] inflatador: ahh, i forgot. sec [19:41:29] turns out okta logged out my calendar :P [19:52:07] So after some discussion with our AWS team, my plan is to first do an in-place upgrade of our wikis' Elasticsearch clusters (dev and then live) from 6.5 to 6.8, and then as part of my MW 1.39 upgrade, do an in-place upgrade of the 6.8 cluster to 7.10. I know that for the 1.39 upgrade I'll need to run the CirrusSearch re-index procedure for each [19:52:07] wiki, but I shouldn't need to do anything when upgrading the MW 1.38 wikis' clusters from Elasticsearch 6.5 to 6.8, correct? [20:29:05] justinl: correct, nothing needed there [20:38:52] Cool, thanks. Btw, in the CirrusSearch UPGRADE doc, it says that if you prefer not to do reindexing to at least upgrade the Cirrus metastore and to see the README for more info, but there's nothing about the metastore in there. What exactly does that do? [20:40:43] justinl: metastore is an index that holds metadata, iirc there is 1 doc per index. It should auto-upgrade itself, but you can manually do it with the maintenance/MetaStore.php script in cirrus [20:41:15] it's shared between all wikis using a cluster, it's useful for some things like tracking what git hash was used to create an index [20:41:23] probably not relevant to most users :) [20:42:39] i suppose it's also used for the saneitizer which does background checking of index sanity, but i wouldn't expect you're running that [20:45:29] Nope, not even aware of such a thing. For me the OpenSearch clusters are basically just point and shoot, very basic, nothing custom. [20:48:22] Actually, to that effect, my plan now is to create a new cluster for each major MW upgrade, so now I'd have a MW 1.38 cluster and then a new cluster specific to the 1.39 upgrade, then 1.43 later, etc. I also use an S3 bucket as an OpenSearch cluster index snapshot container, so since the keys are only unique per wiki (starting with the db name), I [20:48:23] couldn't have snapshots for two different major versions in the same S3 bucket, meaning each cluster should have its own bucket. That said, is it possible with CirrusSearch to tweak the key names to add such an identifier to allow one S3 bucket to hold snapshots for different MW versions? That would be useful during such upgrades. [20:49:42] justinl: hmm, i don't quite follow. At least in the snapshot'ing i've done with the elasticsearch api we put an arbitrary snapshot name in, we usually use a ticket number [20:50:37] is it something to do with the partial snapshots? [20:53:06] No, never mind, I had just forgotten my procedure for creating indexes. I'd written a python script to do so but haven't used it in over a year. I would just do e.g. `./opensearch.py create-index-snapshot mysnap1` or whatever. Too many things to remember. :) [22:40:57] hey search folks! thought this might be interesting to someone here: https://forum.opensearch.org/t/anyone-using-opensearch-with-mediawiki-cirrussearch/16478 [22:46:11] Hey, that someone is me! I'm actively working on upgrading my AWS-based wikis from MediaWiki 1.38 to 1.39, which is requiring me to also upgrade my CirrusSearch Elasticsearch clusters from 6.5 to 6.8 and then 7.10. [22:47:10] Or to be more correct, I'm working towards such an upgrade. It's taking a lot of learning and planning due to a number of various software version compatibility issues. [22:48:10] Oh, dang, I misread that, he means the actual OpenSearch software, not Amazon OpenSearch. [23:45:20] * ebernhardson tries to write betterworks stuff, instead decides it's easier to give a multi-paragraph response to a user on phabricator about their query :P