[08:27:52] <dcausse>	 arwiki got indexed, seems to work ok so far https://ar.wikipedia.org/w/api.php?action=query&format=json&list=search&formatversion=2&srsearch=why%20do%20stars%20shine?%3F&cirrusSemanticSearch&srlimit=5, will update the demo
[09:18:35] <dcausse>	 updated https://people.wikimedia.org/~dcausse/semsearch_hl_quick_demo.html
[09:57:34] <dcausse>	 hm I think that with the new 3.8 cirrus image we lost prometheus metrics
[10:11:24] <dcausse>	 sigh... somehow 3 shards got stuck in recovery in opeansearch-semantic-search@codfw leaving all indices in yellow, add to explicitly cancel some to unblock things... 
[10:13:12] <dcausse>	 https://phabricator.wikimedia.org/P96526, 1.6 days in this state apparently...
[10:13:29] <dcausse>	 dropped all audilogs as well to ease movement
[10:14:39] <dcausse>	 lunch
[12:28:05] <dcausse>	 https://gitlab.wikimedia.org/repos/search-platform/cirrussearch-opensearch-image/-/merge_requests/31 to keep prometheus
[13:17:03] <inflatador>	 <o/
[13:21:45] <dcausse>	 o/
[13:24:32] <inflatador>	 dcausse ACK, just merged https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1345099 as well
[13:32:54] <dcausse>	 inflatador: thanks! will restart these clusters
[13:44:59] <ebernhardson>	 \o
[13:49:05] <dcausse>	 o/
[13:49:34] <inflatador>	 .o/
[13:53:33] * ebernhardson starts the opensearch-semantic-search-ssd cluster again, but expecting it to get stuck finding enough memory
[13:54:07] <dcausse>	 meh... restarted the test cluster without applying the helm changes...
[13:54:10] <inflatador>	 Is that just because there aren't enough k8s workers at the moment?
[13:54:41] <ebernhardson>	 inflatador: i think it's because topolvm-balanced can only allocate to 4 nodes (dse-k8s-worker100[1-4])
[13:55:02] <ebernhardson>	 inflatador: i suppose my hope is perhaps there are pods there that don't specifically require topolvm-balanced, they just landed there, and can be moved away?
[13:55:27] <ebernhardson>	 i don't know how much i can actually get from them, grafana shows those nodes are 128G memory so there is only 512G total,  the cluster is currently asking for 16*16=256G
[13:56:14] <ebernhardson>	 dcausse: yea...something i've also done a few times.  silly how easy it is
[13:56:23] <dcausse>	 are there other pods assigned to these machines? I thought they'd be free for this test
[13:56:46] <ebernhardson>	 i don't really know how to check, i didn't find specific data in grafana and we cant access `kubectl get nodes` and related
[13:57:25] <inflatador>	 ebernhardson I'll take a look. I'm guessing that there are workloads that can move, but I'm also interested in the disk config on those hosts...if it's not specialized we could maybe add more hosts to `topolvm-balanced` node class or whatever you call it
[13:57:43] <ebernhardson>	 inflatador: thanks!
[13:58:21] <ebernhardson>	 it's currently sitting with opensearch-semantic-seach-ssd-masters-1 in the Pending status, the 3 masters have to each go on a separate host, so that could also be a potential angle
[13:59:19] <inflatador>	 https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/regex.yaml#102 that's where the workers are defined. Those are also the oldest k8s workers we have, so it makes sense that other workloads would be there already
[13:59:46] <ebernhardson>	 the only info i could get is a bit under-conrete, this is from kubectl describe: 0/35 nodes are available: 1 Insufficient memory, 2 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 25 node(s) did not have enough free storage, 3 node(s) were unschedulable, 4 node(s) had untolerated taint {dedicated: wdqs}. preemption: 0/35 nodes are available: 1 No preemption
[13:59:48] <ebernhardson>	 victims found for incoming pod, 34 Preemption is not helpful for scheduling.
[14:04:03] <inflatador>	 Ben is in process of draining the existing hosts
[14:04:27] <inflatador>	 should free up some capacity for semantic-search shortly
[14:07:40] <ebernhardson>	 thanks!
[14:08:19] <ebernhardson>	 indeed we are up to 7 nodes now, so it's making progress!
[14:08:56] <ebernhardson>	 dcausse:  is best bet for getting the indices on the new cluster to re-run the spark scripts with a new endpoint?
[14:11:41] <dcausse>	 ebernhardson: yes should be the simplest approach, in stat1009.eqiad.wmnet:~dcausse/disco there should be a bunch of bash scripts
[14:11:54] <dcausse>	 forgot how I named them looking
[14:13:55] <dcausse>	 the import scripts should be "import_$dc_$lan.sh" or something close, most might still target last week snapshot but could be updated to 20260920
[14:14:09] <ebernhardson>	 ok, thanks! doesn't sound too bad
[14:14:46] <dcausse>	 tho, I just copy/pasted the cmd line + env var from airflow logs (stripping the keytab config args)
[14:24:42] <ebernhardson>	 we have 16 nodes, cluster came up! it's only 256g memory, but starting somewhere
[14:26:37] <dcausse>	 nice!
[14:34:43] <dcausse>	 ok finally restarting the big opensearch-semantic clusters to pick-up the prometheus-exporter
[14:38:17] <ebernhardson>	 oh...duh. i probably have to get promethues into this new cluster as well
[14:38:58] <dcausse>	 ah my bad, should have bumped this namespace too
[14:39:20] <ebernhardson>	 no worries! i read the emails...and just didn't think of it
[15:36:32] <ebernhardson>	 dcausse: can you `chmod -R a+r ~/disco` ?  There are some files only readable by you.  Was trying to recreate, the venv looks like it might be a mix of conda and pip so recreating has not been working for me
[15:36:36] <ebernhardson>	 on stat1009
[15:36:47] <dcausse>	 ebernhardson: sure
[15:37:23] <ebernhardson>	 thx!
[15:37:23] <dcausse>	 meh chmod: changing permissions of '/home/dcausse/disco/venv/lib/python3.10/site-packages/_distutils_hack/__pycache__/__init__.cpython-310.pyc': Operation not permitted
[15:37:32] <ebernhardson>	 :S
[15:37:47] <ebernhardson>	 i've never seen this before, where a venv has random permissions
[15:37:49] <dcausse>	 ah
[15:38:17] <dcausse>	 I think I ran some with sudo -u analytics-search, so possibly python compiled some stuf?
[15:38:51] <dcausse>	 ah it's from you
[15:38:52] <ebernhardson>	 yea perhaps, hmm
[15:39:18] <dcausse>	 already world readable anyways
[15:39:45] <ebernhardson>	 looks like i could read and copy it all now, thanks!
[15:40:00] <dcausse>	 np!
[15:44:52] <dcausse>	 inflatador: is it ok to change some stuf in the opensearch-on-k8s dashboard? (trying to adapt few graphs to avoid having duplicated time series for a same node when pod restarts)
[15:45:47] <inflatador>	 dcausse sure, feel free to make any changes you need
[15:46:00] <dcausse>	 thanks, doing
[15:49:52] <ebernhardson>	 inflatador: i poked over the wikitech opensearch-on-k8s but not finding, how do i get opensearch-semantic-search-ssd.svc.eqiad.wmnet to exist as dns/routing?
[15:55:10] <ebernhardson>	 looks like might need an operations/dns patch, i can prep that, but not sure if more is needed
[15:55:10] <inflatador>	 ebernhardson oops, let me see if I can get those docs added. I think it will be a change similar to https://gerrit.wikimedia.org/r/c/operations/dns/+/1237236 and probably a puppet change
[15:59:26] <ebernhardson>	 I think https://gerrit.wikimedia.org/r/c/operations/dns/+/1345160 is sufficient, it skips codfw and dns discovery, i doubt we need them
[16:05:13] <inflatador>	 Nice! I think we will need a Puppet patch for https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/common/profile/services_proxy/envoy.yaml and https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/hieradata/common/service.yaml , I'll get started on that
[16:11:36] <dcausse>	 ok metrics are back
[16:11:43] <dcausse>	 heading out, have a nice week-end
[16:13:28] <inflatador>	 .o/