[05:43:19] <kart_>	 [2025-07-10 05:39:27] s3cfg written to models/.s3cfg
[05:43:19] <kart_>	 ERROR: Configuration file not available.
[05:43:19] <kart_>	 ERROR: models/.s3cfg: None
[05:43:19] <kart_>	 Seems s3cmd still not able to find the config.
[06:07:56] <kart_>	 It happens same with local setup and with --debug it is parsing the file and still throwing up ERROR.
[06:10:43] <elukey>	 kart_: o/ do you have the logs by any chance? It would be good to see them
[06:14:59] <kart_>	 Sorry, didn't see the message while I was updating the update on the task
[06:15:04] <kart_>	 Same as above error.
[06:15:26] <kart_>	 I've put it to https://phabricator.wikimedia.org/T335491#10990368 as well to update the status.
[06:31:27] <kart_>	 elukey: it seems usual behavior with s3cmd elsewhere? Can you give similar usecase where we are using s3cmd?
[06:44:21] <elukey>	 back sorry
[06:44:35] <elukey>	 I am unable to repro though, at least on stat1011
[06:44:54] <elukey>	 when you say "local" what do you mean? 
[06:45:07] <elukey>	 we don't use s3cmd elsewhere that I know
[06:45:15] <elukey>	 most of the time it is a python-based lib
[06:48:07] <elukey>	 also I found https://github.com/s3tools/s3cmd/issues/903 that it is very interesting
[06:48:36] <elukey>	 the ERROR: .s3cfg: None may mean that there is something in the config file that the parser doesn't like
[06:51:50] <kart_>	 ah
[06:52:12] <kart_>	 local = tested locally to check if s3cmd is getting config file.
[06:54:20] <kart_>	 seems key/secret not parsing properly? It should be.
[06:56:07] <isaranto>	 o/ good morning
[06:56:58] <kart_>	 elukey: stat1001 with curren s3cfg works fine? In that case, we cam check if there is any issue with authentication
[06:57:10] <kart_>	 *current
[06:57:41] <kart_>	 Also, we can supress false output. It is just noise.
[06:59:13] <elukey>	 I currently get 403 access denied from stat1011
[06:59:19] <elukey>	 but IIRC Tobias was able to make it work
[07:00:55] <elukey>	 ok nono with s3://wmf-ml-models/ it works
[07:02:20] <elukey>	 even with the env variables etc..
[07:03:29] <kart_>	 so, file is there and it is correct.
[07:03:37] <kart_>	 the ERROR is misleading.
[07:05:20] <elukey>	 kart_: ok I was able to repro on stat1011.. if I don't export any AWS_ variable, and try to execute the command, I get the same failure
[07:05:48] <elukey>	 so for some reason, the AWS variables are not available when entrypoint.sh runs
[07:06:46] <kart_>	 Interesting 
[07:09:36] <elukey>	 can you try to redeploy in staging so i can inspect the pod?
[07:10:04] <kart_>	 sure
[07:10:38] <kart_>	 started elukey 
[07:11:26] <elukey>	 so kubectl describe pod machinetranslation-staging-5bb5449d48-dgvsx -n machinetranslation | grep AWS 
[07:11:31] <elukey>	 returns empty :D
[07:12:14] <kart_>	 :/
[07:12:52] <elukey>	 the secret is there
[07:14:43] <elukey>	 ahh wait I may know the issue
[07:16:56] <elukey>	 yeah ok 
[07:17:13] <elukey>	 so you have a custom version of deployment.yaml in the machinetranslation chart
[07:17:53] <elukey>	 meanwhile "app.generic.container" (under the vendor templates) is the snippet that defines the inclusion of the env variables
[07:18:01] <elukey>	 but you don't use it, so they never get rendered from the chart
[07:22:13] <kart_>	 Do we need  {{- include "app.generic.container" . | indent 8 }} that's all? or some more magic?
[07:22:38] <elukey>	 kart_: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1167742
[07:22:57] <elukey>	 the include is a bit big and it applies to the definition of the whole container
[07:23:17] <elukey>	 I'd suggest to refactor the config later on, but for the moment the one that I posted should be sufficient
[07:23:23] <kart_>	 ah. Thanks.
[07:23:47] <elukey>	 of course CI failed :D
[07:24:12] <kart_>	 :/
[07:29:02] <kart_>	 Be back after the lunch.
[07:30:58] <elukey>	 kart_: fixed!
[07:31:11] <elukey>	 if you have a moment to review I can then merge/deploy
[07:36:55] <elukey>	 it is super simple and I self-merged
[07:36:58] <elukey>	 will test it in a bit
[07:45:09] <klausman>	 Morning!
[07:46:36] <elukey>	 [2025-07-10 07:45:45] Extracting models/nllb200-600M.tgz into models/nllb200-600M
[07:47:49] <elukey>	 kart_: worked! staging is up :)
[07:48:00] <elukey>	 morning :)
[07:53:44] <klausman>	 very nice! how long did the startup take?
[07:57:28] <aiko>	 good morning
[07:57:39] <klausman>	 Morning, Aiko!
[07:58:14] <aiko>	 o/
[07:59:02] <kart_>	 elukey: awesome. Thanks!
[08:02:40] <kart_>	 We can plan next deployment after watching log for sometime.
[08:03:52] <aiko>	 klausman: should we proceed with depooling codfw in prod?
[08:04:16] <klausman>	 Yes, I will do that in a moment
[08:05:22] <aiko>	 ack!
[08:08:03] <klausman>	 $ confctl --object-type discovery select 'dnsdisc=inference,name=codfw' get
[08:08:05] <klausman>	 {"codfw": {"pooled": false, "references": [], "ttl": 300}, "tags": "dnsdisc=inference"}
[08:08:11] <klausman>	 codfw should be depooled
[08:10:21] <kart_>	 klausman: re startup, we might need to remove liveness_probe: part from values-staging.yaml and test?
[08:11:11] <klausman>	 you mean to see if the default values are good enough?
[08:21:07] <aiko>	 klausman: q - after you apply the knative change in admin_ng, and all the pods reschedule, should/can we run httpbb tests to vefrify everything is working fine? will the models be accessible when codfw is depooled?
[08:23:24] <klausman>	 We should try it either way. If it works: great, we have successfully verified it; All fail: well, that's depooled; _some_ fail: we need to investigate
[08:25:05] <elukey>	 aiko: you can definitely run httpbb, just use as endpoint inference.svc.codfw.wmnet instead of inference.discovery.wmnet
[08:25:34] <klausman>	 ah, yes, the site-specific disco endpoint should still work
[08:26:15] <klausman>	 https://grafana.wikimedia.org/goto/wEQvo9sHR?orgId=1 looks like all traffic has drained (the remainder is likely just monitoring)
[08:28:15] <klausman>	 elukey, aiko: https://phabricator.wikimedia.org/P78862 diff for ml-serve-codfw (also some external-services stuff that I haven't pastebinned)
[08:28:19] <aiko>	 I see! thank both for the answers :)
[08:29:13] <klausman>	 I will apply the adminng stuff in a few moments unless someone stops me :)
[08:30:23] <klausman>	 here we go
[08:30:56] <kart_>	 klausman: yes.
[08:31:14] <klausman>	 kart_: yeah, that sounds like a good idea.
[08:31:42] <kart_>	 only for staging, we've.
[08:31:42] <kart_>	   initialDelaySeconds: 15
[08:31:42] <kart_>	   periodSeconds: 10
[08:31:42] <kart_>	   failureThreshold: 6
[08:35:53] <klausman>	 aiko: all pods have been restarted (well, except the system pods and ores-legacy since they don't use this image)
[08:37:04] <aiko>	 ack! let me check if they are all in running status
[08:41:34] <aiko>	 recommendation-api-ng doesn't use that image either, so it's not been restarted
[08:41:43] <klausman>	 good point
[08:43:59] <elukey>	 bartosz: really great work on the script!
[08:44:47] <aiko>	 nice looks like all pods are in good state
[08:45:11] <aiko>	 gonna run httpbb tests
[08:45:25] <klausman>	 ack
[08:48:42] <aiko>	 https://www.irccloud.com/pastebin/VypzcECm/
[08:49:05] <klausman>	 \o/
[08:49:58] <bartosz>	 o/ elukey: thank you! and additional thanks for the review <3 
[08:51:34] <aiko>	 \o/ we can start deploying the credential changes
[08:52:05] <klausman>	 let me run a scripted diff across everything in the ml-services subdir
[08:53:21] <aiko>	 nice!
[08:53:39] <klausman>	 Ok, looks good, besides a an oodity: https://phabricator.wikimedia.org/P78864
[08:54:01] <klausman>	 I suspect that while recapi does not use the changed secrets, they're still part of its env
[08:56:10] <klausman>	 I'll start applying the change one NS at a time, starting with article-descriptions
[08:56:23] <aiko>	 they might also use s3 to get the embedding in swift? https://analytics.wikimedia.org/published/wmf-ml-models/recommendation-api/
[08:56:32] <klausman>	 ah, right
[08:57:11] <klausman>	 ok, changed the secrets on art-desc, bouncing the service
[08:58:25] <klausman>	 INFO:root:Successfully copied s3://wmf-ml-models/article-descriptions/ to /mnt/models
[08:58:27] <klausman>	 \o/
[08:59:46] <klausman>	 hmmm. the pod is still in "Initializing" state
[09:00:00] <klausman>	 and just as I type that it goes to Running :D
[09:00:13] <aiko>	 yes now it's running 2/3 
[09:00:21] <klausman>	 ... and 3/3
[09:00:30] <aiko>	 \o/
[09:00:43] <klausman>	 can you run the httpbb test against that service?
[09:01:31] <aiko>	 doing it
[09:01:34] <aiko>	 passed :)
[09:01:56] <klausman>	 excellent
[09:02:05] <klausman>	 continuing with the art-models NS
[09:02:29] <klausman>	 secrets applyied, bouncing art-country
[09:03:30] <klausman>	 art-country is startsed and running
[09:05:31] <klausman>	 art-quality also restarted and running. can you test them?
[09:05:48] <klausman>	 (after this, I'll proceed with the rest of the NSes and we cna then test them all at the end)
[09:09:56] <aiko>	 mm just found out we don't have test_article-models.yaml on deployment node?  https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/profile/files/httpbb/liftwing/production/test_article-models.yaml
[09:10:14] <klausman>	 Interesting
[09:10:20] <aiko>	 not under /srv/deployment/httpbb-tests/liftwing/production/
[09:11:22] <klausman>	 Since that dir is not a git repo, I suspect the file we have in puppet is not wired up to be distributed
[09:12:10] <klausman>	 Yep, missing from modules/profile/manifests/httpbb.pp
[09:13:15] <klausman>	 making a patrch
[09:14:51] <klausman>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1167827
[09:16:39] <aiko>	 thanks for the patch! +1
[09:16:49] <klausman>	 also missing for staging, adding that now
[09:18:21] <klausman>	 Hmmm. Logo Detection, Rec-API are also not wired up. 
[09:19:45] <aiko>	 ahh
[09:20:27] <aiko>	 re: next deployment, I think we can deploy revscoring-editquality-goodfaith NS and revscoring-editquality-damaging NS individually because they have quite a lot of models, and then we can do the rest at once. wdyt?
[09:21:03] <klausman>	 sure, that works
[09:22:07] <klausman>	 Updated the patch to add the missing tests
[09:25:35] <aiko>	 thanksss!
[09:28:36] <aiko>	 this probably needs to be noted in our liftwing doc.. when we add new httpbb tests, also need to update that modules/profile/manifests/httpbb.pp
[09:29:36] <klausman>	 yes, agreed
[09:33:27] <isaranto>	 klausman: could you also merge this patch since I don't have +2 rights? (httpbb tests for edit check) https://gerrit.wikimedia.org/r/c/operations/puppet/+/1149634
[09:33:59] <klausman>	 That patch is missing the very bit I added above :)
[09:35:31] <klausman>	 Added a comment that explains it
[09:37:42] <klausman>	 aiko: the article-modules (and rec-api+logo-detect) tests are now available on the deployment machine, can you retry the test?
[09:38:29] <aiko>	 yes! just tested, and passed :)
[09:38:37] <klausman>	 excellent, thank you.
[09:38:47] <klausman>	 Will now push the change to revscoring-editquality-goodfaith
[09:39:03] <aiko>	 ack
[09:39:28] <isaranto>	 oh got it, thanks! will add that
[09:40:12] <klausman>	 It's very clearly easy to miss, as we all did :)
[09:40:34] <klausman>	 I'll now bounce the pods in revscoring-editquality-goodfaith
[09:45:30] <klausman>	 ok, all pods restarted, you can commence tests
[09:46:24] <aiko>	 cool! doing it
[09:47:07] <aiko>	 PASS: 35 requests sent to inference.svc.codfw.wmnet. All assertions passed. \o/
[09:47:22] <klausman>	 excellent. proceeding with rs-eq-damaging
[09:51:24] <klausman>	 config pushed and all pods bounced
[09:51:37] <aiko>	 niceee
[09:52:26] <aiko>	 PASS: 35 requests sent to inference.svc.codfw.wmnet. All assertions passed! 
[09:52:54] <klausman>	 great. I'm gonna run do a few errands and get lunch, and when I'm back we can proceed with the other NSes
[09:53:31] <aiko>	 no problem! sounds good
[09:54:08] <klausman>	 I'll ping when I'm back
[09:56:54] <aiko>	 alrighty
[10:02:56] * aiko get lunch too :)
[10:07:53] <isaranto>	 I added the missing bit in the edit check httpbb patch https://gerrit.wikimedia.org/r/c/operations/puppet/+/1149634
[11:57:37] * aiko back
[12:23:38] * klausman back as well
[12:24:30] <klausman>	 aiko: any preference regarding the next NS to do?
[12:27:52] <aiko>	 maybe the rest of revscoring-NS?
[12:28:00] <klausman>	 yeah, sgtm.
[12:28:28] <klausman>	 If we do them alphabetically, rs-articlequality is next
[12:28:43] <aiko>	 ack!
[12:29:53] <klausman>	 config pushed, bouncing pods
[12:30:57] <klausman>	 and they're all back
[12:32:24] <aiko>	 niceee
[12:34:27] <kart_>	 elukey: Should we file a task about updating deployment.yaml for machinetranslation/MinT?
[12:36:07] <aiko>	 then we have revscoring-articletopic, revscoring-draftquality, revscoring-drafttopic, and revscoring-editquality-reverted
[12:36:10] <elukey>	 kart_: if you want yes, it is not strictly needed at the moment but it may be good
[12:36:20] <klausman>	 aiko: I can do them all in one go
[12:36:49] <aiko>	 that'd be great, and I'll test them all when they're up running
[12:36:56] <klausman>	 wfm
[12:38:51] <kart_>	 elukey: Can you do that, I'll miss the background/context. Maybe low priority as of now.
[12:44:20] <klausman>	 aiko: all RS pods done
[12:45:43] <aiko>	 ok! running httpbb tests
[12:48:05] <aiko>	 greattt, all passed :)
[12:48:24] <klausman>	 Good. I'll do all of the following NSes now: articletopic-outlink  experimental llm logo-detection readability recommendation-api-ng revertrisk revision-models
[12:48:53] <klausman>	 edit-check has an unrelated diff, so I'll coordinate with Ilias on that one. and the other NSes should have no diff anyway
[12:49:23] <aiko>	 edit-check one is mine
[12:49:37] <aiko>	 I merged https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1167565
[12:49:37] <klausman>	 ah, right
[12:50:26] <aiko>	 there are no services in experimental NS on codfw
[12:50:27] <klausman>	 Should I do that, then?
[12:50:53] <klausman>	 I'll still push the cred update on experimental, ther just won't be pods to bounce
[12:51:05] <klausman>	 ("that" above referring to editcheck)
[12:51:27] <aiko>	 yeah we can do that, I just tested in staging 
[12:51:55] <klausman>	 roger
[12:52:01] <aiko>	 re experimental: got it!
[12:58:12] <aiko>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/1167858 and the httpbb test for edit-check needs to be updated.. whenever you have a moment
[12:58:59] <klausman>	 +1'd for now. will +2 and merge once I'm done with the current set of services
[12:59:03] <elukey>	 aiko you have a future in SRE :D
[12:59:59] <aiko>	 really!!! :D
[13:01:12] <klausman>	 I agree with Luca
[13:01:51] <aiko>	 that's a big compliment, thank u <3
[13:02:48] <klausman>	 well deserved!
[13:05:26] <klausman>	 All NSes updated and pods bounced, did another pass to see if there are any NSes with diffs, and there are none
[13:06:48] <aiko>	 ack, gonna run tests
[13:11:54] <aiko>	 only recommendation-api-ng fails, all others passed 
[13:11:57] <aiko>	 looking
[13:15:23] <aiko>	 ah the host for recommendation-api-ng should be different right? 
[13:15:29] <aiko>	 error msg: https://phabricator.wikimedia.org/P78877
[13:17:23] <klausman>	 Probably, I am not sure
[13:19:02] <aiko>	 I don't get why the host header is https://recommendation-api-ng.discovery.wmnet:31443
[13:22:01] <aiko>	 I know this service works different from our other isvc, but I'm not so familiar with how it works
[13:23:23] <aiko>	 let me investigate 
[13:23:29] <klausman>	 yeaI'll also dig a bit
[13:36:45] <klausman>	 I think this is because codfw is still drained.
[13:38:21] <aiko>	 so the site-specific endpoint doesn't work for recommendation-api-ng
[13:38:28] <klausman>	 Or rather: the inference endpoint.... yes
[13:39:35] <aiko>	 yeah it doesn't use the inference endpoint, because it's not based on kserve
[13:39:43] <klausman>	 curl -vv 'https://recommendation-api-ng.svc.codfw.wmnet:31443/service/lw/recommendation/api/v1/translation' works 9sorta, it's missing query parms)
[13:40:01] <aiko>	 I saw a query example here https://phabricator.wikimedia.org/T371465#10080149
[13:40:05] <aiko>	 time curl "https://recommendation-api-ng.discovery.wmnet:31443/service/lw/recommendation/api/v1/translation?source=en&target=fr&count=3&seed=Apple"
[13:41:27] <klausman>	  httpbb test_recommendation-api-ng.yaml --hosts recommendation-api-ng.svc.codfw.wmnet --https_port 31443 --insecure <- this also sorta works, but has the wrong aprams
[13:41:40] <klausman>	 (--insecure gets around the SSL cert failure)
[13:42:12] <klausman>	 How about I re-pool codfw, we quickly test and if it still doesn't work, I depool again and we continue investigating?
[13:45:13] <aiko>	 https://www.irccloud.com/pastebin/mOc8mUFT/
[13:45:15] <aiko>	 this works 
[13:45:42] <klausman>	 Well, then I'd mark the service as working and we re-pool?
[13:46:00] <klausman>	 and if httpbb still doesn't work afterwards, the test is broken, not the service
[13:46:40] <aiko>	 yes, we need to fix the httpbb test
[13:47:23] <klausman>	 pooling now
[13:47:41] <klausman>	 done
[13:50:39] <aiko>	 \o/ great, thank you!!
[13:51:05] <klausman>	 I'll keep an eye on traffic #s and eror codes
[13:54:01] <elukey>	 nice work folks!
[13:57:45] <klausman>	 ty <3
[14:02:26] <elukey>	 is eqiad going to be done next week?
[14:04:28] <wikibugs>	 06Machine-Learning-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install ml-serve101[2345] - https://phabricator.wikimedia.org/T393948#10992180 (10Jclark-ctr) ml-serve1015 is now racked into E 12  and added to netbox   @elukey Let me know when you’re finished with any testing you want to do. I’ll stay...
[15:02:40] <wikibugs>	 06Machine-Learning-Team: Spark Job in airflow-devenv cannot access Hive Metastore because of Kerberos Authentication Failure - https://phabricator.wikimedia.org/T398907#10992522 (10kevinbazira) Thanks to @brouberol, who has been super helpful when resolving WMF Airflow issues. He also provided clarity on where o...
[15:03:15] <kart_>	 klausman: re staging start and end time for downloads, Start: Jul 10, 2025 @ 07:45:00, End: Jul 10, 2025 @ 07:47:02
[15:03:31] <aiko>	 elukey: I think that'd be the plan, as Friday is not ideal for deployment 
[15:04:51] <elukey>	 aiko: you already talk like an SRE!
[15:05:13] <aiko>	 lollll
[15:06:18] <elukey>	 isaranto: we should stop looking for the new SRE
[15:07:14] <isaranto>	 aiko telling luca not to deploy on fridays :D go aikooooooooooo
[19:36:52] <wikibugs>	 06Machine-Learning-Team: Spark Job in airflow-devenv cannot access Hive Metastore because of Kerberos Authentication Failure - https://phabricator.wikimedia.org/T398907#10993364 (10brouberol) I have merged a change into `airflow-dags` (see https://phabricator.wikimedia.org/T394297#10993349) that //should// resol...