[10:16:36] Unless someone else really want to do it, I'll be moving the deployment server to codfw this afternoon. I will be complaining that having deployment.eqiad.wmnet point to a codfw host is rather misleading [10:25:00] Raine recently complained about this too - anything we could do there? [10:25:40] (and thanks Simon for moving the deployment server) [10:27:28] matthieulec: change a lot of repos :D https://codesearch.wmcloud.org/search/?q=deployment.eqiad.wmnet&files=&excludeFiles=&repos= [10:28:21] and wikitech and probably all deployers have that hardcoded in their ssh config too [10:28:38] https://wikitech.wikimedia.org/wiki/deployment.eqiad.wmnet [11:09:28] We'd have to start by finding a better name, but I'm not sure that using deployment.wikimedia.org is what we want. I'm guessing it's on the .wmnet for a reason [11:10:03] But find a new name and just. have deployment.eqiad.wmnet as a CNAME for the new one? [11:26:40] <_joe_> yeah we definitely don't want the deployment servers have a public IP [14:10:03] o/ is there anything that would cause connection timeouts to config-master.discovery.wmnet from eqiad (config = zarcillo-aux-k8s-eqiad.config)? [14:15:27] I'm just waiting for the deployers to finish up and then I'll move to the codfw deployment server [14:32:09] The deployments took a bit longer than expect and there might be more in the queue. I'll move the deployment server tomorrow morning (EU time) where nothing is scheduled [15:34:05] apergos: thanks for looking at T440319 , what do you mean by "Beta is not fully back" - can we help with anything? [15:37:53] cezmunsta: config-master is not serviceops so idk [15:38:55] claime: to clarify, it was more about the ability to reach it from the pod, it has started erroring in the past few days [15:38:59] Hey, I'm trying to execute `mw-script` on deploy1003 and it's failing with `Job.batch "mw-script.codfw.2pi2hrhr" is invalid: [spec.template.spec.containers[1].restartPolicy: Forbidden: may not be set for non-init containers, spec.template.spec.containers[2].restartPolicy: Forbidden: may not be set for non-init containers]` [15:39:35] bjensen: ^ [15:39:50] sorry, the command I meant is `mwscript-k8s` [15:39:58] ah, apologies, taking a look now [15:42:03] I mean that deployment-prep is not restored to regular use yet. matthieulec [15:42:39] bjensen: do you want entire stack trace? I want to run fixStuckGlobalRename [15:42:48] that is where I would be trying to do some testing of jobrunner, changeprop and etc. [15:43:46] pmiazga: no worries, i'm pretty confident of the bits that need flipping [15:44:25] cezmunsta: Are you using config-master.discovery.wmnet in your application config? Because your egress policy only opens the port to config-master1001.eqiad.wmnet [15:44:53] so since the switchover, config-master.discovery.wmnet points to config-master2001.codfw.wmnet [15:46:19] \o I am cleaning up an old Lift Wing service and one of the REST GW rewrite rules can be removed: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1360449 A review would be great, TIA! [15:47:06] klausman: stampstamp [15:47:44] claime: that sounds like a likely explanation and the one that I was expecting ... it is "inherited" code ... and I was just trying to get to the bottom of the error. Do you think that it is better to restrict to the FQDN for the DC (which then has the issue of decom/replace or to hit the public endpoint? [15:48:23] cezmunsta: just add the egress for codfw [15:48:35] pmiazga: https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1360457 ought to fix this, so if you don't mind trying again after that's merged, i'd appreciate it [15:49:03] cezmunsta: and keep using config-master.discovery.wmnet [15:49:09] sure, no worries - let me know when to test [15:49:32] cezmunsta: best would maybe be to add it to external services and use that at some point [15:50:14] pmiazga: should be good to test now, thanks! [15:51:56] bjensen - still fails, different error now. [15:52:13] ```Job.batch "mw-script.codfw.pz4jazwk" is invalid: [spec.template.spec.initContainers[0].lifecycle: Forbidden: may not be set for init containers without restartPolicy=Always, spec.template.spec.initContainers[0].readinessProbe: Forbidden: may not be set for init containers without restartPolicy=Always, spec.template.spec.initContainers[1].lifecycle: Forbidden: may not be set for init containers without [15:52:13] restartPolicy=Always]``` [15:52:39] claime: ack, thanks for the info - I will take a look [15:52:51] now it says it `may not be set for init containers without restartPolicy`, before it was `Forbidden: may not be set for non-init containers` [15:52:55] hm, suspect i missed something, but am not sure precisely what, would it be possible to get the whole trace? [15:53:18] cezmunsta: 10.192.0.15 is config-master2001.codfw.wmnet [15:54:03] bjensen: you should get the same result running mwscript-k8s -- Version.php --wiki=enwiki [15:55:51] claime: mw-script runs local mcrouter? [15:57:55] bjensen: shouldn't afaik, rzl? [15:58:40] The LW/article-desc change has been merged and pushed. Thanks again! [16:00:33] the other thing that could be tripping me up is helmfile.d/services/mw-script/values-mediawiki-common.yaml overriding values.yaml in that same directory, if that's happening? [16:00:48] perplexing [16:09:06] i think the latter thing is what's happening [16:10:51] sorry missed that -- yeah I expect mw-script to use the shared mcrouter, but `describe pod` is all I would be doing to check [16:23:43] I just did `mwscript-k8s -- Version.php --wiki=enwiki` -> same error `Job.batch "mw-script.codfw.1c46z1bf" is invalid: [spec.template.spec.initContainers[0].lifecycle: Forbidden: may not be set for init containers without restartPolicy=Always, spec.template.spec.initContainers[0].readinessProbe: Forbidden: may not be set for init containers without restartPolicy=Always, spec.template.spec.initContainers[1].lifecycle: [16:23:43] Forbidden: may not be set for init containers without restartPolicy=Always]` [16:30:37] right, the same problem applies to all mw-script jobs at the moment [16:31:03] i'm trying to work out how to fix the config such that we no longer attempt to create initContainers for these jobs [16:31:40] (and instead create regular containers, as we did before a change this morning for most deployments) [16:35:49] problem is this error output isn't being forthcoming about _which_ container is misconfigured, which is making things difficult [16:43:21] ah, looks like claime's found it, we should have a fix ready shortly [16:53:20] pmiazga: okay, we should be good to test again [16:59:13] sec, in the meeting [17:03:15] the Version call worked, let me run the fixStuckGlobalRename now [17:05:55] and worked. Thank you bjensen [17:06:13] no worries, thanks!