[00:29:28] (03PS1) 10Zoranzoki21: [zuul/parameter_functions.py] Add param_dependencies for MassMessageEmail [integration/config] - 10https://gerrit.wikimedia.org/r/715634 [00:29:42] (03CR) 10Zoranzoki21: "I did this for https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MassMessageEmail/+/694741" [integration/config] - 10https://gerrit.wikimedia.org/r/715634 (owner: 10Zoranzoki21) [01:01:43] PHPUnit sucks. Maybe we should fork it. [01:07:31] my problem is with the TestSuite class. Most of the problems with PHPUnit radiate out from TestSuite [01:09:16] I was trying to figure out yesterday why I can't run CI-like extension tests on my laptop without it panic swapping [01:09:56] the main problem, I believe, is the fact that all Test instances, i.e. every test case, is instantiated at startup and remains live until the end [01:10:26] it starts off with 1.6GB of memory and each object gets a bit larger when it is run [01:11:47] TimStarling: This, incidentally, is also why --filter or --group results in spinning wheels for a few minutes before any test starts running [01:12:03] since it's loading all test suites and expanding all data providers [01:12:14] before applying the filter or group [01:12:51] That's with just core and one or two extensions, so it does eventually begin [01:13:43] * Krinkle recently re-discovered this when rewriting https://www.mediawiki.org/wiki/Manual:PHP_unit_testing/Running_the_tests#Select_by_group [01:14:42] --filter is an expensive feature, there's not a really cheap way to implement matching against the results of all data providers [01:16:52] hm.. I didn't know it matched againt the (keys?) of data providers as well. [01:16:58] I thoguht it was class name and test name. [01:17:21] Althoguh even without --filter it seems to load and prepare all cases before execution begins [01:17:47] that's among the thing that seems to make timestamp stuff fail often locally since a lot of time elapses between provider and test. [01:17:49] it matches the test name which includes the name of the dataset, which is the key from the data provider [01:18:44] yes it loads everything, that's what I'm saying, gigabytes of memory used with --filter=nonexistent [01:19:05] (ofc that should be fixed regardless by mocking the time, but people only notice that when it fails and most tests are rarely run locally and thus take months before they fail in CI often enough for somene to notice/understand/fix) [01:19:29] also Sebastian is annoying and has blocked all sorts of useful additions [01:19:29] It seems like at the very least it could start exeecuting stuff that is included esp when no filters are applied. [01:19:38] but maybe there's another edge case for why that can't happen [01:19:40] like parallel execution [01:19:43] right [01:20:16] he deprecated and removed an assert method because people kept filing bugs about it, misunderstanding its purpose [01:20:23] if https://phabricator.wikimedia.org/T225730 ever gets funded, Aaron would likely work on integrating paratest for our CI [01:20:35] so he was like "if you don't understand it, you don't deserve it" [01:21:05] he's got a proof of concept that made all MW core tests run in under 5min locally, and that's on Windows /WSL. [01:23:01] 10Continuous-Integration-Config, 10Release-Engineering-Team (Next), 10MediaWiki-Core-Tests, 10Code-Health, and 6 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Krinkle) [01:23:33] 10Continuous-Integration-Config, 10Release-Engineering-Team (Next), 10MediaWiki-Core-Tests, 10Code-Health, and 6 others: Reduce runtime of MW shared gate Jenkins jobs to 5 min - https://phabricator.wikimedia.org/T225730 (10Krinkle) [01:25:45] memory usage might be an issue for that too [01:27:12] right, it would not reduce memory per-se indeed. I don't think it would increase it much per-se though, that is, the memory for the test cases would still be allocated for all, but not for the same one in multiple procs, assuming paratext uses the suites/directories to specify upfront what each proc loads. [01:27:13] are there actual "new" features in PHPUnit we're looking to? honestly each PHPUnit upgrade just seems like work rather than actual new features, forking would mean we aren't actually losing that much [01:27:26] though there's added general PHP memory and the shared db tables etc [01:27:58] you'd be forking probably the most used package on packagist though [01:28:52] https://packagist.org/explore/popular suggests the most used package is psr/log [01:29:03] :) [01:29:39] that page has interesting sorting [01:30:08] (I feel like Tim and I have discussed forking PHPUnit every so often after the PHPUnit RCE popped up) [01:30:45] yeah I imagine there's a few small packages that phpunit itself uses, and then some untested projects using that perhaps directly. I can't recall another unit test package even existing though. [01:31:21] I am continually surprised that there is no major competitor, I feel most other languages have multiple semi-competing unit test runners [01:31:59] e.g. in Python there's the built-in unittest, nosetests and pytest [01:32:28] there are a bunch of libraries, although the assertions/constraints are not a library and that's probably the most useful thing in it [01:33:42] the assertion classes are marked as stable though so you can just make the new thing depend on phpunit/phpunit [03:42:53] argghurgle [03:43:29] static data provider methods are faster [03:43:45] non-static data provider methods pointlessly create the object and throw it away [05:17:18] 10Project-Admins, 10User-Urbanecm: Create User-IN project for IN / Q28 - https://phabricator.wikimedia.org/T289915 (10IN) >>! In T289915#7316910, @Aklapper wrote: > The user IN has *created* 18 tasks so far. There are also exactly these 18 tasks tagged with #User-IN now. > That means https://phabricator.wikim... [05:56:40] James_F: I'm happy to help with it, but I'd rather not take the lead [06:13:38] 10Project-Admins, 10User-IN: change tag #user-IN to workboard - https://phabricator.wikimedia.org/T290052 (10IN) [06:47:56] 10Project-Admins, 10User-IN: change tag #user-IN to workboard - https://phabricator.wikimedia.org/T290052 (10Aklapper) 05Open→03Invalid "Labels" do not exist, and this ticket makes no sense. Please do stop filing nonsense tickets if you would like to be active here. Please read https://www.mediawiki.org/wi... [06:48:31] 10Project-Admins, 10User-Urbanecm: Create User-IN project for IN / Q28 - https://phabricator.wikimedia.org/T289915 (10Aklapper) > The other is to add calcium tablets to the task I created. I do not know what that is. [07:59:06] (03PS5) 10Jeena Huneidi: WIP Adds kubernetes cluster [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) [08:02:24] (03PS6) 10Jeena Huneidi: WIP Adds kubernetes cluster [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) [08:05:27] (03CR) 10Jeena Huneidi: "Switched to Minikube. lmk if this works for you. It's late so I probably missed something." [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [09:45:40] 10Project-Admins, 10User-IN: change tag #user-IN to workboard - https://phabricator.wikimedia.org/T290052 (10Peachey88) @Aklapper I think there might be some language interchange here causing confusion between the name label/tags/project @IN to enable work boards on your current project, is quite easy, Go int... [09:48:28] 10Project-Admins, 10User-IN: change tag #user-IN to workboard - https://phabricator.wikimedia.org/T290052 (10Aklapper) There might be, and I am asking @IN to read and follow https://www.mediawiki.org/wiki/Phabricator/Help and https://www.mediawiki.org/wiki/Phabricator/Project_management to get acquainted to Ph... [11:11:32] (03CR) 10D3r1ck01: [C: 03+1] "Looks good to me! Thank you!" [integration/config] - 10https://gerrit.wikimedia.org/r/715634 (owner: 10Zoranzoki21) [11:51:28] 10Project-Admins, 10User-Urbanecm: Create User-IN project for IN / Q28 - https://phabricator.wikimedia.org/T289915 (10IN) >>! In T289915#7320851, @Aklapper wrote: >> The other is to add calcium tablets to the task I created. > > I do not know what that is. Sorry, just because I used the translator, and then... [11:55:20] 10Project-Admins, 10User-IN: change tag #user-IN to workboard - https://phabricator.wikimedia.org/T290052 (10IN) >>! In T290052#7321191, @Aklapper wrote: >There have been several unhelpful and invalid Phabricator tickets created lately, and they have created notifications and they bind resources. Thanks. It's... [12:20:44] hi releng, could anyone tell me if logstash1003.eqiad.wmnet is still the current logstash host to be sending logs to? (or if i should be asking in another channel) [12:41:30] nikkinikk_: #wikimedia-observability woukd know for sure. Else the proper host to use is probably defined in puppet config [12:43:21] hashar: great thank you! [12:45:46] hashar: actually i should ask, do you know in which repo i could find the puppet config? 😬 [12:59:39] nikkinikk_: operations/puppet in gerrit [13:01:16] majavah: thank you :) [14:29:45] !log Restarting CI Jenkins for plugins upgrade [14:29:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:51:47] nikkinikk_: I don't know the exact use case, but at the surface it sounds unusual to send logs directly there, I don't know if the use case specifically needs that or whether that might be unintentional. I believe we usually go through local rsyslog at which point the rest gets taken care of by shared processes from puppet or k8s (which then take it to kafka and logstash elastic, asynchronously). [14:52:28] Just FYI in case there's any disconnect. If intended, I'm sure it's all good :) [15:02:33] Krinkle: yep! thats exactly what folks in wikimedia-observability mentioned, that sending logs directly to logstash has been deprecated and to follow https://wikitech.wikimedia.org/wiki/Logstash/Interface#Configuring_rsyslog_to_forward_your_logs if i need to send them without being on k8s [15:03:06] Cool [15:06:58] 10Gerrit, 10Release-Engineering-Team (Doing), 10GitLab, 10Icinga, and 4 others: RelEng access to downtime alerts in Icinga for gitlab, gerrit, possibly other services? - https://phabricator.wikimedia.org/T289746 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Optimistically resolving! Feel free to re... [15:26:58] (03PS1) 10Jbond: operations-puppet: update to pick up change to Gemfile [integration/config] - 10https://gerrit.wikimedia.org/r/715752 [15:27:48] (03CR) 10Hashar: [C: 03+2] [zuul/parameter_functions.py] Add param_dependencies for MassMessageEmail [integration/config] - 10https://gerrit.wikimedia.org/r/715634 (owner: 10Zoranzoki21) [15:29:03] (03Merged) 10jenkins-bot: [zuul/parameter_functions.py] Add param_dependencies for MassMessageEmail [integration/config] - 10https://gerrit.wikimedia.org/r/715634 (owner: 10Zoranzoki21) [15:32:11] (03CR) 10Hashar: "deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/715634 (owner: 10Zoranzoki21) [15:37:22] (03CR) 10Hashar: "That has been done on purpose via:" [extensions/Flow] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/714849 (owner: 10Urbanecm) [15:38:19] (03Abandoned) 10Urbanecm: Do not use exclusive access [extensions/Flow] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/714849 (owner: 10Urbanecm) [15:47:53] 10Release-Engineering-Team (Radar), 10Quality-and-Test-Engineering-Team (QTE), 10serviceops-radar, 10CommRel-Specialists-Support (Jul-Sep-2021), and 2 others: Expand the list of group 1 wikis to contain at least one (preferably 2) smaller "top ten size" wikis - https://phabricator.wikimedia.org/T286664 (10E... [15:51:15] 10Release-Engineering-Team (Radar), 10Quality-and-Test-Engineering-Team (QTE), 10serviceops-radar, 10CommRel-Specialists-Support (Jul-Sep-2021), and 2 others: Expand the list of group 1 wikis to contain at least one (preferably 2) smaller "top ten size" wikis - https://phabricator.wikimedia.org/T286664 (10o... [16:22:43] (03CR) 10Ahmon Dancy: [C: 03+2] Replace regex based variable parser with a lexer [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/708595 (https://phabricator.wikimedia.org/T287507) (owner: 10Dduvall) [16:23:38] (03Merged) 10jenkins-bot: Replace regex based variable parser with a lexer [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/708595 (https://phabricator.wikimedia.org/T287507) (owner: 10Dduvall) [16:26:55] majavah: Ack. Will try to nerd-snipe some others into helping. [16:29:42] (03CR) 10Jforrester: "Neat." [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/708595 (https://phabricator.wikimedia.org/T287507) (owner: 10Dduvall) [16:30:34] How do pipelinelib updates get deployed? [16:31:33] I believe the Jenkins stuff is configured to always use HEAD of the pipelinelib repo. [16:31:36] (i.e., merge to deploy) [16:31:42] Ah, nice. [16:31:55] So if everything starts breaking, you know why [16:31:56] Yeah, was looking for SAL entries for pipelinelib and finding none anywhere. [16:32:06] I'm hoping this fixes our issues. :-) [16:35:38] (03CR) 10Arlolra: [V: 03+2 C: 03+2] "I assume you only went this far because node support changes in v10" [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/715626 (owner: 10Subramanya Sastry) [16:36:11] (03CR) 10Arlolra: [V: 03+2 C: 03+2] More Cite CSS fixes to use custom counter style symbols [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/715627 (owner: 10Subramanya Sastry) [16:36:14] 10Release-Engineering-Team (Doing), 10Release Pipeline, 10Patch-For-Review: java.io.NotSerializableException: groovy.lang.IntRange in Jenkins CI - https://phabricator.wikimedia.org/T287507 (10dduvall) @Jdforrester-WMF the workaround should be deployed already (we pull the plugin from its master branch) if yo... [16:43:38] 10Release-Engineering-Team (Doing), 10Release Pipeline, 10Patch-For-Review: java.io.NotSerializableException: groovy.lang.IntRange in Jenkins CI - https://phabricator.wikimedia.org/T287507 (10Jdforrester-WMF) 05Open→03Resolved >>! In T287507#7322281, @dduvall wrote: > @Jdforrester-WMF the workaround shou... [16:52:28] (03PS7) 10Ahmon Dancy: WIP Adds kubernetes cluster [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [16:59:14] (03CR) 10Ahmon Dancy: "The switch to minikube looks great. A bit simpler (and more familiar)." [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [16:59:46] (03CR) 10Ahmon Dancy: "Patchset 7 tested on Linux and Mac" [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [17:13:31] 10Release-Engineering-Team (Radar), 10Upstream: Upgrade zuul gearman when upstream releases it - https://phabricator.wikimedia.org/T289512 (10thcipriani) [17:41:41] 10Release-Engineering-Team, 10GitLab: Test task to show off phab - https://phabricator.wikimedia.org/T290097 (10thcipriani) [17:45:01] 10Release-Engineering-Team, 10GitLab: Test task to show off phab - https://phabricator.wikimedia.org/T290097 (10thcipriani) 05Open→03Invalid [17:46:13] 10Release-Engineering-Team (Doing), 10GitLab (Initialization), 10User-brennen: Early adoption signup for WMF GitLab - https://phabricator.wikimedia.org/T282842 (10MattCleinman) Android (and possibly iOS) would be interested in investigating an earlier move to GitLab once CI/CD is possible. We have CI/CD work... [17:48:46] 10Continuous-Integration-Infrastructure, 10Browser-Tests, 10Patch-For-Review, 10User-Addshore, 10User-zeljkofilipin: Centrally look for flakey browser tests - https://phabricator.wikimedia.org/T277205 (10Addshore) Data for the 31st August has been added to the sheet https://docs.google.com/spreadsheets/d... [17:49:15] (03CR) 10Subramanya Sastry: Bump puppeteer to 9.1.1 to use a more recent Chrome browser (031 comment) [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/715626 (owner: 10Subramanya Sastry) [18:20:09] 10Release-Engineering-Team: CI build failure for research landing page - https://phabricator.wikimedia.org/T290100 (10Isaac) [18:33:52] (03CR) 10Ahmon Dancy: [C: 03+2] Ensure dependencies from `includes` are present in the CopiesDepGraph [blubber] - 10https://gerrit.wikimedia.org/r/715276 (https://phabricator.wikimedia.org/T289880) (owner: 10Dduvall) [18:38:02] (03Merged) 10jenkins-bot: Ensure dependencies from `includes` are present in the CopiesDepGraph [blubber] - 10https://gerrit.wikimedia.org/r/715276 (https://phabricator.wikimedia.org/T289880) (owner: 10Dduvall) [18:53:10] (03PS8) 10Ahmon Dancy: WIP Adds kubernetes cluster [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [19:01:43] 10Gerrit, 10User-Zabe: Public labs/private repository is empty - https://phabricator.wikimedia.org/T290102 (10Zabe) [19:15:11] 10Beta-Cluster-Infrastructure, 10Analytics, 10Beta-Cluster-reproducible, 10User-Urbanecm: 502, connect failed for intake-analytics.wikimedia.beta.wmflabs.org - https://phabricator.wikimedia.org/T289029 (10AlexisJazz) >>! In T289029#7319883, @Urbanecm wrote: > So, I tried to just reboot that host, and...the... [19:24:04] 10Gerrit, 10Release-Engineering-Team (Doing), 10GitLab, 10Icinga, and 4 others: RelEng access to downtime alerts in Icinga for gitlab, gerrit, possibly other services? - https://phabricator.wikimedia.org/T289746 (10brennen) Confirmed working for a couple of us, thanks again. [19:25:07] 10Continuous-Integration-Config, 10Sonarqubebot, 10Developer Productivity: SonarQube is unhelpfully suggesting ES6 feautures in ES5 code - https://phabricator.wikimedia.org/T289957 (10DannyS712) Just came to report this myself. Can we just disable suggests for ES6-only features by default? Far more mediawiki... [19:32:52] 10Release-Engineering-Team (Radar), 10Quality-and-Test-Engineering-Team (QTE), 10serviceops-radar, 10CommRel-Specialists-Support (Jul-Sep-2021), and 2 others: Expand the list of group 1 wikis to contain at least one (preferably 2) smaller "top ten size" wikis - https://phabricator.wikimedia.org/T286664 (10T... [19:48:45] (03PS9) 10Ahmon Dancy: WIP Adds kubernetes cluster [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [19:52:37] 10Release-Engineering-Team (Doing), 10GitLab (Initialization), 10User-brennen: Increase GitLab session lifetime to something reasonable - https://phabricator.wikimedia.org/T288757 (10brennen) Something like this works, just needs to run after anything that runs the Chef reconfiguration stuff. Not quite sure... [20:13:04] 10Release-Engineering-Team (Radar), 10Quality-and-Test-Engineering-Team (QTE), 10serviceops-radar, 10CommRel-Specialists-Support (Jul-Sep-2021), and 2 others: Expand the list of group 1 wikis to contain at least one (preferably 2) smaller "top ten size" wikis - https://phabricator.wikimedia.org/T286664 (10J... [20:31:56] 10Release-Engineering-Team (Done by Fri 03 Sep), 10Release, 10Train Deployments: 1.37.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T281162 (10Krinkle) [20:47:49] (03PS10) 10Jeena Huneidi: WIP Adds kubernetes cluster [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) [20:52:01] (03PS11) 10Jeena Huneidi: Adds kubernetes cluster [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) [20:55:49] (03CR) 10Ahmon Dancy: [C: 03+1] "I didn't test patchset 11 but the code looks good to me." [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [20:59:04] 10Gerrit, 10User-Zabe: Public labs/private repository is empty - https://phabricator.wikimedia.org/T290102 (10Legoktm) {F34627027} I think you need to be in Trusted-Users to see the repository, per https://gerrit.wikimedia.org/r/plugins/gitiles/labs/private/+/879eda5dcbd5fe7c926455bdf10fe86df9ced59c Not sure... [21:18:25] 10Gerrit, 10User-Zabe: Public labs/private repository is empty - https://phabricator.wikimedia.org/T290102 (10Zabe) >>! In T290102#7323088, @Legoktm wrote: > {F34627027} > > I think you need to be in Trusted-Users to see the repository, per https://gerrit.wikimedia.org/r/plugins/gitiles/labs/private/+/879eda5... [21:29:28] 10Gerrit, 10User-Zabe: Public labs/private repository is empty - https://phabricator.wikimedia.org/T290102 (10Zabe) Oh and also `Anonymous-Users` can read the repository as I can read it when I go into the 'incoginto mode'. [21:31:55] 10Gerrit, 10User-Zabe: Registered users can't read labs/private while anonymous users can - https://phabricator.wikimedia.org/T290102 (10Zabe) [21:35:25] !log staging new blubberoid release to deploy https://gerrit.wikimedia.org/r/c/blubber/+/715276 [21:35:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:38:25] !log deploying new blubberoid to eqiad/codfw following successful testing in staging [21:38:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:47:49] 10Release-Engineering-Team (Doing), 10Release Pipeline (Blubber): Blubber Docker compiler output does not include copied stages referenced transitively via includes - https://phabricator.wikimedia.org/T289880 (10dduvall) 05Open→03Resolved Deployed. [21:52:19] (03PS12) 10Jeena Huneidi: Adds kubernetes cluster [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) [21:53:01] (03CR) 10Jeena Huneidi: Adds kubernetes cluster (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/715290 (https://phabricator.wikimedia.org/T287993) (owner: 10Jeena Huneidi) [22:51:31] 10Gerrit, 10User-Zabe: Registered users can't read labs/private while anonymous users can - https://phabricator.wikimedia.org/T290102 (10thcipriani) 05Open→03Resolved a:03thcipriani >>! In T290102#7323122, @Zabe wrote: >>>! In T290102#7323088, @Legoktm wrote: >> {F34627027} >> >> I think you need to be... [22:58:31] legoktm: do you know anything about magic that might be needed to add somewhere when setting up a helmfile.d service that is using egress rules and service discovery? I'm trying to add stuff for toolhub and jenkins is blowing up in the validate_deployments step. The head scratcher for me is that with `rake run_locally[validate_deployments]` I get the toolhub failures but also a lot of other failures in other charts, so It feels like there is [22:58:32] some CI specific config magic somewhere that I don't know about. [22:58:56] Failure at https://integration.wikimedia.org/ci/job/helm-lint/5080/console [22:59:06] * legoktm looks [22:59:56] I think the problem is that the discovery.listeners[0]=search-https_eqiad thing is not found in some lookup file [23:01:59] one thing to be aware of is that helm-lint runs against the published chart, not whatever you might have stacked in Gerrit [23:02:17] hmmm... that might be related [23:03:32] if you run `helm template .` locally does it work? [23:05:14] yes, but the bit that the linter is failing on is config that is in the helmfile.d values and not the chart directly. [23:06:06] * bd808 tries to grok where services_proxy values come from for other charts [23:10:17] in prod it comes from /etc/helmfile-defaults/general-eqiad.yaml [23:10:58] I see you have https://gerrit.wikimedia.org/g/operations/deployment-charts/+/3993f7f6816ad2d61c20f700e5b300aad3ad7144/charts/toolhub/.fixtures/egress_enabled.yaml [23:11:10] maybe that's not being loaded somehow? [23:11:35] oh [23:11:43] https://codesearch.wmcloud.org/operations/?q=services_proxy&i=nope&files=&excludeFiles=&repos= [23:11:58] bd808: all the other services have the .fixtures.yaml file in their helmfile.d rather than in the chart [23:13:46] 10Release-Engineering-Team (Next), 10GitLab, 10serviceops, 10User-brennen: GitLab major version upgrade: 14.x - https://phabricator.wikimedia.org/T289802 (10brennen) Note upgrade path docs here: https://docs.gitlab.com/ce/update/index.html#upgrade-paths [23:13:49] legoktm: ah ha! I think that means I need both actually. The one with the chart is part of the skel that create_new_service.sh applies. [23:15:01] I bet that .fixtures/service_discovery.yaml is related too. Ok I have more things to dig into [23:15:36] :D I hope that was it [23:17:33] I'm now thinking that "search-https_eqiad" is not expected to be used and that may be the biggest problem. It is in /etc/envoy/envoy.yaml but it is not in /etc/helmfile-defaults/general-eqiad.yaml [23:25:49] if it needs to be in general-eqiad.yaml, then it needs to be added via puppet I believe [23:27:06] https://gerrit.wikimedia.org/g/operations/puppet/+/2e34a404bdd9a7b1e21049f20015a1bc35d46200/hieradata/common/profile/services_proxy/envoy.yaml I *think* [23:27:16] is search-chi-eqiad something different? [23:30:27] legoktm: I think all the search-(sub cluster)-(dc) things actually point to the same Elasticsearch entrypoints, but I'm honestly not sure. I left some comments that I hope effie can answer better. [23:30:42] She's the one trying to trick me into enabling egress :) [23:32:08] sounds good :)