[12:07:28] hnowlan (pinged due to git blame https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/services/change-propagation/+log/refs/heads/master) I'm planning to increase the partition count for the eqiad.mediawiki.job.htmlCacheUpdate topic, that seems to be related to changeprop. Do you know if the associated consumer is using consumer groups (which [12:07:28] will cause it to take the new partitions into account on the fly)? [12:08:43] cf https://gerrit.wikimedia.org/g/mediawiki/services/change-propagation/+/9f5e06a8a0c680206a52cc1121d1e0304ea4114f/config.jobqueue.wikimedia.yaml#54 [12:45:21] brouberol: changeprop's htmlCacheUpdate only really knows about databases for partitioning - I can't quite remember whether the consumer groups will take account but it most likely will at minimum need a restart [12:45:26] You'd be best off asking serviceops [12:46:26] thanks! [13:16:38] Hi, I think there is an issue with puppet and IDP: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328573 [13:17:53] When I created the first revision, pcc failed on absent secret. When I added a placeholder secret to the labs/private, it started failing on pre-compile because other fields are empty. [13:27:23] <_joe_> not sure I understand. Do you have a link to a failed pcc run after you added the secrets to labs/private? [13:28:26] _joe_: in this diff, rev1 is done before, and rev 2 is after adding secrets to labs/private [13:28:41] <_joe_> ahhh ok so it's expected [13:28:53] <_joe_> rev1 will fail pcc, if it's already merged [13:29:47] _joe_: do you mean that is it expected that pcc will fail no matter what? [13:29:54] <_joe_> uhm no wait, I'm looking at the pcc output and it does not make sense [13:30:01] yeah [13:30:02] there is the run before merging to labs/private https://puppet-compiler.wmflabs.org/output/1328573/7616/idp2005.wikimedia.org/change.idp2005.wikimedia.org.err [13:30:05] I've been scratching my head about it [13:30:26] there is a run after merging to labs/private https://puppet-compiler.wmflabs.org/output/1328573/7618/idp2005.wikimedia.org/prod.idp2005.wikimedia.org.err [13:31:10] something in the CAS puppetization must be instantiating a Apereo_cas::Service[airflow_experiment_platform] because it found hiera for it, even if it was just a placeholder secret [13:31:29] <_joe_> yes I was getting there [13:31:37] <_joe_> that's the only explanation [13:32:17] 227 │ $services.each |String $service, Hash $config| { [13:32:19] 228 │ apereo_cas::service { $service: [13:32:21] 229 │ * => $config, [13:32:23] 230 │ } [13:32:25] 231 │ } [13:32:27] in modules/apereo_cas/manifests/init.pp [13:32:33] <_joe_> yeah [13:32:50] <_joe_> ok so yes, that's an unfortunate choice :) [13:32:52] atsukoito: if you make that conditional on the id key being present in $config, I think both sides will work [13:33:09] <_joe_> I would rather invert the logic [13:33:27] <_joe_> and use $config as the source of the loop, but it's not on atsukoito to fix it now I guess [13:37:28] cdanis: my diff is instantiating it i guess, because I'm adding a service [13:37:42] atsukoito: yeah exactly, and the labs/private change is used for both sides of the PCC [13:38:04] i'll make a diff now [13:38:11] i'm happy to review [13:38:21] thanks :) [13:48:58] fixing in https://phabricator.wikimedia.org/T435815 [13:51:39] atsukoito: can I merge you puppet-merge patch? [13:52:09] yup! [13:52:30] {{done}} [14:09:58] cdanis: _joe_ : https://gerrit.wikimedia.org/r/c/operations/puppet/+/1328610 [14:25:46] Does anyone have an example of selecting 10Gbps hosts with cumin? The only Puppet fact for connection speed I found is `net_driver.${device}.speed`, which varies based on NIC hardware [14:26:03] thanks cdanis, _joe_, PCC on IDP is back [14:26:18] of course it is very likely I missed something ;) [14:28:42] inflatador: worst case, you can do something like this https://phabricator.wikimedia.org/P8744 [14:33:48] cdanis: oh-h.. now IDP fails in prod with `Error while evaluating a Resource Statement, Evaluation Error: Error while evaluating a Function Call, airflow_experiment_platform: $client_secret required when using OidcRegisteredService ` [14:34:21] I was following our old doc on spinning up OIDC, and it seem that we need to actually generate a secret in advance [14:34:23] cdanis ACK, will take a look. Thanks! [14:34:36] https://wikitech.wikimedia.org/wiki/Data_Platform/Systems/Airflow/Kubernetes/Operations [14:57:28] atsukoito: oh, yeah, you will need to have a secret in the actual private repo to have puppet work on idp in prod [14:58:04] from the documentation i felt like it will be generated by provider, but I was mistaken XD [14:58:12] I'll update our docs here