[11:27:22] <jelto>	 jayme: puppet on contint hosts fails since around 2 hours with
[11:27:22] <jelto>	 Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Evaluation Error: Error while evaluating a Function Call, value returned from k8s::fetch_clusters has wrong type, entry 'staging' expects a value for key 'etcd_srv_name' (file: /etc/puppet/modules/profile/manifests/kubernetes/deployment_server.pp, line: 41, column: 28) on node contint1002.wikimedia.org
[11:27:22] <jelto>	 I'm parsing your latest puppet merges currently. Do you think this is related to anything you merged today?
[11:27:57] <jayme>	 jelto: damn...yea
[11:27:59] <jayme>	 sorry
[11:29:37] <jelto>	 no problem. Just noticed it. As far as I can see it just blocks new puppet runs, no other impact and no changes we have to deploy to contint atm
[11:35:43] <jayme>	 thanks for noticing. Running PCC rn on https://gerrit.wikimedia.org/r/c/operations/puppet/+/940339  - that shuld fix it
[11:47:08] <jelto>	 jayme: thanks, puppet is happy again on contint hosts
[11:47:26] <jayme>	 🎉
[12:59:50] <wikibugs>	 10serviceops, 10DC-Ops, 10SRE, 10ops-eqiad: Q4:rack/setup/install rdb101[34] - https://phabricator.wikimedia.org/T326170 (10Jclark-ctr)
[14:09:30] <wikibugs>	 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Requesting permission to use kafka-main cluster to transport CirrusSearch updates - https://phabricator.wikimedia.org/T341625 (10pfischer) @JMeybohm, @dcausse and I put together [[ https://docs.google.com/spreadsheets/d/1Fp44MdLxUVlxi03MBD...
[14:14:16] <wikibugs>	 10serviceops, 10Data-Platform-SRE, 10Discovery-Search (Current work): Requesting permission to use kafka-main cluster to transport CirrusSearch updates - https://phabricator.wikimedia.org/T341625 (10Joe) A two quarters rollout would mean that we can't really wait for the removal of the old topics; this would...
[14:59:47] <wikibugs>	 10serviceops, 10Foundational Technology Requests, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Kubernetes v1.23 use PKI for service-account signing (instead of cergen) - https://phabricator.wikimedia.org/T329826 (10JMeybohm)
[15:00:00] <wikibugs>	 10serviceops, 10Patch-For-Review: Allow for multiple confd instances in pupper - https://phabricator.wikimedia.org/T341669 (10JMeybohm) 05Open→03Resolved a:03JMeybohm
[19:33:57] <wikibugs>	 10serviceops, 10MW-on-K8s, 10Release Pipeline: Pushes to docker-registry fail for images with compressed layers of size >1GB - https://phabricator.wikimedia.org/T288198 (10dancy) Another case of failing to push a large image: T342084.  Is it possible to configure NGINX to use a different (large, on-disk) sto...
[19:43:29] <wikibugs>	 10serviceops, 10MW-on-K8s, 10Release Pipeline: Pushes to docker-registry fail for images with compressed layers of size >1GB - https://phabricator.wikimedia.org/T288198 (10dancy)
[20:19:13] <wikibugs>	 10serviceops, 10SRE, 10Abstract Wikipedia team (Phase λ – Launch), 10Patch-For-Review, 10Service-deployment-requests: New Service Request: function-orchestrator and function-evaluator (for Wikifunctions launch) - https://phabricator.wikimedia.org/T297314 (10cmassaro) Calls to the orchestrator now work! W...
[20:24:24] <wikibugs>	 10serviceops, 10SRE, 10Abstract Wikipedia team (Phase λ – Launch), 10Patch-For-Review, 10Service-deployment-requests: New Service Request: function-orchestrator and function-evaluator (for Wikifunctions launch) - https://phabricator.wikimedia.org/T297314 (10cmassaro) @JMeybohm, it appears that there's a...