[08:45:21] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE: contint/releases/hosts with helm installed: puppet - Could not find group deployment - https://phabricator.wikimedia.org/T307740 (10hashar) From the contint1001 /var/log/puppet.log* files, the last good run was: ` Apr 27 15:32:34 contint1001 puppe... [08:52:47] 10serviceops, 10MW-on-K8s, 10Kubernetes, 10Patch-For-Review, 10Release-Engineering-Team (Radar): Kubernetes credentials on deployment servers should be available to deployers, not all users - https://phabricator.wikimedia.org/T305729 (10hashar) Puppet fails on the contint* hosts cause `/var/cache/helm` w... [09:00:47] 10serviceops, 10MW-on-K8s, 10Scap, 10Release-Engineering-Team (Radar): Deploy MediaWiki images for kubernetes from the deployment servers - https://phabricator.wikimedia.org/T302539 (10elukey) [09:00:56] 10serviceops, 10MW-on-K8s, 10Kubernetes, 10Patch-For-Review, 10Release-Engineering-Team (Radar): Kubernetes credentials on deployment servers should be available to deployers, not all users - https://phabricator.wikimedia.org/T305729 (10elukey) 05Resolvedβ†’03Open Reopening since it seems that more dis... [09:51:25] 10serviceops, 10SRE: Service Ops SRE support for iOS notifications update - https://phabricator.wikimedia.org/T306397 (10akosiaris) For what is worth, I think we 've peaked. In the 30day graph {F35137940} we can see the increase in traffic. However, it's so low in volume (prometheus counts 552 apns in the l... [10:45:32] 10serviceops: Productionise mc20[38-55] - https://phabricator.wikimedia.org/T293012 (10akosiaris) > we should consider if it makes sense to make an exception and renumber these hosts from 2037 so they are in par with eqiad. Niah, that would create confusion. Also, numbers don't need to match up between the 2 D... [11:09:05] <_joe_> James_F: I finally found the issue with the node14/16 publishing process, the images are now available [11:09:39] _joe_: Brilliant, thank you! [11:09:50] 10serviceops, 10Infrastructure-Foundations, 10SRE-tools: Add a kubernetes module to spicerack - https://phabricator.wikimedia.org/T300879 (10Joe) 05Openβ†’03Resolved [11:10:05] <_joe_> did you reopen the task btw? [11:10:56] No, I don’t think so. [11:11:09] Sorry, been on a off-site all week. [11:11:22] <_joe_> hey don't be [11:11:32] <_joe_> I was the one wqho didn't check if the image was actually published [11:11:43] <_joe_> turns out the docker daemon said "ok published!" [11:11:46] <_joe_> but logged an auth error [11:12:01] Helpful. :-) [11:12:07] <_joe_> and it's on me really, I know I'm dealing with quality software [11:12:21] <_joe_> I should assume it's broken until proven otherwise [11:12:24] With sufficient bugs, all eyes are shallow. [11:46:09] 10serviceops, 10MW-on-K8s, 10Kubernetes, 10Patch-For-Review, 10Release-Engineering-Team (Radar): Kubernetes credentials on deployment servers should be available to deployers, not all users - https://phabricator.wikimedia.org/T305729 (10JMeybohm) Do we really think we need a global/shared cache directory... [11:58:38] 10serviceops, 10MW-on-K8s, 10Kubernetes, 10Patch-For-Review, 10Release-Engineering-Team (Radar): Kubernetes credentials on deployment servers should be available to deployers, not all users - https://phabricator.wikimedia.org/T305729 (10akosiaris) helm 2 did have a different structure regarding these thi... [13:23:34] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update Kubernets clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [13:24:04] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10Majavah) [13:25:33] 10serviceops, 10MW-on-K8s, 10Kubernetes, 10Patch-For-Review, 10Release-Engineering-Team (Radar): Kubernetes credentials on deployment servers should be available to deployers, not all users - https://phabricator.wikimedia.org/T305729 (10elukey) Another use case mentioned in T307927#7921020 is that, IIUC,... [13:43:31] 10serviceops, 10SRE: Service Ops SRE support for iOS notifications update - https://phabricator.wikimedia.org/T306397 (10Tsevener) @akosiaris cool, thanks! My instinct is that it feels a bit low - I wonder if pushes are getting dropped somewhere. It would be cool if we could somehow check how many Echo notific... [14:43:38] 10serviceops, 10Kubernetes: Replace kubeyaml in deployment-charts CI - https://phabricator.wikimedia.org/T306165 (10JMeybohm) [18:17:44] 10serviceops, 10SRE-Access-Requests, 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10brennen) [18:23:05] 10serviceops, 10SRE-Access-Requests, 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10thcipriani) [18:24:28] 10serviceops, 10SRE-Access-Requests, 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10thcipriani) Sounds good from from my side: seems... [18:28:49] 10serviceops, 10SRE-Access-Requests, 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10RLazarus) We're past the European work day, so I... [18:29:01] 10serviceops, 10SRE-Access-Requests, 10GitLab (CI & Job Runners), 10Release-Engineering-Team (GitLab-a-thon 🦊), 10User-brennen: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10RLazarus) p:05Triageβ†’03Medium [18:36:13] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests, and 3 others: Access to trusted gitlab runners for gitlab-roots (or appropriate similar group) - https://phabricator.wikimedia.org/T308350 (10RLazarus) Hmm, also: As a group access change, this should be reviewed and approved in the... [19:13:29] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) DNS change deployed. ` host gitlab-replica-new.wikimedia.org gitlab-replica-new.wikimedia.org has address 208.80.154.15 gitlab-replica-new.wi... [20:05:27] 10serviceops, 10SRE: Renew puppet cert for etcd.codfw.wmnet - https://phabricator.wikimedia.org/T302153 (10Dzahn)