[00:50:56] 10serviceops, 10RESTBase, 10Release-Engineering-Team: RESTBase scap deployment failed - https://phabricator.wikimedia.org/T349318 (10thcipriani) I figured out how it's possible for a server to get into this state. **Timeline** 1. Successful scap deployment 2. Remove `/etc/nagios/nrpe.d/check_endpoints_rest... [01:50:23] 10serviceops, 10Beta-Cluster-Infrastructure, 10Thumbor, 10Beta-Cluster-reproducible: deployment-prep needs a Thumbor instance - https://phabricator.wikimedia.org/T344605 (10AlexisJazz) I want to test a security issue but can't do it properly because beta cluster has no thumbnailer and I have no sysop bit e... [08:54:40] 10serviceops, 10Beta-Cluster-Infrastructure: Unable to upload files on Beta Commons - https://phabricator.wikimedia.org/T340908 (10hashar) I merely did the troubleshooting at T340908#8985768 , then I don't know anything about how memcached is setup or how to drive it through Puppet. I'd guessed @jijiki based o... [08:55:47] 10serviceops, 10collaboration-services, 10GitLab (CI & Job Runners): Standardize Debian package builds on GitLab CI - https://phabricator.wikimedia.org/T304491 (10MatthewVernon) Cool, I'm glad it's working for you :) I'm not wedded to the `.*-wikimedia.*` naming restriction, but I think some restriction on... [09:11:03] 10serviceops, 10Prod-Kubernetes: PodSecurityPolicies will be deprecated with Kubernetes 1.21 - https://phabricator.wikimedia.org/T273507 (10JMeybohm) Thanks for putting this together. IIUC the decision we need to make is basically: "Are we okay with running MediaWiki with the Privileged profile" because all ot... [09:36:49] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Use cert-manager for service-proxy certificate creation - https://phabricator.wikimedia.org/T300033 (10JMeybohm) [09:39:53] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Use cert-manager for service-proxy certificate creation - https://phabricator.wikimedia.org/T300033 (10JMeybohm) [09:53:51] 10serviceops, 10Similarusers: Remove similar-users service from k8s - https://phabricator.wikimedia.org/T345274 (10JMeybohm) >>! In T345274#9131624, @kostajh wrote: > @Niharika @Tchanders any concerns with this? Can we please get your feedback on this? It would help to decide whether to put additional work i... [09:58:20] 10serviceops, 10Similarusers: Remove similar-users service from k8s - https://phabricator.wikimedia.org/T345274 (10kostajh) >>! In T345274#9267646, @JMeybohm wrote: >>>! In T345274#9131624, @kostajh wrote: >> @Niharika @Tchanders any concerns with this? > > Can we please get your feedback on this? It would h... [10:05:35] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Use cert-manager for service-proxy certificate creation - https://phabricator.wikimedia.org/T300033 (10JMeybohm) [10:07:31] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Use cert-manager for service-proxy certificate creation - https://phabricator.wikimedia.org/T300033 (10JMeybohm) [10:31:43] 10serviceops, 10Prod-Kubernetes: PodSecurityPolicies will be deprecated with Kubernetes 1.21 - https://phabricator.wikimedia.org/T273507 (10elukey) One thing that I didn't get yet is if we could run PSA alongside with Open Policy Agent, that could be a compromise. I suspect there shouldn't be any problem, but... [11:43:18] 10serviceops, 10RESTBase, 10Release-Engineering-Team: RESTBase scap deployment failed - https://phabricator.wikimedia.org/T349318 (10hnowlan) Looks like https://gerrit.wikimedia.org/r/c/operations/puppet/+/961003 was the source of the check disappearing, my bad. [12:27:45] 10serviceops, 10MW-on-K8s, 10MediaWiki-Engineering: EtcdConfig using stale data: lost lock in /srv/mediawiki/php-1.42.0-wmf.1/includes/config/EtcdConfig.php on line 218 - https://phabricator.wikimedia.org/T349376 (10Clement_Goubert) [13:37:36] 10serviceops, 10CX-cxserver, 10Citoid, 10Content-Transform-Team-WIP, and 8 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118 (10Jdforrester-WMF) [13:38:00] 10serviceops, 10CX-cxserver, 10Citoid, 10Content-Transform-Team-WIP, and 8 others: Migrate node-based services in production to node18 - https://phabricator.wikimedia.org/T349118 (10Jdforrester-WMF) [15:49:48] 10serviceops, 10Prod-Kubernetes, 10Kubernetes, 10Patch-For-Review: Use cert-manager for service-proxy certificate creation - https://phabricator.wikimedia.org/T300033 (10JMeybohm) [15:53:01] 10serviceops, 10Similarusers: Remove similar-users service from k8s - https://phabricator.wikimedia.org/T345274 (10JMeybohm) >>! In T345274#9267674, @kostajh wrote: >>>! In T345274#9267646, @JMeybohm wrote: >>>>! In T345274#9131624, @kostajh wrote: >>> @Niharika @Tchanders any concerns with this? >> >> Can w... [16:25:19] 10serviceops, 10Similarusers: Remove similar-users service from k8s - https://phabricator.wikimedia.org/T345274 (10Tchanders) Thanks @JMeybohm We did once talk about owning this service ourselves (as AHT), and the decision was no that time... @Madalina, @kostajh @Niharika, might be something to revisit?