[02:00:33] 10serviceops, 10Beta-Cluster-Infrastructure, 10SRE, 10Scap, 10Release-Engineering-Team (Seen): Scap can't clear opcache on mw servers in Beta Cluster - https://phabricator.wikimedia.org/T237033 (10Krinkle) @thcipriani @dancy I believe the equivalent of the `beta-scap-eqiad` job from back then (which is n... [08:13:36] 10serviceops, 10Release-Engineering-Team, 10SRE, 10Scap: Deploy Scap version 4.8.1 - https://phabricator.wikimedia.org/T309116 (10JMeybohm) a:03JMeybohm [08:36:14] 10serviceops, 10Release-Engineering-Team, 10SRE, 10Scap: Deploy Scap version 4.8.1 - https://phabricator.wikimedia.org/T309116 (10JMeybohm) ` mwdebug1002:~$ scap pull Traceback (most recent call last): File "/usr/bin/scap", line 32, in from scap import cli File "/usr/lib/python3/dist-packa... [08:39:50] 10serviceops, 10Release-Engineering-Team, 10SRE, 10Scap: Deploy Scap version 4.8.1 - https://phabricator.wikimedia.org/T309116 (10JMeybohm) 05Open→03Stalled [10:53:18] 10serviceops, 10observability: alertmanager probes not hounering service catalog state - https://phabricator.wikimedia.org/T309774 (10jbond) [10:53:52] 10serviceops, 10observability: alertmanager not honouring service catalog "state" - https://phabricator.wikimedia.org/T309774 (10jbond) [11:12:59] 10serviceops, 10Infrastructure-Foundations, 10SRE, 10SRE-Access-Requests: allow certain users to disable puppet on mwdebug hosts - https://phabricator.wikimedia.org/T305979 (10MoritzMuehlenhoff) This was discussed in the Infrastructure Foundations team meeting and was found to be a okay (to grant the permi... [11:13:44] 10serviceops, 10Infrastructure-Foundations, 10SRE: allow certain users to disable puppet on mwdebug hosts - https://phabricator.wikimedia.org/T305979 (10MoritzMuehlenhoff) [11:57:07] ahoyhoy, could someone have a look at these state changes for image-suggestion? The service is up and working but these are the last bits left https://gerrit.wikimedia.org/r/c/operations/puppet/+/799357 https://gerrit.wikimedia.org/r/c/operations/puppet/+/799358 https://gerrit.wikimedia.org/r/c/operations/puppet/+/799998 thanks! [12:37:27] hnowlan: hey o/ [12:38:03] * jayme looking [13:06:14] thanks! [13:34:19] 10serviceops, 10SRE: Migrate node-based services in production to node14 - https://phabricator.wikimedia.org/T306995 (10Jdforrester-WMF) [13:55:52] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [13:56:13] 10serviceops, 10Kubernetes, 10Patch-For-Review: Replace kubeyaml in deployment-charts CI - https://phabricator.wikimedia.org/T306165 (10JMeybohm) 05Open→03Resolved We're now validating with kubeconform [13:56:17] 10serviceops, 10Prod-Kubernetes, 10Kubernetes: Update Kubernetes clusters to v1.23 - https://phabricator.wikimedia.org/T307943 (10JMeybohm) [14:01:29] 10serviceops, 10Data-Catalog, 10Data-Engineering, 10SRE, and 2 others: New Service Request: DataHub - https://phabricator.wikimedia.org/T303049 (10JMeybohm) >>! In T303049#7898695, @JMeybohm wrote: > I finally managed to verify and document the steps needed to put a service under Ingress. I did also update... [14:16:19] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration: Migrate thumbor to Kubernetes - https://phabricator.wikimedia.org/T233196 (10hnowlan) [14:26:26] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration: Migrate thumbor to Kubernetes - https://phabricator.wikimedia.org/T233196 (10hnowlan) [14:37:11] 10serviceops: error when running puppet on otrs1001 - https://phabricator.wikimedia.org/T309788 (10Arnoldokoth) [14:45:57] 10serviceops, 10SRE, 10Thumbor, 10Thumbor Migration: Migrate thumbor to Kubernetes - https://phabricator.wikimedia.org/T233196 (10MoritzMuehlenhoff) As for "Thumbor currently runs in firejail, do we lose anything by dropping it in k8s", that's fine. firejail was our workaround for the original service abst... [14:49:19] 10serviceops, 10Patch-For-Review: error when running puppet on otrs1001 - https://phabricator.wikimedia.org/T309788 (10Arnoldokoth) 05Open→03Resolved [15:42:05] 10serviceops, 10Release-Engineering-Team, 10SRE, 10Scap, 10Patch-For-Review: Deploy Scap version 4.8.2 - https://phabricator.wikimedia.org/T309116 (10dancy) 05Stalled→03Open [15:42:32] 10serviceops, 10Release-Engineering-Team, 10SRE, 10Scap, 10Patch-For-Review: Deploy Scap version 4.8.2 - https://phabricator.wikimedia.org/T309116 (10dancy) Fixed at tag 4.8.2. >>! In T309116#7975904, @JMeybohm wrote: > Probably missing dependencies: > ` > mwdebug1002:~$ scap pull > Traceback (most rece... [16:17:59] gitlab has now been migrated from VMs to dedicated hardware [16:49:20] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Jelto) Migration of production GitLab from `gitlab1001` to `gitlab1004` was successful. Downtime was around 65 minutes. Backups were configured for th... [16:54:53] 10serviceops, 10Parsoid, 10VisualEditor, 10Performance-Team (Radar), 10Platform Engineering Roadmap: Preemptively warm caches for Parsoid output - https://phabricator.wikimedia.org/T301371 (10daniel) [16:58:13] 10serviceops, 10Parsoid, 10VisualEditor, 10Performance-Team (Radar), 10Platform Engineering Roadmap: Preemptively warm caches for Parsoid output - https://phabricator.wikimedia.org/T301371 (10daniel) p:05Triage→03Medium [17:24:13] 10serviceops, 10Parsoid, 10VisualEditor, 10Performance-Team (Radar), 10Platform Engineering Roadmap: Preemptively warm caches for Parsoid output - https://phabricator.wikimedia.org/T301371 (10daniel) [17:27:48] 10serviceops, 10Parsoid, 10VisualEditor, 10Performance-Team (Radar), 10Platform Engineering Roadmap: Preemptively warm caches for Parsoid output - https://phabricator.wikimedia.org/T301371 (10daniel) I think this should be implemented very closely to how we do cache warming for "normal" parser output, th... [18:11:48] 10serviceops, 10Beta-Cluster-Infrastructure, 10SRE, 10Scap, 10Release-Engineering-Team (Seen): Scap can't clear opcache on mw servers in Beta Cluster - https://phabricator.wikimedia.org/T237033 (10dancy) >>! In T237033#7975492, @Krinkle wrote: > @thcipriani @dancy I believe the equivalent of the `beta-sc...