[07:18:04] 06serviceops: Cannot Run Golang or Rust Binaries with Provided AppArmor Profile - https://phabricator.wikimedia.org/T377468#10245136 (10JMeybohm) FWIW I can't reproduce this on my machine. Even with the apparmor profile we use in production I'm able to run a hello world golang application without issues. Maybe a... [07:23:05] 06serviceops, 06Release-Engineering-Team, 10Scap: scap fails with non-zero exit status 126 when running mwshell - https://phabricator.wikimedia.org/T377692 (10dcausse) 03NEW [07:24:57] 06serviceops, 06Release-Engineering-Team, 10Scap: scap fails with non-zero exit status 126 when running mwshell - https://phabricator.wikimedia.org/T377692#10245150 (10dcausse) p:05Triage→03Unbreak! tentatively marking as UBN as this seems to block all MW deployments. [07:29:46] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 4 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#10245155 (10MoritzMuehlenhoff) [08:48:07] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10245308 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.reimage-stacked-control-plane started by jayme@cumin1002 Reimaging k8s control planes of clu... [08:52:11] 06serviceops, 10MW-on-K8s, 10observability, 10WMF-JobQueue: runJobs.log isn't being written to - https://phabricator.wikimedia.org/T377512#10245325 (10Clement_Goubert) [09:27:31] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10245531 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.reimage-stacked-control-plane started by jayme@cumin1002 Reimaging k8s control planes of clu... [09:42:18] 06serviceops, 06Release-Engineering-Team, 10Scap: scap fails with non-zero exit status 126 when running mwshell - https://phabricator.wikimedia.org/T377692#10245590 (10dcausse) p:05Unbreak!→03Medium I could deploy successfully after the revert, lowering prio but keeping the ticket open in case it's usefu... [09:43:07] 06serviceops, 06Release-Engineering-Team, 10Scap: scap fails with non-zero exit status 126 when running mwshell - https://phabricator.wikimedia.org/T377692#10245593 (10dcausse) p:05Medium→03Triage [09:46:21] 06serviceops, 06Release-Engineering-Team, 10Scap: scap fails with non-zero exit status 126 when running mwshell - https://phabricator.wikimedia.org/T377692#10245602 (10jnuche) 05Open→03Resolved a:03jnuche Did a temporary revert as a quick fix https://gitlab.wikimedia.org/repos/releng/scap/-/merge_r... [09:51:55] 06serviceops: Cannot Run Golang or Rust Binaries with Provided AppArmor Profile - https://phabricator.wikimedia.org/T377468#10245634 (10cmassaro) Hmm, weird! I am not on Mac (or are you saying you're on Mac?). I'll try running with your source file and Dockerfile. [09:59:44] 06serviceops: Cannot Run Golang or Rust Binaries with Provided AppArmor Profile - https://phabricator.wikimedia.org/T377468#10245654 (10cmassaro) Interesting, yeah, I get a similar error with your source file and Dockerfile: ` $ docker run --security-opt apparmor=wikifunctions-evaluator ce2be345e1a2 runtime: st... [10:01:35] 06serviceops: Cannot Run Golang or Rust Binaries with Provided AppArmor Profile - https://phabricator.wikimedia.org/T377468#10245656 (10akosiaris) To debug apparmor profiles, an easy way is to add the `complain` flag. So, `flags=(attach_disconnected)` becomes `flags=(complain)` and the actions will be allowed an... [10:08:57] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10245680 (10ops-monitoring-bot) Cookbook cookbooks.sre.k8s.reimage-stacked-control-plane started by jayme@cumin1002 Reimaging k8s control planes of clu... [10:15:38] 06serviceops: Cannot Run Golang or Rust Binaries with Provided AppArmor Profile - https://phabricator.wikimedia.org/T377468#10245703 (10cmassaro) I tried using `strace`. Here's my new Dockerfile: ` FROM golang:1.23 AS builder WORKDIR /app COPY . . RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -ldflags... [10:20:27] 06serviceops: Cannot Run Golang or Rust Binaries with Provided AppArmor Profile - https://phabricator.wikimedia.org/T377468#10245722 (10cmassaro) >>! In T377468#10245656, @akosiaris wrote: > To debug apparmor profiles, an easy way is to add the `complain` flag. So, `flags=(attach_disconnected)` becomes `flags=(c... [10:53:10] akosiaris, effie, jayme: there's a proposal to review the codfw Calico issues at today's ritual, would you be available to talk about it? Apologies for the short notice. [10:54:12] sobanski: I'll be around [10:57:39] sobanski: it will depend on how are things at home around that time [11:15:48] sobanski: I 'll be around [11:16:02] Thanks! [12:13:16] 06serviceops: Phabricator cli for serviceops - https://phabricator.wikimedia.org/T377311#10245993 (10jijiki) [12:14:19] 06serviceops: Phabricator cli for serviceops - https://phabricator.wikimedia.org/T377311#10245996 (10jijiki) [13:46:32] 06serviceops, 10Prod-Kubernetes, 06Traffic, 07Kubernetes, 13Patch-For-Review: Reverse DNS for k8s pods IPs - https://phabricator.wikimedia.org/T344171#10246381 (10CDanis) 05Open→03Resolved a:03CDanis ` root@db1169:~# ss -tr | grep :mysql ESTAB 0 945 db1169.eqiad.wmnet:mysql 10-67-163... [13:53:17] 06serviceops, 06Infrastructure-Foundations, 10netops, 10Prod-Kubernetes: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10246403 (10cmooney) p:05Triage→03Medium What's the current thinking here? Are we agreed to widen both allocations to /17... [14:24:32] 06serviceops: Cannot Run Golang or Rust Binaries with Provided AppArmor Profile - https://phabricator.wikimedia.org/T377468#10246580 (10cmassaro) Hmm, `complain` mode didn't change anything. Here is what I have found so far. When I change all the `deny` rules to `audit` rules, I see this in the first few lines... [14:49:28] 06serviceops, 06Content-Transform-Team-WIP, 10Page Content Service, 10RESTBase Sunsetting, and 2 others: hewiki: Route mobile-html to the backing node service instead of RESTBase - https://phabricator.wikimedia.org/T372746#10246737 (10MSantos) [14:52:35] 06serviceops, 10MW-on-K8s, 10observability, 10WMF-JobQueue: runJobs.log isn't being written to - https://phabricator.wikimedia.org/T377512#10246783 (10jijiki) It seems that it is simply another case of stalled documentation, what you are looking for is in `mwlog1002:/srv/mw-log/JobExecutor.log`. I updated... [14:53:26] 06serviceops, 06Infrastructure-Foundations, 10netops, 10Prod-Kubernetes: WikiKube clusters close to exhausting Calico IPPool allocations - https://phabricator.wikimedia.org/T375845#10246786 (10akosiaris) Good question. Let me add some data points. We currently use: ` root@deploy1003:~# kube_env admin eqia... [14:56:59] 06serviceops, 10MW-on-K8s, 10observability, 10WMF-JobQueue: runJobs.log isn't being written to - https://phabricator.wikimedia.org/T377512#10246799 (10jijiki) p:05Triage→03Low [14:59:11] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Rename wikikube worker nodes during OS reimage - https://phabricator.wikimedia.org/T365571#10246817 (10jijiki) p:05Triage→03Medium [15:39:09] 06serviceops, 10MW-on-K8s, 10observability, 10WMF-JobQueue: runJobs.log isn't being written to - https://phabricator.wikimedia.org/T377512#10247033 (10jijiki) 05Resolved→03Open [15:39:26] 06serviceops, 10MW-on-K8s, 10observability, 10WMF-JobQueue: runJobs.log isn't being written to - https://phabricator.wikimedia.org/T377512#10247034 (10jijiki) p:05Low→03Triage [15:39:54] 06serviceops, 10MW-on-K8s, 10observability, 10WMF-JobQueue: runJobs.log isn't being written to - https://phabricator.wikimedia.org/T377512#10247030 (10jijiki) 05Open→03Resolved a:03jijiki [17:01:53] 06serviceops, 10MW-on-K8s, 10TimedMediaHandler, 13Patch-For-Review, 07Video: shellbox-video pods being restarted prematurely - https://phabricator.wikimedia.org/T373517#10247494 (10hnowlan) Minor datapoint that hasn't been noted - when testing with a [[ https://test2.wikipedia.org/wiki/File:Spinner-long.... [19:05:06] 06serviceops, 06Release-Engineering-Team, 10Scap: Scap fails with `docker: Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock` - https://phabricator.wikimedia.org/T376023#10248068 (10dancy) 05Open→03Resolved [21:23:57] 06serviceops, 13Patch-For-Review: Turn up PHP 8.1-flavored k8s deployments for all MediaWiki services - https://phabricator.wikimedia.org/T377040#10248510 (10Scott_French) Aside from eventually enabling paging and httpbb checks, the mw-web-next and mw-api-ext-next services are up, along with all supporting bit... [21:28:02] 06serviceops, 10MW-on-K8s: Functional replacement for importImages.php on Kubernetes - https://phabricator.wikimedia.org/T377497#10248513 (10Pppery) That documentation isn't quite accurate. The goal of server-side uploads as they are used today is to work around the fact that uploads of large files are flaky f...