[00:09:40] 06serviceops, 06Content-Transform-Team-WIP, 10MW-on-K8s, 06SRE, and 4 others: A lot of `[info] Wikitext for this page has duplicate ids:` in logstash for mw-parsoid. Possibly related to PageBundle - https://phabricator.wikimedia.org/T358588#10240560 (10ABreault-WMF) 05Open→03Resolved [01:36:21] 06serviceops, 06Content-Transform-Team, 10Electron-PDFs, 07Essential-Work: Download to PDF: HTTP 500 error on some wikis for some users - https://phabricator.wikimedia.org/T376438#10240653 (10Jake_Wartenberg) Users continue to report this problem, such as in otrs ticket 2024101810000205. I was also able t... [06:16:18] 06serviceops, 10MW-on-K8s: Functional replacement for importImages.php on Kubernetes - https://phabricator.wikimedia.org/T377497#10240682 (10Joe) I have thought of a few options for this: * I think the easiest solution to this problem is to make a two-step process, and it involves changing the script we use q... [06:19:31] 06serviceops, 10MW-on-K8s: Functional replacement for importImages.php on Kubernetes - https://phabricator.wikimedia.org/T377497#10240683 (10Joe) I should add, this is yet another example of how the unmaintained and substantially abandoned parts of MediaWiki, like the file uploads and manipulation stack, are a... [07:19:54] 06serviceops, 10observability, 10Observability-Logging, 10Prod-Kubernetes, and 2 others: containerd logs are not properly parsed during ingestion to logstash - https://phabricator.wikimedia.org/T377132#10240832 (10JMeybohm) This looks great, thanks! While checking I saw that for non JSON logs, timestamp,... [08:22:21] 06serviceops, 10MW-on-K8s: Functional replacement for importImages.php on Kubernetes - https://phabricator.wikimedia.org/T377497#10240939 (10Urbanecm_WMF) >>! In T377497#10240682, @Joe wrote: > [...] > And finally, by far my favourite option: > * Given now uploads by url are async, just raise the file size lim... [08:57:30] 06serviceops: kafka-main100[6789] and kafka-main1010 implementation tracking - https://phabricator.wikimedia.org/T363214#10241035 (10JMeybohm) a:05JMeybohm→03jijiki [09:20:24] 06serviceops, 06Content-Transform-Team, 10Electron-PDFs, 07Essential-Work: Download to PDF: HTTP 500 error on some wikis for some users - https://phabricator.wikimedia.org/T376438#10241107 (10hnowlan) >>! In T376438#10238271, @Jgiannelos wrote: > I suspect that the issue is that we don't close or somehow w... [09:24:48] 06serviceops, 06Content-Transform-Team, 10Electron-PDFs, 07Essential-Work: Download to PDF: HTTP 500 error on some wikis for some users - https://phabricator.wikimedia.org/T376438#10241121 (10hnowlan) Chromium is leaking processes, leaving `chromium_crashpad`s lying around after a failure most likey: ` ro... [09:37:54] 06serviceops, 06Content-Transform-Team, 10Electron-PDFs, 07Essential-Work: Download to PDF: HTTP 500 error on some wikis for some users - https://phabricator.wikimedia.org/T376438#10241137 (10Jgiannelos) Done, I will keep an eye on the logs. [09:49:25] 06serviceops, 06Content-Transform-Team, 10Electron-PDFs, 07Essential-Work: Download to PDF: HTTP 500 error on some wikis for some users - https://phabricator.wikimedia.org/T376438#10241153 (10akosiaris) For what is worth, I also update the dashboard at https://grafana-rw.wikimedia.org/d/U4TuF-lMk/proton?or... [11:00:35] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10241468 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestagemaster2005.codfw.wmnet with OS bo... [11:31:55] 06serviceops, 06Content-Transform-Team, 10Electron-PDFs, 07Essential-Work: Download to PDF: HTTP 500 error on some wikis for some users - https://phabricator.wikimedia.org/T376438#10241562 (10TheDJ) [[ https://github.com/puppeteer/puppeteer/issues/2778 | Here are some people with similar experiences ]]. A... [11:44:04] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10241582 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestagemaster2005.codfw.wmnet with OS bookwo... [12:04:14] 06serviceops: kafka-main100[6789] and kafka-main1010 implementation tracking - https://phabricator.wikimedia.org/T363214#10241659 (10jijiki) 05Stalled→03In progress [12:31:38] 06serviceops: Phabricator cli for serviceops - https://phabricator.wikimedia.org/T377311#10241730 (10jijiki) [12:56:16] 06serviceops: Cannot Run Golang or Rust Binaries with Provided AppArmor Profile - https://phabricator.wikimedia.org/T377468#10241796 (10akosiaris) Hi, Can you re-run these with strace so that we can figure out whether it open `/dev/null` for read or write? I 99% expect read, but wanna be sure. I think allow rea... [13:04:55] 06serviceops, 10MW-on-K8s: Functional replacement for importImages.php on Kubernetes - https://phabricator.wikimedia.org/T377497#10241831 (10Joe) >>! In T377497#10240939, @Urbanecm_WMF wrote: >>>! In T377497#10240682, @Joe wrote: >> [...] >> And finally, by far my favourite option: >> * Given now uploads by ur... [13:42:27] 06serviceops: Cannot Run Golang or Rust Binaries with Provided AppArmor Profile - https://phabricator.wikimedia.org/T377468#10241948 (10cmassaro) For Rust, I found this: `openat(AT_FDCWD, "/dev/null", O_RDWR) = -1 EACCES (Permission denied) `, so looks like it's read + write. Interestingly, this `openat` call... [14:00:12] 06serviceops, 10Deployments, 06Release-Engineering-Team: sync-testservers-k8s takes 4 minutes when deploying a mediawiki-config change - https://phabricator.wikimedia.org/T374907#10242021 (10akosiaris) I 've evaluated the change today {F57624043} I have to point out that I don't really notice a difference.... [14:39:20] 06serviceops, 10MW-on-K8s, 06SRE-OnFire, 13Patch-For-Review, 10Sustainability (Incident Followup): mwscript-k8s creates too many resources - https://phabricator.wikimedia.org/T376795#10242220 (10akosiaris) [14:39:26] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10242221 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestagemaster2003.codfw.wmnet with OS bo... [14:40:24] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10242184 (10JMeybohm) [14:48:09] 06serviceops: Cannot Run Golang or Rust Binaries with Provided AppArmor Profile - https://phabricator.wikimedia.org/T377468#10242273 (10JMeybohm) >>! From https://github.com/golang/go/commit/d4dd1de19fcef835fca14ad8cb590dbfcf8e9859 > On Unix-like platforms, enforce that the standard file descriptions (0, > 1, 2)... [15:13:20] 06serviceops, 06Data-Engineering, 10Prod-Kubernetes, 10Data-Platform-SRE (2024.10.19 - 2024.11.08), and 3 others: Migrate Search Platform-owned helm charts to Calico Network Policies - https://phabricator.wikimedia.org/T373195#10242351 (10BTullis) [15:26:22] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10242483 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestagemaster2003.codfw.wmnet with OS bookwo... [15:26:49] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10242485 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestagemaster2004.codfw.wmnet with OS bo... [15:57:44] 06serviceops: Cannot Run Golang or Rust Binaries with Provided AppArmor Profile - https://phabricator.wikimedia.org/T377468#10242635 (10cmassaro) Hmm. I get the same issue even when I add `/dev/null rw` to the profile. I notice these lines differ in the two `strace` outputs. With AppArmor enabled: ` read(0, "2... [16:10:08] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10242679 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestagemaster2004.codfw.wmnet with OS bookwo... [16:10:40] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10242680 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jayme@cumin1002 for host kubestagemaster2005.codfw.wmnet with OS bo... [16:28:21] are there any documented recommendations and/or requirements about the logging output of WMF-owned http microservices? [16:44:19] cdanis: This might help: https://wikitech.wikimedia.org/wiki/Logstash/Common_Logging_Schema - I think that the general guidance is always to try to use ECS compatible structured logs. [16:54:25] 06serviceops, 10Prod-Kubernetes, 07Kubernetes, 13Patch-For-Review: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#10242825 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jayme@cumin1002 for host kubestagemaster2005.codfw.wmnet with OS bookwo... [19:51:14] 06serviceops, 10observability, 10Observability-Logging, 10Prod-Kubernetes, and 2 others: containerd logs are not properly parsed during ingestion to logstash - https://phabricator.wikimedia.org/T377132#10243351 (10JMeybohm) 05Open→03Resolved >>! In T377132#10240832, @JMeybohm wrote: > This looks gr...