[05:08:05] 06serviceops, 06SRE, 10Data Products (Data Products Sprint 11), 07Service-deployment-requests: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production - https://phabricator.wikimedia.org/T361835#9712224 (10SGupta-WMF) @WDoranWMF Yep , it makes sense . I confirmed with @mforns that API paths and... [08:03:39] 06serviceops, 10Prod-Kubernetes, 07Kubernetes: Migration to containerd and away from docker - https://phabricator.wikimedia.org/T362408#9712356 (10JMeybohm) @akosiaris could you please double check in your test environment that containerd will still enforce the default apparmor profile (see //Remove apparmor... [09:01:46] got this during deploy [09:02:04] https://www.irccloud.com/pastebin/ZTXy7ouC/ [09:03:28] https://www.irccloud.com/pastebin/TW7QaWY8/ [09:10:40] Amir1: I'm getting similar errors from the docker-report scripts on build2001 [09:10:59] 😭 [09:11:23] my change hasn't been deployed to k8s now, it's deployed to bare metal though [09:11:49] Ugh [09:13:32] Maybe it's a base image issue [09:14:27] Let me check [09:16:50] non-LTS buster architectures was archived recently (https://lists.debian.org/debian-devel-announce/2024/03/msg00003.html), so given backports does not get LTS support I suspect that's why [09:17:09] "Please keep in mind that backports doesn't follow LTS. Which means that we will drop support for oldstable (stretch) around one year after the release of buster. " [09:17:12] I guess they removed i [09:17:14] *it [09:20:24] Crap [09:22:09] I wonder if we're actually installing anything from backports in the mediawiki images though [09:25:00] good question. I recall that at least some of the shellbox images do, not sure about mw itself [09:33:32] That sources.list is part of the base buster image [09:52:45] I thought we are on bullseye [09:55:57] Amir1: Not for mediawiki, and not for some assorted services [09:56:04] taavi: you're right about shellbox [09:56:19] * Amir1 cries in a corner [10:09:39] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518 (10Clement_Goubert) 03NEW [10:10:01] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9712711 (10Clement_Goubert) 05Open→03In progress p:05Triage→03Unbreak! [10:23:34] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9712726 (10MoritzMuehlenhoff) [10:26:02] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9712728 (10Clement_Goubert) [10:33:01] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9712750 (10jijiki) [10:33:04] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9712748 (10hashar) > integration/config/operations-puppet (shellcheck) The image is based on Buster since the Puppet master in production use /... [10:40:59] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9712771 (10hnowlan) >>! In T362518#9712748, @hashar wrote: >> mediawiki/vagrant (php7.4-wikidiff2) > > #mediawiki-vagrant is based on Buster, t... [10:45:15] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9712773 (10MoritzMuehlenhoff) >>! In T362518#9712748, @hashar wrote: >> integration/config/operations-puppet (shellcheck) > > The image is base... [10:55:49] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9712795 (10MoritzMuehlenhoff) [10:57:16] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9712798 (10JMeybohm) [11:17:31] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9712847 (10Clement_Goubert) [11:20:18] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9712849 (10Clement_Goubert) [12:39:23] 06serviceops, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 5 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#9713041 (10Gehel) [12:41:55] 06serviceops, 10Prod-Kubernetes, 10Data-Platform-SRE (2024.04.15 - 2024.05.05), 07Kubernetes, 13Patch-For-Review: Improve how we address outside k8s infrastructure from within charts (e.g. network policies) - https://phabricator.wikimedia.org/T331894#9713061 (10Gehel) [13:50:59] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9713418 (10JMeybohm) [13:51:16] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9713416 (10JMeybohm) Production images rebuild is done: ` Apr 15 13:46:55 build2001 update-production-images[4127624]: Successfully published i... [13:51:47] 06serviceops, 06Content-Transform-Team-WIP, 10MW-on-K8s, 06SRE, and 4 others: A lot of `[info] Wikitext for this page has duplicate ids:` in logstash for mw-parsoid. Possibly related to PageBundle - https://phabricator.wikimedia.org/T358588#9713422 (10MSantos) [14:29:36] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9713615 (10JMeybohm) [14:32:48] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9713630 (10JMeybohm) [14:35:02] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9713636 (10akosiaris) p:05Unbreak!→03High The immediate issue blocking the train has been resolved and new images have been pushed. Hence, l... [14:42:52] 06serviceops, 10MediaWiki-General, 10MediaWiki-libs-Stats, 10observability, and 2 others: MediaWiki Prometheus support - https://phabricator.wikimedia.org/T240685#9713700 (10colewhite) a:05colewhite→03None [15:14:42] 06serviceops, 10MediaWiki-General, 10MediaWiki-libs-Stats, 10observability, and 2 others: 14MediaWiki Prometheus support - 14https://phabricator.wikimedia.org/T240685#9713872 (10colewhite) 05Open→03Resolved a:03colewhite 14This epic will continue in {T343020} [15:20:06] 06serviceops, 10MW-on-K8s, 06SRE, 06Traffic, and 3 others: A lot of `[info] Wikitext for this page has duplicate ids:` in logstash for mw-parsoid. Possibly related to PageBundle - https://phabricator.wikimedia.org/T358588#9713923 (10MSantos) [15:24:43] 06serviceops, 07Epic: Wikimedia API Gateway Long-term Use - https://phabricator.wikimedia.org/T255034#9713989 (10Krinkle) [15:24:44] 06serviceops, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team, 07Technical-Debt, 07Upstream: Migrate OAuth extension back from wikimedia/oauth2-server fork to upstream - https://phabricator.wikimedia.org/T261462#9713976 (10Krinkle) [15:24:49] 06serviceops, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team, 07Technical-Debt, 07Upstream: Migrate OAuth extension back from wikimedia/oauth2-server fork to upstream - https://phabricator.wikimedia.org/T261462#9713988 (10Krinkle) [15:25:29] 06serviceops, 10MediaWiki-extensions-OAuth, 06MediaWiki-Platform-Team, 07Technical-Debt, 07Upstream: Migrate OAuth extension back from wikimedia/oauth2-server fork to upstream - https://phabricator.wikimedia.org/T261462#9713996 (10Krinkle) [15:43:36] 06serviceops, 06Release-Engineering-Team, 10Scap: scap should support --pause-after-testserver-sync and testserver checks in --k8s-only mode - https://phabricator.wikimedia.org/T362437#9714084 (10Scott_French) [15:47:15] 06serviceops, 06Release-Engineering-Team, 10Scap: 14scap should support --pause-after-testserver-sync and testserver checks in --k8s-only mode - 14https://phabricator.wikimedia.org/T362437#9714122 (10Scott_French) 05In progress→03Resolved 14Many thanks for your help, @dancy! I'd like to add support... [16:49:47] 06serviceops, 06Infrastructure-Foundations, 06Release-Engineering-Team, 13Patch-For-Review: Deprecate buster-backports - https://phabricator.wikimedia.org/T362518#9714546 (10JMeybohm) [17:40:26] 06serviceops, 10VPS-project-Codesearch: Add docker production images repo to codesearch - https://phabricator.wikimedia.org/T362567 (10akosiaris) 03NEW [17:47:37] 06serviceops, 06Infrastructure-Foundations: Drop backports from base images - https://phabricator.wikimedia.org/T362568 (10akosiaris) 03NEW [17:54:09] 06serviceops, 06SRE, 10Data Products (Data Products Sprint 11), 07Service-deployment-requests: Commons Impact Metrics AQS 2.0 Deployment to Staging and Production - https://phabricator.wikimedia.org/T361835#9714919 (10Scott_French) @WDoranWMF and @SGupta-WMF, thank you both for the followup. As for image... [21:40:14] 06serviceops, 10VPS-project-Codesearch, 13Patch-For-Review: Add docker production images repo to codesearch - https://phabricator.wikimedia.org/T362567#9715635 (10Ladsgroup) I merged the change, it'll be live in a day, if not, ping me to kick the VM violently like a True Persian. [21:49:18] 06serviceops, 10VPS-project-Codesearch, 13Patch-For-Review: Add docker production images repo to codesearch - https://phabricator.wikimedia.org/T362567#9715651 (10Scott_French) a:03Scott_French Many thanks for the review, Amir. Great, I'll check back tomorrow and close this out once I've confirmed its live.