[07:32:16] <wikibugs>	 10serviceops, 10Patch-For-Review, 10Performance-Team (Radar): Install php 7.4 in production - https://phabricator.wikimedia.org/T311386 (10Joe) p:05Triage→03High a:03Joe Thanks @tstarling in fact we only tested tcp proxying in beta. I've gone a slightly different way than you suggested, adding an addit...
[08:46:54] <wikibugs>	 10serviceops, 10serviceops-collab, 10GitLab (CI & Job Runners): DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10Aklapper)
[14:28:44] <wikibugs>	 10serviceops, 10Product-Infrastructure-Team-Backlog, 10Proton: Proton failing with system_bus_socket bus failure - https://phabricator.wikimedia.org/T306017 (10Jgiannelos)
[15:11:43] <wikibugs>	 10serviceops, 10Data-Engineering, 10Event-Platform, 10SRE: eventstreams chart should use latest common_templates - https://phabricator.wikimedia.org/T310721 (10JArguello-WMF)
[15:11:55] <wikibugs>	 10serviceops, 10Data-Engineering-Kanban, 10Event-Platform, 10SRE, and 2 others: eventgate chart should use common_templates - https://phabricator.wikimedia.org/T303543 (10JArguello-WMF)
[15:17:04] <wikibugs>	 10serviceops: doc1001 upgrade to buster/bullseye - https://phabricator.wikimedia.org/T311732 (10akosiaris)
[15:17:56] <wikibugs>	 10serviceops: doc1001 upgrade to buster/bullseye - https://phabricator.wikimedia.org/T311732 (10MoritzMuehlenhoff) Last status updates have been here: https://phabricator.wikimedia.org/T247653#7974301
[15:28:04] <wikibugs>	 10serviceops: doc1001 upgrade to buster/bullseye - https://phabricator.wikimedia.org/T311732 (10akosiaris) A let me merge into that one then. Thanks!
[15:28:34] <wikibugs>	 10serviceops: doc1001 upgrade to buster/bullseye - https://phabricator.wikimedia.org/T311732 (10akosiaris)
[15:29:21] <wikibugs>	 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Seen): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10akosiaris)
[15:31:45] <wikibugs>	 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10Patch-For-Review, 10Release-Engineering-Team (Seen): replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10akosiaris) As pointed out in T311732 (now merged as duplicate of...
[19:09:20] <wikibugs>	 10serviceops, 10serviceops-collab, 10GitLab (CI & Job Runners): DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10Dzahn) currently the issue here is not DNS anymore.  but it is now:  'This job is stuck because you don't have any active runners online or available wi...
[19:13:20] <wikibugs>	 10serviceops, 10Release-Engineering-Team, 10serviceops-collab, 10GitLab (Infrastructure), 10User-brennen: GitLab major release: 15.x - https://phabricator.wikimedia.org/T309062 (10Dzahn)
[19:17:28] <wikibugs>	 10serviceops, 10SRE, 10Wikimedia-production-error: PHP7 corruption reports in 2020-2022 (Call on wrong object, etc.) - https://phabricator.wikimedia.org/T245183 (10Arlolra)
[19:49:11] <wikibugs>	 10serviceops, 10serviceops-collab, 10GitLab (CI & Job Runners): DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10Dzahn)
[20:27:27] <wikibugs>	 10serviceops, 10Wikimedia-production-error: Spike of fatal error "Cannot declare class Wikimedia\MWConfig" on mw1379 (2020-06-01) - https://phabricator.wikimedia.org/T254209 (10Krinkle) 05duplicate→03Open
[20:28:35] <wikibugs>	 10serviceops, 10Wikimedia-production-error: Spike of fatal error "Cannot declare class Wikimedia\MWConfig" on mw1379 (2020-06-01) - https://phabricator.wikimedia.org/T254209 (10Krinkle)
[20:28:42] <wikibugs>	 10serviceops, 10Wikimedia-production-error: Spike of fatal error "Cannot declare class Wikimedia\MWConfig" on mw1379 (2020-06-01) - https://phabricator.wikimedia.org/T254209 (10Krinkle) It's not an opcache revalidation issue afaik. For two reasons.  1. We've disabled opcache's live revalidation mode as of last...
[20:29:07] <wikibugs>	 10serviceops, 10Wikimedia-production-error: Brief spikes of fatals "Cannot declare class" - https://phabricator.wikimedia.org/T254209 (10Krinkle)
[20:30:43] <wikibugs>	 10serviceops, 10Wikimedia-production-error: Brief spikes of fatals "Cannot declare class" - https://phabricator.wikimedia.org/T254209 (10Krinkle) It's not an opcache revalidation issue afaik. For two reasons.  1. We've disabled opcache's live revalidation mode as of last week (finally!). 2. The Grafana host gr...
[20:31:03] <wikibugs>	 10serviceops, 10Wikimedia-production-error: Brief spikes of fatals "Cannot declare class" - https://phabricator.wikimedia.org/T254209 (10Krinkle)
[20:31:21] <wikibugs>	 10serviceops, 10Wikimedia-production-error: Spike of impossible "Cannot declare class" fatal errors - https://phabricator.wikimedia.org/T254209 (10Krinkle)
[20:33:17] <wikibugs>	 10serviceops, 10SRE, 10Wikimedia-production-error: PHP7 corruption reports in 2020-2022 (Call on wrong object, etc.) - https://phabricator.wikimedia.org/T245183 (10Krinkle) 05Open→03Resolved a:03Krinkle Any remaining "smells like opcache" problems we see can't be the cause of php-opcache revalidation m...
[22:04:01] <wikibugs>	 10serviceops, 10GitLab, 10Release-Engineering-Team, 10serviceops-collab: Changes to modules/gitlab_runner/templates/config-template.toml.erb have no effect on existing runners - https://phabricator.wikimedia.org/T311746 (10Dzahn)
[22:05:17] <wikibugs>	 10serviceops, 10SRE, 10Znuny, 10serviceops-collab, 10Sustainability (Incident Followup): enhance Znuny (otrs) alerting - https://phabricator.wikimedia.org/T303190 (10Dzahn)
[22:10:03] <wikibugs>	 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab, and 2 others: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn)
[22:11:22] <wikibugs>	 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab, and 2 others: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) @Krinkle and I agreed on doing this tomorrow at 14:00 PST
[22:11:49] <wikibugs>	 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab, and 2 others: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) 05Open→03In progress
[23:42:05] <wikibugs>	 10serviceops, 10Performance-Team, 10Scap: MW wmf-config tmp cache stays outdated after Scap deploy (opcache revalidation is off) - https://phabricator.wikimedia.org/T311788 (10Krinkle) a:03Krinkle
[23:42:13] <wikibugs>	 10serviceops, 10Performance-Team, 10Scap: MW wmf-config tmp cache stays outdated after Scap deploy (opcache revalidation is off) - https://phabricator.wikimedia.org/T311788 (10Krinkle) Okay, let's consider the following timeline for a typical config change as it was prior to last week. This means prior to do...
[23:44:38] <wikibugs>	 10serviceops, 10Performance-Team, 10Scap: MW wmf-config tmp cache stays outdated after Scap deploy (opcache revalidation is off) - https://phabricator.wikimedia.org/T311788 (10Krinkle) >>! In T311788#8042832, @Krinkle wrote: > […] > > With the above information, and the knowledge that "turning off live opcac...