[08:07:24] 10serviceops, 10Icinga, 10Observability-Alerting, 10SRE, and 3 others: incident 20170323-wikibase did not trigger Icinga paging - https://phabricator.wikimedia.org/T161528 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi SRE does get paged nowadays when there's a "low" (FSVO low) availability (i.e. hi... [08:12:21] 10serviceops, 10envoy, 10Patch-For-Review, 10SRE Observability (FY2021/2022-Q4), 10User-fgiunchedi: Using port in Host header for thanos-swift / thanos-query breaks vhost selection - https://phabricator.wikimedia.org/T300119 (10fgiunchedi) Hi all, with the Envoy upgrade and the new option, what's the nex... [08:12:35] 10serviceops, 10envoy, 10Patch-For-Review, 10SRE Observability (FY2022/2023-Q1), 10User-fgiunchedi: Using port in Host header for thanos-swift / thanos-query breaks vhost selection - https://phabricator.wikimedia.org/T300119 (10fgiunchedi) [08:23:56] 10serviceops, 10envoy, 10Patch-For-Review, 10User-fgiunchedi: Using port in Host header for thanos-swift / thanos-query breaks vhost selection - https://phabricator.wikimedia.org/T300119 (10fgiunchedi) [08:51:42] FYI, the update of the codfw ganeti cluster to Bullseye will start next week, this means that the k8s etcd nodes will be switched to plain disk storage temporarily, i.e. there may the usual latency alerts [08:52:37] I'll shuffle the VMs to migrated nodes early (allowing to switch them back), but this will probably still last up to two weeks [15:08:44] 10serviceops, 10Add-Link, 10Growth-Team: Investigate increase and fluctuation in max CPU for linkrecommendation-internal container - https://phabricator.wikimedia.org/T303177 (10MShilova_WMF) [15:25:55] 10serviceops, 10Performance-Team, 10Scap: MW wmf-config tmp cache stays outdated after Scap deploy (opcache revalidation is off) - https://phabricator.wikimedia.org/T311788 (10dancy) Thanks for the analysis Krinkle. [15:43:49] 10serviceops, 10Performance-Team, 10Scap: MW wmf-config tmp cache stays outdated after Scap deploy (opcache revalidation is off) - https://phabricator.wikimedia.org/T311788 (10Joe) A 10 seconds grace period, which was the previous value for revalidate_freq, would be ok I think in the meantime. [17:58:46] 10serviceops, 10CirrusSearch, 10Discovery-Search, 10Infrastructure-Foundations, and 5 others: Half a million of CirrusSearch jobqueue execution errors per hour since 2021-09-30 16:02 - https://phabricator.wikimedia.org/T292291 (10Aklapper) 05Open→03Resolved No replies by anyone, boldly closing - shrug [19:59:43] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab, and 2 others: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) >>! In T247653#7982883, @Krinkle wrote: > 1. [change 744763 (puppet)](https://g... [20:00:18] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab, and 2 others: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) a:05hashar→03Dzahn [20:12:51] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab, and 2 others: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) >>! In T247653#7982883, @Krinkle wrote: > I propose the following rollout: add... [20:58:42] 10serviceops, 10Continuous-Integration-Infrastructure, 10SRE, 10serviceops-collab, and 2 others: replace doc1001.eqiad.wmnet with a buster VM and create the codfw equivalent - https://phabricator.wikimedia.org/T247653 (10Dzahn) >>! In T247653#7982883, @Krinkle wrote: > I propose the following rollout: I s... [22:28:06] 10serviceops, 10Data-Persistence-Backup, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) unfortunately just noticed an Icinga alert for gitlab1003 (nothing mails us about this, that's just if you happen to log at web UI... [22:31:31] 10serviceops, 10Data-Persistence-Backup, 10serviceops-collab, 10GitLab (Infrastructure), and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) ` Jul 01 01:33:24 gitlab1003 gitlab-restore.sh[2196430]: /opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:94:in `ea...