[11:00:49] 10serviceops, 10Observability-Logging: Mutate mmkubernetes k8s fields into ECS fields - https://phabricator.wikimedia.org/T292881 (10Joe) [12:21:13] 10serviceops, 10Observability-Logging: Mutate mmkubernetes k8s fields into ECS fields - https://phabricator.wikimedia.org/T292881 (10JMeybohm) I would second your proposal apart from what I think is a typo. `kubernetes.namespace_name` should be `orchestrator.namespace`. Also we might want to think about carryi... [12:23:27] 10serviceops, 10Observability-Logging: Mutate mmkubernetes k8s fields into ECS fields - https://phabricator.wikimedia.org/T292881 (10Joe) >>! In T292881#7465038, @JMeybohm wrote: > I would second your proposal apart from what I think is a typo. `kubernetes.namespace_name` should be `orchestrator.namespace`. Al... [12:27:23] 10serviceops, 10Observability-Logging: Mutate mmkubernetes k8s fields into ECS fields - https://phabricator.wikimedia.org/T292881 (10JMeybohm) >>! In T292881#7465056, @Joe wrote: > yes makes sense; I've interated your suggestions/corrections. Unfortunately `orchestrator.resource.id` is not defined in https://... [13:20:07] 10serviceops, 10Observability-Logging: Mutate mmkubernetes k8s fields into ECS fields - https://phabricator.wikimedia.org/T292881 (10Joe) >>! In T292881#7465089, @JMeybohm wrote: >>>! In T292881#7465056, @Joe wrote: >> yes makes sense; I've interated your suggestions/corrections. > > Unfortunately `orchestrat... [13:24:04] 10serviceops, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10Kubernetes: Automate issuing of TLS certificates in kubernetes clusters - https://phabricator.wikimedia.org/T294560 (10JMeybohm) [13:30:42] 10serviceops, 10CFSSL-PKI, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10Kubernetes: Automate issuing of TLS certificates in kubernetes clusters - https://phabricator.wikimedia.org/T294560 (10jbond) [13:33:38] 10serviceops, 10CFSSL-PKI, 10Infrastructure-Foundations, 10Prod-Kubernetes, 10Kubernetes: Automate issuing of TLS certificates in kubernetes clusters - https://phabricator.wikimedia.org/T294560 (10Joe) [13:38:49] 10serviceops, 10Observability-Logging: Mutate mmkubernetes k8s fields into ECS fields - https://phabricator.wikimedia.org/T292881 (10Joe) p:05Triage→03High Setting priority to "high" as this would be a blocker for T288851 given we've adopted an ECS schema there. [16:19:07] 10serviceops, 10SRE, 10Sustainability: Jobrunner on Buster occasional timeout on codfw file upload - https://phabricator.wikimedia.org/T275752 (10Legoktm) One thing I think most of us missed is that MediaWiki AFAICT isn't actually uploading any data, it's just sending a X-Copy-From to Swift. Looking at http... [16:53:18] 10serviceops, 10Citoid, 10VisualEditor, 10WMSE-Bug-Reporting-and-Translation-2021, and 2 others: Automatic citation generation using ISBN on Wikipedia doesn't work - https://phabricator.wikimedia.org/T294010 (10Mvolz) According to WorldCat we're probably exceeding a usage cap of 50,000 requests a day. [17:15:38] 10serviceops, 10Observability-Logging: Mutate mmkubernetes k8s fields into ECS fields - https://phabricator.wikimedia.org/T292881 (10colewhite) [18:21:50] 10serviceops, 10Citoid, 10VisualEditor, 10WMSE-Bug-Reporting-and-Translation-2021, and 2 others: Automatic citation generation using ISBN on Wikipedia doesn't work - https://phabricator.wikimedia.org/T294010 (10Dzahn) Can we ask for an exemption for Wikipedia? [20:51:34] 10serviceops, 10SRE, 10Sustainability: Jobrunner on Buster occasional timeout on codfw file upload - https://phabricator.wikimedia.org/T275752 (10Legoktm) Scratch that, the requests that are slow are not using X-Copy-From: ` 2021-10-28 20:45:41 [fc3a0714-35c2-4cbc-8999-ad4c24bc17b4] mw1439 testwiki 1.38.0-wm... [22:25:04] 10serviceops, 10SRE, 10Patch-For-Review: Remove mediawiki::packages::fonts from non thumbor servers - https://phabricator.wikimedia.org/T294378 (10Dzahn) I merged the mediawiki::webserver change to allow absenting fonts via Hiera: https://gerrit.wikimedia.org/r/c/operations/puppet/+/734798/10 Then used it o... [22:47:40] 10serviceops, 10Data-Persistence-Backup, 10GitLab (Infrastructure), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) We have spread the updates related to gitlab-backup-restore / sync across multiple tickets. also see T283076#7466960 [22:48:22] 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: GitLab replica in codfw - https://phabricator.wikimedia.org/T285867 (10Dzahn) We have spread the updates related to gitlab-backup-restore / sync across multiple tickets. also see T283076#7466960 [22:51:05] 10serviceops, 10Data-Persistence-Backup, 10GitLab (Infrastructure), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) So now we have a systemd unit/timer that restores data on the passive gitlab host that it gets synced over from the active host...