[01:09:42] <wikibugs>	 10serviceops, 10GitLab: gitlab1004 - puppet cert revoked? - https://phabricator.wikimedia.org/T309259 (10Dzahn)
[01:10:02] <wikibugs>	 10serviceops, 10GitLab, 10SRE: gitlab1004 - puppet cert revoked? - https://phabricator.wikimedia.org/T309259 (10Dzahn)
[01:12:09] <wikibugs>	 10serviceops, 10Data-Persistence-Backup, 10GitLab, 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn)
[01:12:22] <wikibugs>	 10serviceops, 10SRE, 10GitLab (Infrastructure): gitlab1004 - puppet cert revoked? - https://phabricator.wikimedia.org/T309259 (10Dzahn)
[01:12:54] <wikibugs>	 10serviceops, 10Data-Persistence-Backup, 10GitLab (Infrastructure), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn)
[01:13:32] <wikibugs>	 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: gitlab-restore: version detection fail / restore fail - https://phabricator.wikimedia.org/T308089 (10Dzahn)
[07:28:10] <wikibugs>	 10serviceops, 10Data-Persistence-Backup, 10GitLab (Infrastructure), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10jcrespo) Yesterday's gitlab full backup was of only 42KB FYI. I would consider that a backup failure.  ` id: 445024, ts: 2022-05-26 05:...
[08:47:38] <wikibugs>	 10serviceops, 10Observability-Logging, 10WMF-General-or-Unknown, 10Performance-Team (Radar), 10Sustainability (Incident Followup): Ingest logs from scheduled maintenance scripts at WMF in Logstash - https://phabricator.wikimedia.org/T285896 (10fgiunchedi) We (o11y) discussed this task at the team meeting...
[13:39:07] <hnowlan>	 There's a chain of CRs for moving image-suggestion through the service states here if someone has a few minutes to rubber stamp things :) https://gerrit.wikimedia.org/r/c/operations/puppet/+/799357/1 it's up and running in k8s atm
[15:48:00] <bd808>	 I have DNS and service catalog addition patches pending for developer-portal that could use a review and merge by a root -- https://gerrit.wikimedia.org/r/q/topic:developer-portal+is:open
[16:21:28] <bd808>	 My DNS patch is merged and `curl https://developer-portal.svc.eqiad.wmnet:30443` is working. so close having this all sorted out. :)
[21:17:35] <wikibugs>	 10serviceops, 10SRE, 10GitLab (Infrastructure): gitlab1004 - puppet cert revoked? - https://phabricator.wikimedia.org/T309259 (10Dzahn) 05Open→03Resolved a:03Dzahn Notice: /Stage[main]/Ferm/Service[ferm]/ensure: ensure changed 'stopped' to 'running' (corrective) Info: /Stage[main]/Ferm/Service[ferm]: U...
[21:35:11] <wikibugs>	 10serviceops, 10MW-on-K8s: mw-on-k8s apache config missing cache-control for /static/ files - https://phabricator.wikimedia.org/T309358 (10Krinkle)
[21:35:21] <wikibugs>	 10serviceops, 10MW-on-K8s, 10Performance-Team: mw-on-k8s apache config missing cache-control for /static/ files - https://phabricator.wikimedia.org/T309358 (10Krinkle)
[21:39:25] <wikibugs>	 10serviceops, 10MW-on-K8s, 10Performance-Team: mw-on-k8s apache config missing cache-control for /static/ files - https://phabricator.wikimedia.org/T309358 (10Krinkle)
[23:14:06] <wikibugs>	 10serviceops, 10SRE, 10GitLab (Infrastructure): gitlab1004 - puppet cert revoked? - https://phabricator.wikimedia.org/T309259 (10Dzahn) Now using this machine for https://gerrit.wikimedia.org/r/c/operations/puppet/+/800308 and setting it active in netbox.
[23:18:00] <wikibugs>	 10serviceops, 10GitLab (Infrastructure), 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) set gitlab1004, gitlab-runner1002/1003/1004, gitlab-runner2002/2003/2004 from staged to Active status in netbox. because meanwhile they have act...
[23:22:57] <wikibugs>	 10serviceops, 10Data-Persistence-Backup, 10GitLab (Infrastructure), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) useful link to see which repos use the most space, provided by Brennen:  https://gitlab.wikimedia.org/admin/projects?sort=storag...
[23:26:42] <wikibugs>	 10serviceops, 10Data-Persistence-Backup, 10GitLab (Infrastructure), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) >>! In T274463#7959305, @jcrespo wrote: > Yesterday's gitlab full backup was of only 42KB FYI. I would consider that a backup fa...