[02:04:26] <wikibugs>	 10GitLab (Project Migration), 10Release-Engineering-Team: Create new GitLab project group: Admin - https://phabricator.wikimedia.org/T312057 (10Familyfirst407)
[09:45:53] <jelto>	 mutante: I reverted "gitlab: add prometheus blackbox http monitor" https://gerrit.wikimedia.org/r/c/operations/puppet/+/806476 because this was blocking puppet runs on all gitlab hosts. I guess the fix is quite small, but for updates and to prevent garbage collection of this hosts I reverted, instead of troubleshoot. I hope that's ok for you!
[10:15:11] <wikibugs>	 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Jelto)  >>! In T274463#8045578, @Dzahn wrote: > unfortunately just noticed an Icinga alert for gitlab1003 (nothing mails us about this, th...
[10:17:19] <jelto>	 gitlab1003 (replica) was stuck in restore and not available since last Wednesday mostly due to stuck puppet runs (see message above). I forced a puppet run and a restore and the replica is working again.
[11:37:14] <jelto>	 GitLab needs a short maintenance break of around 10 minutes at 12:30 UTC (in one hour)
[12:11:23] <brennen>	 jelto: ack
[12:39:08] <jelto>	 GitLab maintenance is starting now. A little late due to long running preparation job
[12:55:09] <jelto>	 GitLab should be available again. Took a little bit longer than expected, sorry for that!
[12:58:43] <brennen>	 no worries and thanks for doing the work!
[14:54:34] <wikibugs>	 10GitLab (CI & Job Runners), 10serviceops, 10serviceops-collab: DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10Jelto) >>! In T311241#8042109, @Dzahn wrote: > currently the issue here is not DNS anymore. >  > but it is now:  'This job is stuck because you don't ha...
[15:20:36] <wikibugs>	 10GitLab (CI & Job Runners), 10serviceops, 10serviceops-collab: DNS/networking not working on Trusted Runners - https://phabricator.wikimedia.org/T311241 (10Jelto) I run the script in [repos/releng/gitlab-trusted-runner/](https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/) manually:  ` $ add-pr...
[17:13:31] <mutante>	 jelto: I could swear I ran puppet after the merge and should have notice that so I was really surprised negatively about that one. I'll re-revert and double check and fix it. 
[18:15:54] <wikibugs>	 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) gitlab2001 has been removed from the acme_chief yaml, that allowed it to request certs but it's still up and has the puppet role applied.  This...
[18:32:00] <wikibugs>	 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn) >>! In T307142#7967245, @Jelto wrote: > https://gitlab-replica.wikimedia.org/ will be migrated from `gitlab2001` (the old ganeti VM) to `gitlab1...
[18:39:47] <wikibugs>	 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin2002 for hosts: `gitlab2001.codfw.wmnet` - gitlab2001.codfw.wmnet (**FAIL**...
[18:52:22] <wikibugs>	 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10ops-monitoring-bot) cookbooks.sre.hosts.decommission executed by dzahn@cumin2002 for hosts: `gitlab2001.wikimedia.org` - gitlab2001.wikimedia.org (**PA...
[18:56:02] <wikibugs>	 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn)
[18:56:49] <wikibugs>	 10GitLab (Infrastructure), 10serviceops, 10Patch-For-Review: bring new gitlab hardware servers into production - https://phabricator.wikimedia.org/T307142 (10Dzahn)
[20:25:07] <wikibugs>	 10GitLab (Infrastructure), 10Data-Persistence-Backup, 10serviceops, 10serviceops-collab, and 2 others: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) >>! In T274463#8051032, @Jelto wrote: > I reverted the change which introduced the Blackbox exporter in https://gerrit.wikimedia.or...