[10:51:53] <wikibugs>	 10GitLab (Infrastructure), 06collaboration-services: SystemdUnitFailed (backup-restore.service on gitlab replicas) - https://phabricator.wikimedia.org/T388308#10617779 (10Jelto) 05Open→03Resolved The restore on the replica was moved to a later schedule with the change above.  The last restore I trigger...
[15:37:04] <wikibugs>	 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: SystemdUnitFailed (backup-restore.service on gitlab replicas) - https://phabricator.wikimedia.org/T388308#10619477 (10Jelto)
[15:37:16] <wikibugs>	 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: SystemdUnitFailed (backup-restore.service on gitlab replicas) - https://phabricator.wikimedia.org/T388308#10619479 (10Jelto) 05Resolved→03Open
[15:38:22] <wikibugs>	 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: SystemdUnitFailed (backup-restore.service on gitlab replicas) - https://phabricator.wikimedia.org/T388308#10619487 (10Jelto) This happened again because the sync job is missing a dependency to the `partial-backup.service`. The change ab...
[17:26:42] <wikibugs>	 10GitLab, 06collaboration-services, 10Stewards-Onboarding-Tool: Update profile::stewards::gitlab_api_token to match new token - https://phabricator.wikimedia.org/T388212#10620094 (10LSobanski) p:05Triage→03Medium
[18:26:38] <wikibugs>	 10GitLab, 06collaboration-services, 10Stewards-Onboarding-Tool: Update profile::stewards::gitlab_api_token to match new token - https://phabricator.wikimedia.org/T388212#10620361 (10Dzahn) 05Open→03Resolved I have updated the token in the private puppet repo and ran puppet on stewards1001. It has bee...
[19:47:24] <wikibugs>	 10GitLab (Upstream pit of despair 🕳️): GraphQL error: Field 'troubleshootJobWithAi' doesn't exist on type 'Pipeline' - https://phabricator.wikimedia.org/T388459 (10Reedy) 03NEW
[19:47:54] <wikibugs>	 10GitLab, 07Upstream: GraphQL error: Field 'troubleshootJobWithAi' doesn't exist on type 'Pipeline' - https://phabricator.wikimedia.org/T388459#10620836 (10Reedy)
[19:47:57] <Reedy>	 forgot about this channel
[19:48:07] <Reedy>	 https://gitlab.wikimedia.org/repos/data-engineering/metrics-platform/-/merge_requests/81/pipelines doesn't seem to want to run pipelines
[19:48:41] <wikibugs>	 10GitLab (Upstream pit of despair 🕳️): GraphQL error: Field 'troubleshootJobWithAi' doesn't exist on type 'Pipeline' - https://phabricator.wikimedia.org/T388459#10620842 (10Reedy)
[19:48:51] <Reedy>	 I  have no idea which encantation of tags is right
[19:49:18] <wikibugs>	 10GitLab (Upstream pit of despair 🕳️): GraphQL error: Field 'troubleshootJobWithAi' doesn't exist on type 'Pipeline' - https://phabricator.wikimedia.org/T388459#10620851 (10Reedy)
[20:03:15] <wikibugs>	 10GitLab (Upstream pit of despair 🕳️), 07Upstream: GraphQL error: Field 'troubleshootJobWithAi' doesn't exist on type 'Pipeline' - https://phabricator.wikimedia.org/T388459#10620914 (10bd808)
[20:20:29] <bd808>	 Reedy: Tyler and I are both looking at your problem. I can't say either of us are making much sense of it yet.
[20:22:06] <Reedy>	 heh
[20:25:02] <Reedy>	 But yeah... why it's not running anything at all now...
[20:25:03] <Reedy>	 I'm gonna nip out to the supermarket, back within an hour
[21:33:16] <wikibugs>	 10GitLab (Upstream pit of despair 🕳️), 07Upstream: GraphQL error: Field 'troubleshootJobWithAi' doesn't exist on type 'Pipeline' - https://phabricator.wikimedia.org/T388459#10621190 (10Reedy) Expand the "failed jobs" part.. Makes the banner appear  {F58760279 size=full}
[22:53:58] <bd808>	 Reedy: I made my own fork and MR and seem to have gotten things to run -- https://gitlab.wikimedia.org/repos/data-engineering/metrics-platform/-/merge_requests/82/pipelines
[22:54:29] <bd808>	 I'm not really sure what I changed that is material in the config. :/
[22:55:40] <Reedy>	 I've definitely tripped over something that it doesn't like
[22:55:49] <bd808>	 https://gitlab.wikimedia.org/repos/data-engineering/metrics-platform/-/merge_requests/82/diffs?commit_id=d769f858a81d15b3e234236d5594e7a46f919a71 is the delta I layered on top of a cherry-pick of your patch.
[22:57:54] <bd808>	 If you want to smash those changes into yours or stack that commit in your MR I'm fine with that.
[23:04:17] <Reedy>	 For all it really matters,  retitling yours and closing mine does the same job