[09:16:02] <wikibugs>	 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: Troubleshoot GitLab nftables throttling after switchover - https://phabricator.wikimedia.org/T400971#11101306 (10ABran-WMF) the last iteration does not seem to have a negative impact on Gerrit and has reduced `gitlab1004` DENYLIST to a...
[11:08:27] <wikibugs>	 10GitLab: GitLab Private Repository Request for: repos/sre/XCHEESESCORE - https://phabricator.wikimedia.org/T401921#11101832 (10ABran-WMF) Empty private repository created: [[ https://gitlab.wikimedia.org/repos/sre/XCHEESESCORE | repos/sre/XCHEESESCORE ]]
[11:08:46] <wikibugs>	 10GitLab, 06collaboration-services: GitLab Private Repository Request for: repos/sre/XCHEESESCORE - https://phabricator.wikimedia.org/T401921#11101837 (10ABran-WMF) p:05Triage→03Medium a:03ABran-WMF
[13:39:58] <wikibugs>	 10GitLab, 06collaboration-services: GitLab Private Repository Request for: repos/sre/XCHEESESCORE - https://phabricator.wikimedia.org/T401921#11102269 (10ABran-WMF) 05Open→03Resolved closing this, let me know if there is any issue
[14:16:21] <jnuche>	 👋 in case there's someone around in this channel: GitLab can't run any pipelines at the moment, I've tried a few repos and they were all affected, other people have reported the issue as well
[14:28:05] <dcaro>	 thanks!
[14:30:37] <arnaudb>	 investigating, https://gerrit.wikimedia.org/r/c/operations/puppet/+/1178880 was suspected to be the cause, I'm not sure it is because the policy is set to accept and no "fake" throttling is happening: https://grafana.wikimedia.org/goto/ntk2PVuNg?orgId=1 
[14:37:19] <arnaudb>	 reverting the change just in case
[14:39:30] <mutante>	 arnaudb: or set "profile::firewall::nftables_throttling::ensure" to absent for like 1 minute and try to run a pipeline
[14:39:56] <arnaudb>	 will do if that doesn't fix the situation, good idea mutante 
[14:42:38] <mutante>	 I dont see any specifically failed pipelines.. the latest one passed an hour ago. then just none show up.
[14:43:05] <arnaudb>	 the one I'm triggering manually are marked "pending" (https://gitlab.wikimedia.org/repos/sre/schema-changes/-/jobs/590383)
[14:43:12] <mutante>	 maybe we should try "turning it off and on again"
[14:43:18] <mutante>	 aka restarting gitlab
[14:44:13] <arnaudb>	 lets maybe try this before disabling throttling 
[14:44:23] <mutante>	 if it was some kind of networking issue to talk to runners or so I would expect them to show as failed?
[14:44:24] <arnaudb>	 (my revert has not changed anything)
[14:46:17] <mutante>	 ok, let me just do the restart
[14:46:25] <arnaudb>	 oh I started it
[14:46:33] <mutante>	 ok, alright
[14:46:42] <mutante>	 gitlab-ctl restart ?
[14:46:53] <arnaudb>	 yep, as per the cheat sheet: https://wikitech.wikimedia.org/wiki/GitLab/Cheat_Sheet :D
[14:47:02] <mutante>	 same :)
[14:49:13] <arnaudb>	 nope, same result
[14:49:25] <mutante>	 well, I was going to report something more positive
[14:49:31] <arnaudb>	 ?
[14:49:33] <mutante>	 suddenly there are newly created pipelines
[14:49:41] <mutante>	 https://gitlab.wikimedia.org/repos/sre/schema-changes/-/pipelines
[14:49:44] <mutante>	 shows one running as well
[14:50:16] <mutante>	 and one is failed now.. but actively failed is better than "nothing" ?
[14:50:27] <dcaro>	 I think so?
[14:50:32] <arnaudb>	 indeed
[14:50:50] <arnaudb>	 jnuche: can you retry the one you had problem with initially?
[14:51:24] <mutante>	 I see PASSED :)
[14:51:31] <mutante>	 24 seconds ago
[14:52:26] <mutante>	 https://c.tenor.com/eqqPlamdM1gAAAAd/tenor.gif
[14:52:39] <arnaudb>	 :D
[14:52:39] <jnuche>	 arnaudb: yeah, working for me too :)
[14:52:52] <jnuche>	 lol
[14:53:02] <arnaudb>	 nice, thanks jnuche I'll reapply my thresholds then, will notify here when the policy switches back to drop