[05:37:12] hello, fyi there are multiple legit looking email to noc@ about issues reaching gerrit, probably a too tight requestctl rule [06:18:20] _joe_: Raine: morning, looking for a stamp on https://gitlab.wikimedia.org/repos/sre/hiddenparma/-/merge_requests/194 when you have a moment [06:29:34] <_joe_> I get a 500 for that page :/ [06:30:05] _joe_: Gitlab has been flapping since at least late yesterday [06:30:54] It looks like it started midnight UTC [06:31:01] So the last 6.5 hours actually [06:43:14] <_joe_> taavi: merged [07:06:05] and now the image is not building [07:15:59] <_joe_> taavi: well no, the pipeline fails because gitlab is in trouble [07:37:10] I believe am am the cause (as of yesterday evening) currently deleting artifacts [07:37:14] so I expect CI might already work again [07:40:26] Freed up 10GB so far apparently [07:41:02] we should have known [07:41:15] I had a very productive sunday [07:41:32] we will deduct that from any future free drinks [07:41:54] actually, you now owe free drinks [07:42:22] I think drink tabs should be a hidden feature of phabricator [07:42:39] +1 [07:46:45] in other gitlab news, prior to me breaking it, it was behaving very poorly sunday morning, lots of 500s, general errors about calling some APIs [07:47:54] (or maybe it was Saturday evening) [08:26:46] addshore: 301 to sre collab folks [08:27:04] ( #wikimedia-sre-collab) [08:27:39] ooooh, not been in there before :) [08:31:16] there is a first time for everything dear, isnt there [08:47:42] addshore: arnaud.b is working on the Gitlab woes [17:21:12] a little php gotcha that I stumbled across for the newer versions https://gerrit.wikimedia.org/r/c/operations/puppet/+/1356954 [17:27:13] eep '^^ [18:49:48] What's the practical effect of the change that hnowlan mentioned? The commit message says "route php8 versions correctly". I ask as someone currently doing php 8.5 testing (under https://phabricator.wikimedia.org/T440319) [18:50:24] bpirkle: it affects how logs are routed on logstash [18:52:01] right now php8.5 logs for mediawiki and the php-fpm syslog (I think) aren't getting the usual handling, so they probably aren't showing up in the usual mediawiki-related logstash dashboards either [18:53:32] Thank you. Sorry for the ELI5 followup, but is the manual testing I did by enabling WikimediaDebug / k8s-mwdebug-next and then checking Logstash:mwdebug still valid, or should I recheck after this is resolved? [19:33:32] I would probably re-check [20:18:35] cdanis: thank you