[08:37:21] I am going to change all the repos I own on Gitlab to fast forward. I hope that's not against any norm. [08:54:27] I'll be upgrading serpens (LDAP server in codfw) to bullseye in a bit, should not be noticeable in practice since practically all LDAP requests are going against the replicas [09:23:25] update of serpens is completed [09:26:57] I am getting some puppet linting unexpected errors: "Could not find declared class openstack::nova::common::zed::bullseye" [09:29:34] Andrew removed zed templates/manifests yesterday, do you have a link for the full output? [09:29:44] (it's a codename for openstack) [09:30:23] https://integration.wikimedia.org/ci/job/operations-puppet-tests-buster/2095/console [09:30:44] I have the same error https://integration.wikimedia.org/ci/job/operations-puppet-tests-buster/2094/console [09:30:51] weirdly, it didn't happen a few minutes ago [09:31:32] however, seems consistent now, so probably related to a recent change or issue: https://integration.wikimedia.org/ci/job/operations-puppet-tests-buster/2097/console [09:34:13] I am going to make es4 and es5 RO in MW, if you see something strange please let me know [09:59:08] I think this should fix the CI failure, but need to doublecheck with WMCS SREs that bobcat is really the appropriate replacement: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1031406 [10:01:44] bobcat is correct, but the OS version needs to be bumped as well [10:04:17] good point, updating [10:20:10] jynus, jelto: I merged https://gerrit.wikimedia.org/r/c/operations/puppet/+/1031406 and CI is working again, you only need to rebase [10:23:58] great thanks for the quick help ! [10:51:46] Weird that it started failing now and not before? [10:51:56] Also thanks! [11:33:43] asked in serviceops yesterday, asking from a bigger audience here. Maybe someone can help :D [11:33:52] Hi, is there a way to force php-fpm to show the nice error message instead of this if the error type is of a certain type (here, DBUnexpectedError?) https://phabricator.wikimedia.org/T360930#9791250 [11:33:52] I wrote this message here https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1014101/10/includes/libs/rdbms/loadmonitor/LoadMonitor.php#125 [11:33:52] but as we knew, it's not bubbling up [11:49:04] I have no idea how exception log surfacing works in mw sorry :/ [11:51:26] Amir1: Maybe this https://gerrit.wikimedia.org/g/mediawiki/core/+/2f4ed86411fa67b90a0979030242aa3bdb0fc2b6/includes/exception/MWExceptionRenderer.php#45 ? [11:52:25] oh no, not from mw side, that will give the details. I want the responses looking like this: https://phabricator.wikimedia.org/T364691 [11:52:43] I know how to enable that, it's just it'll be useless to users [11:53:06] and potentially in some cases, leak private info [11:53:23] Ah I misunderstood then, sorry [12:01:42] maybe we can show something instead of it doesn't have HTML tags and is a 5xx? [12:02:58] I don't quite get why it's not caught by MWExceptionHandler [12:03:06] Does it throw a fatalErrorType? [12:03:21] (I'm codesearch-rubberducking this, I have no idea what I'm doing) [12:12:36] claime: the fatal happens so early most stuff are not set [12:12:45] which is sorta the point :D [12:15:59] fair enough [12:21:19] Amir1: is it throwing a 503? [12:21:50] let me double check, I can just lower the threshold in mwdebug so you can play with it [12:25:52] claime: I broke it in mwdebug1002 [12:25:56] try any wiki there https://en.wikipedia.org/wiki/Main_Page [12:26:37] it gives a 500 [12:27:04] but an actual valid HTML [12:27:44] Yeah, so we have an apache defined error page for 503 [12:28:06] Which is the one being shown in https://phabricator.wikimedia.org/T364691 [12:28:12] Bye stashbot [12:28:30] We don't have one for 500s [12:29:44] can we make it take 500 too? please 🥺 [12:29:50] looking at the source, it would seem it does go through MWExceptionRenderer [12:30:38] (it has the "Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information.") [12:31:13] that's for local dev setups, we can't enable it for production [12:31:20] it'll start leaking private info [12:31:27] yeah yeah I know [12:31:36] what I mean is that MWExceptionRenderer is loaded [12:32:14] Let me try something on mwdebug [12:41:35] Ugh, I figured [12:41:46] ErrorDocument 500 only applies to apache 500s, not php [12:42:16] Find a way to make it throw a 503? :/ [12:47:43] let me try that part [13:21:05] moritzm: ok to merge Moritz Mühlenhoff: admin: Update deployment description (af05b685a9) :? [13:22:58] I'm gonna be bold and merge it given it's just a description change [13:23:22] +1 if it was ever needed :) [13:39:31] claime: This patch should force 503 in php but when I use it in mwdebug, it's still 500. I don't know if it gets overriden somewhere outside of mw or it doesn't work in mw [13:39:32] https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1031437 [13:47:50] Amir1: Do other DBUnexpectedError end up with the same page type [13:50:07] it depends, something they get caught and shown in a better type but I think all get 500 [13:52:43] Yeah, I think the 500/503 is a red herring actually [14:00:50] It may be that it's a mediawiki exception handling thing, and I'm having trouble wrapping my head around why it's not being displayed like other exceptions