[00:32:09] <kindrobot>	 Did something happen with the https://proxy-eqiad1.wmflabs.org:5668/dynamicproxy-api/v1/wikifunctions/mapping API recently? I'm suddenly getting 403 forbidden from some requests in CI. I've even tried generating new application credentials and I'm still getting the same error.
[08:17:02] <dcaro>	 👀 looking
[08:24:16] <dcaro>	 I think it might have been related to the change on openstack role names, looking
[08:24:22] <dcaro>	 T330759
[08:24:23] <stashbot>	 T330759: Modernize openstack rbac - https://phabricator.wikimedia.org/T330759
[08:35:54] <dcaro>	 getting "2023-03-07 08:34:18.613 2793859 INFO flask_keystone [-] Couldn't authenticate user 'None' with X-Identity-Status 'Invalid'" in the logs
[08:39:12] <dcaro>	 kindrobot:  just to verify, you were able to access that url without specifying any auth before?
[08:39:28] <dcaro>	 (or it's the auth that's not working?)
[08:42:16] <dcaro>	 (because I see 200s from authenticated requests)
[09:37:46] <dcaro>	 !log admin Changing ceph crush map to allow rack HA on eqiad1 cluster (T331141)
[09:37:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[09:37:53] <stashbot>	 T331141: Change crushmap in eqiad to have rack HA - https://phabricator.wikimedia.org/T331141
[10:11:06] <dcaro>	 !log there was a little unavailability for some VMs while ceph was starting to rebalance things, but it seems stable and moving data around (T331141)
[10:11:07] <stashbot>	 dcaro: Unknown project "there"
[10:11:08] <stashbot>	 T331141: Change crushmap in eqiad to have rack HA - https://phabricator.wikimedia.org/T331141
[10:11:16] <dcaro>	 !log admin there was a little unavailability for some VMs while ceph was starting to rebalance things, but it seems stable and moving data around (T331141)
[10:11:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[11:06:27] <taavi>	 legoktm: apt-browser seems to be missing some packages, for example https://apt-browser.toolforge.org/buster-wikimedia/thirdparty/kubeadm-k8s-1-22/ is missing docker-ce which is in that component
[11:08:59] <taavi>	 !log toolsbeta upgrading kubernetes to 1.22 T286856
[11:09:04] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL
[11:09:04] <stashbot>	 T286856: Upgrade Toolforge Kubernetes to latest 1.22 - https://phabricator.wikimedia.org/T286856
[13:18:25] <andrewbogott>	 !log baserow turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:18:28] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Baserow/SAL
[13:22:23] <andrewbogott>	 !log capacity-exchange turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:22:25] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Capacity-exchange/SAL
[13:23:34] <andrewbogott>	 !log citelearn turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:23:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Citelearn/SAL
[13:26:45] <andrewbogott>	 !log etytree turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:26:47] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Etytree/SAL
[13:28:38] <andrewbogott>	 !log huwiki-dev turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:28:41] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Huwiki-dev/SAL
[13:30:41] <andrewbogott>	 !log mariadb104-test turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:30:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mariadb104-test/SAL
[13:33:16] <andrewbogott>	 !log ml-collab-2022 turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:33:19] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Ml-collab-2022/SAL
[13:33:47] <andrewbogott>	 !log mobile turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:33:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mobile/SAL
[13:36:09] <andrewbogott>	 !log mwcli turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:36:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Mwcli/SAL
[13:37:56] <andrewbogott>	 !log netops-clab turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:37:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Netops-clab/SAL
[13:38:08] <andrewbogott>	 !log openocr turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:38:10] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Openocr/SAL
[13:39:14] <andrewbogott>	 !log schematreerecommender turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:39:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Schematreerecommender/SAL
[13:40:52] <andrewbogott>	 !log services turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:40:55] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Services/SAL
[13:44:23] <andrewbogott>	 !log wikidumpparse turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:44:27] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikidumpparse/SAL
[13:46:08] <andrewbogott>	 !log wikiloop turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:46:11] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikiloop/SAL
[13:46:41] <andrewbogott>	 !log wikinewsie turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:46:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikinewsie/SAL
[13:48:56] <andrewbogott>	 !log wikipathways turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:48:58] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wikipathways/SAL
[13:49:56] <andrewbogott>	 !log wmde-dashboards turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:49:59] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wmde-dashboards/SAL
[13:51:05] <andrewbogott>	 !log wmde-templates-alpha turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:51:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wmde-templates-alpha/SAL
[13:51:17] <andrewbogott>	 !log wmdeanalytics turning off VMs are per https://wikitech.wikimedia.org/wiki/News/Cloud_VPS_2022_Purge -- project appears abandoned
[13:51:20] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Wmdeanalytics/SAL
[14:23:06] <kindrobot>	 dcaro: thanks for getting back. I'm heaving on meeting this morning. When are you done for the day?
[14:23:09] <kindrobot>	 *heavy
[14:30:22] <dcaro>	 in ~4h
[14:48:04] <legoktm>	 taavi: can you file a bug please?
[14:48:40] <taavi>	 legoktm: yes, where? I don't see a phab project for that
[14:50:26] <legoktm>	 Just Tools for now, I'll create a project later
[14:50:56] <taavi>	 sure
[14:54:52] <taavi>	 T331443
[14:54:54] <stashbot>	 T331443: apt-browser missing some packages - https://phabricator.wikimedia.org/T331443
[15:18:28] <kindrobot>	 dcaro I just tried again and got a new error: Bad Gateway
[15:19:39] <kindrobot>	 This is the tool I'm using to make the request (in CI) https://gitlab.wikimedia.org/repos/abstract-wiki/mkwebproxy
[15:20:27] <taavi>	 kindrobot: at least the bad gatewaey errors are now gone
[15:21:21] <kindrobot>	 Alright, now I'm back to "403 Forbidden". ;)
[15:21:49] <taavi>	 hm, let me take a look?
[15:22:40] <kindrobot>	 Thanks!
[15:22:40] <taavi>	 weirdly I see this in the logs: `Enforcing policy proxy:create for user kindrobot (projectadmin) and project wikifunctions`, but I'd expect that in addition to projectadmin you would have some other roles in that project
[15:23:03] <taavi>	 do you have some restrictions set in the application credential?
[15:23:56] <kindrobot>	 No I do not.
[15:24:29] <taavi>	 oh, now I see what's wrong
[15:24:41] <kindrobot>	 Oh? :D
[15:25:48] <taavi>	 andrewbogott: this is T330759 breaking stuff. application credentials are assigned roles they have access to, so changing projectadmin policies to use the member role causes old application credentials to stop working
[15:25:52] <stashbot>	 T330759: Modernize openstack rbac - https://phabricator.wikimedia.org/T330759
[15:27:22] <kindrobot>	 Is there a workaround I can do for now? This is blocking our CI.
[15:28:03] <taavi>	 you can create an application credential that has 'member' role instead of 'projectadmin' assigned
[15:28:51] <kindrobot>	 Oh, that would actually be preferable! Is there something I have to do to give it access to make web proxies?
[15:29:25] <andrewbogott>	 taavi: is it possible for us to reassign those application credentials directly?
[15:29:58] <andrewbogott>	 Or at least get a list of them?
[15:30:06] <taavi>	 no, just like you did not need anything special for projectadmin to work. those two are essentially the same thing but with a different name, we're moving our configuration to be closer to what upstream openstack does
[15:30:23] <andrewbogott>	 taavi: was that to me or to kindrobot ?
[15:30:42] <taavi>	 andrewbogott: at least keystone has a database table `application_credential_role`, I wonder if we can just mass update that or if there will be some caching as well
[15:30:53] <taavi>	 kindrobot: or if you wait a minute you can test our proper solution
[15:31:03] <andrewbogott>	 taavi: I'm looking for cli options.
[15:31:47] <taavi>	 andrewbogott: I don't see anything that would let you do that
[15:32:33] <andrewbogott>	 taavi: I have a script I used to detect all user/projectadmin roles and create equivalent member roles. Likely I can adapt that to do the same with application creds.
[15:32:36] <andrewbogott>	 Let me hack a bit...
[15:32:51] <andrewbogott>	 kindrobot: OK hanging around a half hour or so?
[15:39:02] <andrewbogott>	 kindrobot, taavi, what project are we talking about?
[15:39:12] <taavi>	 wikifunctions
[15:39:29] <andrewbogott>	 thx
[15:41:20] <andrewbogott>	 any idea why
[15:41:23] <andrewbogott>	 https://www.irccloud.com/pastebin/DAjgUSSI/
[15:41:25] <andrewbogott>	 ?
[15:41:48] <taavi>	 those are on kindrobot's user, not on novaadmin I assume
[15:42:17] <andrewbogott>	 hmmm
[15:43:00] <andrewbogott>	 Ah, I see them.  OK...
[15:48:01] <kindrobot>	 taavi, I can wait to test it out. Which role is more "correct" to use for things like CI going foward?
[15:52:37] <andrewbogott>	 kindrobot: 'member'
[15:52:56] <andrewbogott>	 taavi: if I delete the existing cred record and create a new updated one will that still work, or are users persisting the ID of the old cred somehow?
[15:53:41] <taavi>	 andrewbogott: the credential id (or the secret) changing would mean that people would need to update their configuration, yes
[15:53:50] <andrewbogott>	 bah
[15:54:07] <andrewbogott>	 https://www.irccloud.com/pastebin/5Klzlxg5/
[15:54:17] <andrewbogott>	 So that's a bit of a dead end
[15:54:27] <andrewbogott>	 I guess I will try the mysql approach
[15:57:17] <andrewbogott>	 kindrobot: I see three cred sets, aw-gitlab-bot, aw-gitlab-bot-admin, proxy-maker -- which one shall I use for my test case?
[15:57:54] <kindrobot>	 I'm using proxy-maker at the moment.
[15:57:59] <andrewbogott>	 'k
[16:00:01] <andrewbogott>	 kindrobot: try now?
[16:00:12] * andrewbogott ran update application_credential_role set role_id='38676f30eaeb44518bf7e144a73c8da6' where role_id='4d8cad783d6342efa8414d7d36fbc034' and application_credential_id='58';
[16:00:49] <kindrobot>	 It works!
[16:01:34] <kindrobot>	 Thank you!
[16:01:35] <andrewbogott>	 ok.  taavi how do you feel about me making that substitution everywhere?
[16:01:48] <andrewbogott>	 update application_credential_role set role_id='38676f30eaeb44518bf7e144a73c8da6' where role_id='4d8cad783d6342efa8414d7d36fbc034';
[16:01:54] <andrewbogott>	 and then
[16:02:24] <andrewbogott>	 update application_credential_role set role_id='f75a3c410bca4e96a1cf6ac103b0ccaf' where role_id='f473273fac7146b3bdbf22e5d4504f95' 
[16:02:29] * andrewbogott hoping you will double-check those IDs
[16:02:53] <taavi>	 let's take an extra backup first, and try it after that?
[16:03:03] <taavi>	 and sure, give me a sec for that
[16:03:54] <taavi>	 the IDs look correct
[16:04:34] <andrewbogott>	 mysqldump -u root keystone > ~andrew/keystonebackupforserviceroles.sql
[16:04:34] <andrewbogott>	 done
[16:05:05] <andrewbogott>	 taavi: Do you have time to update the docs with the new role names?
[16:05:28] <andrewbogott>	 Soon I'll delete those old roles and then it will be less confusing... but I don't want to delete them until some more bugs shake out
[16:06:43] <andrewbogott>	 !log admin updated application credential roles, replacing 'user' with 'reader' and 'projectadmin' with 'member':  update application_credential_role set role_id='f75a3c410bca4e96a1cf6ac103b0ccaf' where role_id='f473273fac7146b3bdbf22e5d4504f95'  and update application_credential_role set role_id='38676f30eaeb44518bf7e144a73c8da6' where role_id='4d8cad783d6342efa8414d7d36fbc034'
[16:06:48] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[16:08:10] <andrewbogott>	 kindrobot: your other creds should work now too. If you have a moment to test it would be nice for me to know for sure :)
[16:15:40] <kindrobot>	 Sure!
[16:16:49] <kindrobot>	 It works. :)
[16:17:26] <kindrobot>	 I appreciate you both :D
[16:20:44] <andrewbogott>	 I broke it, seems only fair to also fix it :)  Thanks for testing
[16:27:00] <taavi>	 andrewbogott: went ahead and updated https://wikitech.wikimedia.org/wiki/Help:Cloud_services_user_roles_and_rights. although I suspect we're going to get very confused about these at some point
[16:27:42] <andrewbogott>	 taavi: you mean because 'member' is such a bad name?
[16:28:13] <taavi>	 yes, and because we previously used member for one thing in that documentation and now it's for a different thing
[16:28:28] <taavi>	 and viewer is somewhat undescriptive because it can also access VMs
[16:29:31] <andrewbogott>	 taavi: yeah, it's bad news since 'member' used to refer to a 'user' or a 'projectadin'
[16:29:36] <andrewbogott>	 and now it's just one of those things :/
[16:29:53] <andrewbogott>	 I still think it's going to be better to use the upstream standards but I don't love it.
[16:31:24] <taavi>	 openstack-browser is now using the new names
[16:31:39] <taavi>	 and there's a patch for striker pending review since it adds members to the toolforge project
[16:32:04] <andrewbogott>	 that's great!
[22:42:43] <mutante>	 I am trying to get a "phorge" (phab-fork) to work on a cloud VPS instance. It keeps telling me I have "unresolved setup issues", where most of them made sense to me and I could fix.. but 2 of them give me pause, especially "Compressed Requests Not Received Properly" "This software sent itself a test request that was compressed with "Content-Encoding: gzip", but received different bytes than it 
[22:42:49] <mutante>	 sent.".. mod_deflate is loaded.. and nothing changes about this if I unload deflate. It feels to me like this could be due to being behind domainproxy. Ever heard of issues related to compression for webservers behind it?
[22:43:18] <mutante>	 wish I could just test it really quick with a public IP to confirm or deny that
[22:43:48] <herzog>	 https://phorge.wmcloud.org/ ?
[22:43:55] <mutante>	 it's all default debian httpd
[22:43:57] <mutante>	 herzog: yea
[22:44:24] <mutante>	 deflate.conf and deflate.load are in mods-enabled
[22:44:40] <mutante>	 and I tested what happens if I unload it.. no change at all in what phorge setup wizard claims
[22:45:54] <mutante>	 herzog: the whole thing looks kind of broken 
[22:46:29] <mutante>	 working but also broken :) if you see what I mean
[22:46:37] <herzog>	 I've never managed a VPS intance
[22:46:56] <herzog>	 but yes I get you
[22:47:05] <mutante>	 I have, but I did not have this problem with other applications 
[22:47:15] <mutante>	 that also use httpd (apache) and are behind this proxy
[22:47:51] <herzog>	 account creation seems to work though
[22:48:00] <mutante>	 the gzip thing is odd, usually I dont have to configure anything about it
[22:48:05] <mutante>	 when installing default apache2 package
[22:49:02] <mutante>	 herzog: yea, but look at HTML source
[22:49:15] <mutante>	 it starts with ... <div class="phabricator-nav has-local-nav" id="UQ0_76">?
[22:49:39] <mutante>	 not with normal headers.. like wtf
[22:49:47] <herzog>	 <!DOCTYPE html><html><head><meta charset="UTF-8" /><title>♟ Wait For Approval</title><meta name="viewport" content="width=device-width, etc.
[22:49:52] <mutante>	 wait a second..
[22:49:59] <herzog>	 and everything in one line
[22:50:01] <herzog>	 geez
[22:50:08] <mutante>	 well, I see different things because I am logged as initial admin
[22:50:12] <mutante>	 in the setup wizard
[22:51:21] <mutante>	 here is the CSS https://phorge.wmcloud.org/res/defaultX/phabricator/f538846d/core.pkg.css
[22:51:33] <mutante>	 but it looks like no CSS exists
[22:52:31] <herzog>	 geez, some spacing, /* descriptions */ etc would be good
[22:53:43] <mutante>	 it's like the actual application works, I could create users, edit the menu bar, change config.. everything
[22:53:50] <mutante>	 just why does it look like crap :)
[22:54:00] <mutante>	 and why does it claim the compression thing
[22:54:11] <mutante>	 and do these 2 things have any relation.. oh well
[22:54:50] <mutante>	 I did not mean to spam this channel with all of that.. just because if I could I would take the proxy out of the equation for a second just to see if that changes anything
[22:56:00] <bd808>	 mutante: would it work to setup a socks5 tunnel with ssh to the instance so you can bypass the front proxy?
[22:57:11] <mutante>	 here is an interesting thing. If I just select the entire HTML source.. save it to a local file.. and then open it as file:/// in my browser.. I get the normal design with the CSS applied
[22:57:31] <mutante>	 bd808: oh, that's an idea, yea, I can try that. thanks
[23:00:47] <mutante>	 feels like it might have to do with CSS and JS being loaded from a src="http://phorge.wmcloud.org/... rather than a relative path.  be right back
[23:04:22] <mutante>	 I am doing the socks5 proxy via bastion-restricted but so far no difference.. trying other things
[23:09:32] <mutante>	 oh, I found something:) so it has a config setting:   "phabricator.base-uri": "http://phorge.wmcloud.org/",
[23:09:55] <mutante>	 I tried to just change that to "phabricator.base-uri": "https://phorge.wmcloud.org/",  and bam.. it works
[23:10:10] <mutante>	 so https instead of http in the URLs that lead to CSS and JS
[23:10:57] <mutante>	 even though I can totally open the http:// versions from home as well
[23:11:11] <wm-bb>	 <lucaswerkmeister> ah, but CSP probably blocks them
[23:11:26] <mutante>	 but if it works with https.. I will be fine with that :)
[23:11:27] <wm-bb>	 <lucaswerkmeister> I think that might be added by wmcloud
[23:11:37] <mutante>	 aha, thanks Lucas_WMDE
[23:12:23] <mutante>	 herzog: look at it now
[23:12:42] <herzog>	 mutante: I see nothing until my account is approved
[23:12:52] <herzog>	 for the little I see, it's the same as before :)
[23:14:38] <mutante>	 herzog: for the record, I see your user and for me everything looks MUCH MUCH better now.. now I just gotta figure out how to approve you..
[23:15:03] <mutante>	 herzog: done
[23:15:05] <herzog>	 the frontpage hasn't changed if it helps :)
[23:15:14] <mutante>	 https://phorge.wmcloud.org/T1
[23:15:31] <herzog>	 Looks nice
[23:15:34] <mutante>	 well, does it look like a phabricator or does it look like just some black text on white background ?
[23:16:24] <herzog>	 it looks like a phab to me
[23:17:54] <mutante>	 great. then you never even saw how broken it looked to me until now :)
[23:18:06] <mutante>	 without CSS and JS
[23:19:07] <mutante>	 well, I can create a ticket. so my DB also works. that's sucess for now. back to the unrelated setup issues. thanks all
[23:19:21] <herzog>	 Phriction <-- lol
[23:19:37] <mutante>	 haha
[23:19:56] <mutante>	 https://phorge.wmcloud.org/w/ :p
[23:20:02] <mutante>	 it's even /w/
[23:31:04] <mutante>	 !log devtools - phorge-1001 - MariaDB [(none)]> SET GLOBAL sql_mode = "STRICT_ALL_TABLES,STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION"; 
[23:31:07] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL
[23:33:20] <mutante>	 !log devtools - phorge-1001 - MariaDB [(none)]> SET GLOBAL local_infile=0;
[23:33:22] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL
[23:34:28] <mutante>	 !log devtools - phorge-1001 - MariaDB [(none)]> SET GLOBAL max_allowed_packet=33554432;
[23:34:31] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Devtools/SAL
[23:37:07] <mutante>	 related to earlier, it also shows this lovely warning "This server thinks that you are using HTTP, but your client is convinced that it is using HTTPS. This is a serious misconfiguration with subtle, but significant consequences."
[23:43:25] <bd808>	 mutante: the reverse proxy does send an X-Forwarded-Proto header, but as I recall Phab doesn't look for that.
[23:44:15] <bd808>	 Evan made a lot of highly opinionated choices that I imagine will persist in phorge
[23:46:15] <mutante>	 bd808: ACK, thank you
[23:47:21] * mutante finds "Quora - Why is Phabricator such a pain to setup?" hehe
[23:53:27] <herzog>	 lol
[23:58:46] <roy649>	 Is whining about phabricator fair game for this channel?
[23:59:13] <roy649>	 If so, let me know, because I've got some stuff saved up that I'd like to whine about :-)