[00:53:29] !log anticomposite@tools-bastion-13 tools.stewardbots stewardbots/StewardBot/manage.sh restart # replaying old EventStreams [00:53:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [15:11:54] o/, I'm getting a 401 when I go to horizon.wikimedia.org (just a blob of JSON, no website with login or anything), am I doing something wrong? and/or should I create a ticket? [15:13:06] kamila_: it works for me :-( [15:13:15] are you logged into idp? [15:13:21] I am [15:13:55] I also tried logging out and back in on idm.w.o, but I don't know if that's sufficient [15:14:20] do you get the same if you visit other related URL? example: https://horizon.wikimedia.org/project/ [15:14:41] yep [15:15:03] what's in the JSON you get? [15:15:16] any debug message or any other hint? [15:15:45] {"error":{"code":401,"message":"The request you have made requires authentication.","title":"Unauthorized"}} [15:15:51] huh I think this is the same issue that was reported by a user on cloud@ [15:16:49] right [15:16:53] and we never followed up [15:17:20] I was hoping it was a temporary problem :D [15:17:38] could it be that the user is not attached to any project, and that's the error? [15:17:53] a.ndrew is the horizon expert but he's out today I think [15:18:39] I see kamila is in several projects [15:18:42] https://usercontent.irccloud-cdn.com/file/BWExW7DK/image.png [15:19:13] let me open a phab ticket [15:19:16] thank you! [15:19:31] I was just added to the project now btw [15:19:35] in case that's relevant [15:19:36] oh [15:19:40] which one? [15:19:48] appservers [15:20:01] I'm checking the horizon logs in cloudweb1003, but I don't see the 401s there [15:20:03] and the form also threw an error, j.ayme said [15:20:11] but I got added [15:21:17] kamila_: could you please paste a screenshot at T388137 ? [15:21:17] T388137: horizon: some users get 401 unauthorized - https://phabricator.wikimedia.org/T388137 [15:21:22] 16:02 hmm...Danger: There was an error submitting the form. Please try again. [15:21:22] oh, you're in the list now :D [15:21:34] arturo: will do [15:22:14] thanks [15:24:27] kamila_: can you expand the idp attributes, to see the memberOf list' [15:26:01] kamila_: do you have a second network you could try with, for instance switch from home wiki to phone hotspot? [15:29:45] I also suggest retrying in a private browser tab to make sure you don't have a bad cookie someplace [15:33:16] home network and mobile hotspot are the same [15:34:25] private window too [15:34:29] * kamila_ pastes to task [15:36:09] ok. For context: the auth process requires accessing openstack.eqiad1.wikimediacloud.org port 25000, I'm finding that that port is blocked on some people's networks (including one of mine at the moment) [15:36:10] arturo: I have memberOf [cn=ops,ou=groups,dc=wikimedia,dc=org, cn=wmf,ou=groups,dc=wikimedia,dc=org, cn=project-appservers,ou=groups,dc=wikimedia,dc=org] [15:36:26] andrewbogott: oh, interesting [15:36:40] I can try with a VPN I suppose [16:41:10] kamila_: long shot, but I wonder if it could be something on your machine/browser blocking some ports, and not on your network? [16:42:03] I have a firewall, but open for outgoing [16:43:22] yeah that would be very unusual, but you never know :) [16:44:25] also I see from your screenshot that the 401 is coming from a request on that port (25000) [16:44:33] so you can reach it [16:45:00] can you share the response headers for that 401 request? [16:46:03] andrewbogott: if the 401 is not coming from some intermediate proxy, it's coming from the keystone API, maybe we have some logs? [16:47:48] If so they should be visible on https://logstash.wikimedia.org/app/dashboards#/view/3ef008b0-c871-11eb-ad54-8bb5fcb640c0?_g=h@e78830b&_a=h@21715bf [16:53:31] hmm not finding anything related :/ [16:55:41] me neither [16:59:22] kamila_: I need to eat lunch and then set up some tests here, I will update on task when I have something. Sorry for the difficulty. [16:59:50] andrewbogott: no worries [17:01:22] dhinus: here are the headers (LMK if the format is annoying) https://www.irccloud.com/pastebin/jPEzNpK0/ [17:02:32] oops is the cookie sensitive? O.O [17:02:48] not if it doesn't work [17:02:54] fair :D [17:12:07] kamila_: thanks for the headers, that req-id actually matches something in the logs! [17:12:31] \o/ [17:12:43] [None req-f72077da-71c8-455b-8e0f-4cfbbce87e23 - - - - - -] Could not find user: [17:12:51] oh fun [17:12:57] I promise I exist! :D [17:13:05] the logs include your full name with a wrong unicode encoding [17:13:11] this could be logstash rendering unicode incorrectly... [17:13:11] oh nooooooooooooooooooooo XD [17:13:18] or, it could be the cause of the problem :D [17:14:14] Sou�kov� strikes again [17:17:00] more like Součková this time :P [17:17:12] yeah, that's a familiar string :D [17:17:59] your evil twin is NOT allowed to access Horizon :D [17:18:08] ISO-8859-1 strikes again [17:18:12] in that case [17:18:38] dhinus: excuse me, *I* am the evil twin [17:19:10] * dhinus sends a message to his younger self fixing ISO-8859-1 issues in 2007 [17:19:33] "it's 2025 and this is still broken" [17:19:48] :'D [17:22:28] Is IRC and mailing the only active way? [17:24:41] How can you join a project? [17:34:57] ItsNyoty: to join a project you need to contact the existing members of the project. You can find them listed on https://openstack-browser.toolforge.org/ [17:36:21] Thanks for answering. [19:19:50] !log anticomposite@tools-bastion-13 tools.stewardbots stewardbots/StewardBot/manage.sh restart # disconnected [19:19:53] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL [22:28:04] !log anticomposite@tools-bastion-13 tools.stewardbots Add temp debugging code to StewardBot for eventstreams issue [22:28:06] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools.stewardbots/SAL