[02:28:31] <Guest14>	 Привет всем
[10:41:30] <wikibugs>	 10GitLab (CI & Job Runners), 10Data Engineering Planning, 10Release-Engineering-Team, 10Performance Issue: Improve speed of Gitlab CI - https://phabricator.wikimedia.org/T311111 (10EChetty)
[15:33:21] <bd808>	 I'm not sure where to log this for posterity, but I just promoted StrikerBot to be an admin in gitlab.wikimedia.org because I have been testing the new gitlab integration in Striker (https://toolsadmin.wikimedia.org/) using an admin bot account. I am rolling out this functionality to prod today.
[15:33:30] <bd808>	 I did leave an admin note on the gitlab user itself.
[15:41:03] <greg-g>	 bd808: probably releng sal?
[15:41:36] <bd808>	 yeah, can do
[16:40:33] <_joe_>	 dduvall: I am trying to merge your nginx sorcery again
[16:40:44] <_joe_>	 uhm seems to fail again
[16:41:03] <_joe_>	 I'll ask someone in serviceops to set up a poontoon where we can work this out
[16:41:37] <_joe_>	  nginx: [emerg] location "/v2/_catalog" is outside location "^/v2/(.*)" 
[16:45:38] <_joe_>	 maybe we need an anchor?
[16:47:35] <_joe_>	 yeah I might have a fix
[16:48:13] <_joe_>	 but there are more issues
[16:49:47] <_joe_>	 so: I'll just leave puppet disabled on the docker registries, and work on a fix there
[16:49:57] <_joe_>	 I might not be done by tonight
[16:55:26] <dduvall>	 _joe_: strange error, ok
[16:55:31] <dduvall>	 thanks for looking at that!
[16:55:52] <_joe_>	 dduvall: the main issues I see right now are with the handling of /v2/
[16:55:55] <_joe_>	 not sure why tbh
[16:56:07] <dduvall>	 let me check my proof of concept to see if there are differences
[16:57:07] <_joe_>	 i'll give myself half an hour to find this out, then continue tomorrow morning
[17:00:27] <dduvall>	 ok. does it give a line number of anything? that's not a lot to go on, nginx :p
[17:02:39] <_joe_>	 so the issues are
[17:02:49] <_joe_>	 https://docker-registry.wikimedia.org/v2 (/srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml:2)
[17:02:51] <_joe_>	     Location header: expected '/v2/', got 'https://docker-registry.wikimedia.org/v2/'.
[17:02:53] <_joe_>	     Docker-Distribution-Api-Version header: expected 'registry/2.0', was missing.
[17:03:11] <_joe_>	 the second issue is what worries me a bit
[17:04:00] <_joe_>	 https://docker-registry.wikimedia.org/v2/restricted/nonexistent/manifests/latest (/srv/deployment/httpbb-tests/docker-registry/test_docker-registry.yaml:81)
[17:04:02] <_joe_>	     Status code: expected 401, got 404.
[17:04:04] <_joe_>	     WWW-Authenticate header: expected 'Basic realm="docker-registry restricted (restricted-read)"', was missing.
[17:04:38] <_joe_>	 it means we're not getting auth requested on /restricted/
[17:04:47] <dduvall>	 ah, right
[17:04:49] <dduvall>	 that's not good
[17:04:52] <_joe_>	 yep
[17:05:04] <_joe_>	 so yeah, let me rollback, we can't leave it like this
[17:05:07] <dduvall>	 ok, so perhaps the location ~ ^/v2/(.*) is taking precedent
[17:05:13] <_joe_>	 yeah I think so
[17:05:22] <dduvall>	 ugh. so crazy
[17:05:53] <dduvall>	 do you think it's feasible to apply this profile to a labs host?
[17:06:12] <dduvall>	 i would love to work out these kinks prior to a merge next time but i don't have anywhere to test the full config currently
[17:06:13] <_joe_>	 yes
[17:06:22] <_joe_>	 we should just spin up a pontoon instance
[17:06:23] <dduvall>	 alright. let me try to do that today
[17:06:32] <dduvall>	 pontoon?
[17:06:34] <_joe_>	 I'll ask someone in serviceops to do so
[17:06:43] <dduvall>	 alright. that'd be awesome
[17:07:04] <_joe_>	 https://wikitech.wikimedia.org/wiki/Puppet/Pontoon
[17:07:14] <_joe_>	 ok, lemme revert for now
[17:07:17] <_joe_>	 I'll list the issues
[17:07:21] <dduvall>	 i thought regex location matches were lower priority than static ones but nginx surprises me on all fronts
[17:07:29] <_joe_>	 yeah me too...
[17:07:58] <_joe_>	 but I think it depends on the layer that matches
[17:08:06] <_joe_>	 uhm let me try one thing
[17:09:29] <dduvall>	 arg. looks like regex ones are before `location /` type directives (no `=`)
[17:10:04] <dduvall>	 so `location ~ /v2/(.*)` is most likely being matched over `location /v2/restricted/`
[17:10:36] <dduvall>	 i wonder if it supports negative lookahead
[17:10:53] <_joe_>	 ok
[17:11:02] <_joe_>	 turning it into a regex would fix that problem
[17:11:21] <dduvall>	 because it's declared first?
[17:11:24] <_joe_>	 but I'm still not fully confident about all this tbh
[17:11:26] <_joe_>	 yes
[17:11:49] <dduvall>	 there's still the issue with the nested `location = /v2/_catalog` it seems
[17:13:08] <_joe_>	 I fixed that making it a regex too
[17:13:14] <dduvall>	 ah
[17:13:26] <dduvall>	 fun :)
[17:14:05] <_joe_>	 but to my point, I'd like to revisit tomorrow. I will send a patch to fix the /restricted/ stuff and tomorrow morning I'll run a more thorough test
[17:14:32] <dduvall>	 sounds good
[17:14:35] <_joe_>	 I'll leave puppet disabled elsewhere
[17:14:38] * dduvall curses nginx
[17:15:46] <dduvall>	 _joe_: and thank you
[17:17:48] <_joe_>	 dduvall: https://gerrit.wikimedia.org/r/c/operations/puppet/+/830203
[17:18:16] <_joe_>	 I hope the commit message conveys enough hatred for nginx
[17:19:18] <dduvall>	 could be a few more nasty words but i'll accept it as is :)
[17:19:35] <_joe_>	 I'm trying to follow the ERC guidelines 
[17:19:50] <dduvall>	 haha
[17:20:05] <dduvall>	 this is where a good handle on sarcasm is important
[17:20:13] <dduvall>	 gracefully walking the line
[17:20:37] <_joe_>	 being a native speaker makes it easier I hope :)
[17:21:01] <dduvall>	 maybe, though a second language speaker has more plausible deniability :)
[21:07:00] <wikibugs>	 10GitLab (Project Migration), 10Quarry: Move quarry to gitlab or github - https://phabricator.wikimedia.org/T308978 (10rook)
[21:07:50] <wikibugs>	 10GitLab (Project Migration), 10Quarry: Move quarry to gitlab or github - https://phabricator.wikimedia.org/T308978 (10rook)