[10:24:55] as FYI I am working on dump-cloud-ip-ranges on puppetserver1001, it seems broken [10:47:22] very weird, it seems that we get a 30x/404 from ms's website, but I can't repro locally [10:47:49] it happens only on puppetserver hosts, as if we were in a special list [10:48:34] tried also with py requests directly, it worked at first but then I started to get 403s [10:54:27] MS ? Microsoft? [10:54:47] that's for the public cloud list building? [10:55:42] exactly yes [10:55:47] GitLab needs a short maintenance break at 12:00 UTC (in one hour) [10:57:07] mmm no ok now I am getting weird results also on my local [10:57:22] the URL is https://www.microsoft.com/en-us/download/details.aspx?id=56519 [11:00:51] I am not 100% sure if this is an issue on MS side or if our requests settings need to be tuned [11:01:32] curl -i on puppetserver1001 returns "HTTP/1.1 200 Connection established" at first, then a 302 to a 404 page [11:01:59] maybe part of the website's page can't render for some reason [11:03:22] interestingly, on puppetserver1001 if I use a plain requests' session() in a python repl I get the page [11:03:39] 200 code [11:05:10] if I change the user agent header with https://github.com/wikimedia/operations-software-pywmflib/blob/master/wmflib/requests.py#L110C17-L110C112 I get a 403 [11:05:13] mah [11:07:26] a simple curl returns a 302 with Location: https://www.microsoft.com/en-us/download/404Error.aspx [11:07:45] so they are definitely varying responses based on UA [11:08:36] a simple curl on my machine that is [11:10:17] judging from headers, that page is behind their CDN and I am willing to be their CDN is doing the best it can to detect and deny bots these days. [11:10:29] I agree yes [11:10:34] which kinda defeats the purpose of that page/file [11:10:39] and they probably have tight settings for bots [11:11:19] but why are you using that page though? [11:11:32] isn't https://download.microsoft.com/download/7/1/D/71D86715-5596-4529-9B13-DA13A5DE5B63/ServiceTags_Public_20250106.json the proper one? [11:12:17] IIUC our python code starts from the id=56519 page, then looks for the download link and tries to fetch it [11:12:19] which, judging from headers is behind something else [11:12:32] a, using it as a discovery? [11:12:35] because I think ti varies over time [11:12:36] yes yes [11:13:24] sigh, because 7/1/D/71D86715-5596-4529-9B13-DA13A5DE5B63 probably changes per day [11:13:40] I tried messing a bit with the date alone in the URL and no dice :-( [11:14:21] This isn't a sustainable game fwiw. Even if you find a combination of things that will work today, it might not tomorrow. [11:15:06] and I am willing to bet it varies on at least 4 different factors. UA, source IP, TLS fingerprinting, HTTP version used. [11:15:29] I agree yes, but I am not sure if MS releases that data in another way [11:16:46] it has been working fine till now, but it was probably due to ms's cdn settings not being touched :D [11:17:50] Maybe reach out to them, someone here must have a contact or something [11:21:32] I'll wait for people to chime in, it is probably not easy to find an email without a contact but I'll try [11:21:56] (going afk for lunch!) [18:34:25] I feel like I must be experiencing some sort of code path blindness. Can anyone hit me with a clue bat explaining where the `include ::eventlogging` in `profile::mediawiki::maintenance` finds its code? I don't see a modules/eventlogging path in my clone or in https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/ [18:37:53] bd808: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1109157 [18:37:56] recently removed [18:40:10] so... puppet is broken on the prod hosts too and not just in beta? [18:41:41] yes [18:41:41] yep, broken on P:mediawiki::maintenance hosts [18:41:43] https://puppetboard.wikimedia.org/node/mwmaint1002.eqiad.wmnet [18:41:44] it does look like that puppet has been broken on mwmaint* since Jan 8th [18:42:23] ottomata: ^ there is a dangling reference to `::eventlogging` in `::profile::mediawiki::maintenance` [18:42:57] * bd808 runs away to lunch [18:44:34] https://gerrit.wikimedia.org/r/1109755 [18:47:31] this seems to be the only reference left [18:48:41] taavi: thanks, will merge this at least and then leave that for ottomata [18:49:50] sounds good [18:52:01] bd808: thanks for the ping, puppet is back on the hosts. [18:52:13] we have two more long running failures on https://puppetboard.wikimedia.org/failures [18:52:27] ms-be105[68] [20:00:05] sukhe: taavi bd808 thanks, sorry! don't know how I missed that! [20:59:11] ottomata: it happens. :)