[06:37:40] good morning, as part of upgrading to Zuul I wanted to have the production system to push some artifacts to the WMCS storage system. And eventually it fails to reach https://openstack.eqiad1.wikimediacloud.org:25000/ [06:38:46] I have filed it as https://phabricator.wikimedia.org/T430479 [06:53:41] greetings [06:59:51] hashar: ack thank you, I'm not sure about wikimediacloud.org being in no_proxy though I'm fairly certain you'd need to use the https proxies from production hosts [07:00:24] in other words the cloud public IPs are not different from any others on the internet in that sense [07:10:13] godog: so I should use webproxy:8080? [07:10:49] hashar: indeed, please try with that and see, it should work [07:11:05] what I don't know if direct access used to work and it doesn't anymore [07:12:51] NO_PROXY='' no_proxy='' curl -x http://webproxy.eqiad.wmnet:8080 https://openstack.eqiad1.wikimediacloud.org:25000/ [07:12:51] curl: (56) CONNECT tunnel failed, response 403 [07:12:55] success! :) [07:13:18] well at least I can connect to the proxy but indeed tunneling is disabled [07:13:42] which I guess make sense, we probably don't want any prod traffic to use tunneling [07:15:33] the ssh tunneling ? yes that's indeed disabled [07:16:16] I misspoke: ssh tunneling is enabled but from the command line, you can't open new tunnels on demand [07:18:30] not ssh tunneling, CONNECT :) [07:19:55] ah yeah of course [07:20:23] what I don't get is curl says that when using --proxy, the requests are transparently converted to http [07:20:35] and one has to pass --proxytunnel to turn on CONNECT [07:20:48] but its behavior doesn't match the doc (it does use CONNECT by default), I will look it up [07:21:15] (though that means credentials will not be encrypted between the host and the proxy) [07:28:56] found it, modules/profile/templates/url_downloader/squid.conf.erb denies CONNECT to port that are not 80 or 443 ;) [07:29:30] and I can reach through the proxy the non port 25000 url https://openstack.eqiad1.wikimediacloud.org/ [07:42:38] heh yes there is a task/discussion about moving to standard ports indeed [07:51:55] well at least the APIS are wide open to the internet so something else is blocking the request from zuul1002.eqiad.wmnet to port 25000 ;) [08:03:54] Morning! [08:11:59] greetings dcaro [08:12:16] could I bother you for a quick/easy review ? https://gerrit.wikimedia.org/r/c/operations/puppet/+/1306159 [08:13:26] sure, 👀 [08:14:02] cheers [08:14:56] +1d [08:15:49] thank you [08:16:43] there's been an increase amount of sessions (7x) hitting toolforge during this weekend, and that triggered ToolforgeWebHighErrorRate again :/ (also, against the same tool) [08:22:48] I found the "issue", the proxying of requests to port out of 80/443 are not allowed. The webproxy did learn to accept some other ports such as Magnum K8s API (added by bd808), so I guess I will add extra ports there and add him as a reviewer ;) [08:43:29] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1306161 :-) [08:45:29] hashar: I recommend engaging more or less the same folks as https://gerrit.wikimedia.org/r/c/operations/puppet/+/1174842 in the sense that the infrastructure foundations is responsible for the proxies [09:40:00] dcaro: just to clarify, the 7x increase was on a single tool only? [09:45:39] aputhin: not exactly, the sessions are shared, it's hitting fourohfour now [09:59:10] So summarizing, there's something trying to hit scholia, when scholia went down, it started hitting fourohfour, and the number of open sessions on haproxy spiked to 7-8x the usual (still high, going down a bit) [10:01:29] https://usercontent.irccloud-cdn.com/file/qhsMvSlg/image.png [10:01:35] https://grafana-rw.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview?orgId=1&from=now-2d&to=now&timezone=utc&var-cluster_datasource=P8433460076D33992&var-cluster=tools [10:02:07] the blue spike on top is scholia, the lower graph is the number of open sessions in haproxy (that is the same haproxy for all tools) [10:04:39] none of that is an issue for other tools per-se, besides using a big chunk of resources (ex. >10MB/s of throughput on fourofour) [11:10:17] * dcaro lunch [14:37:26] dhinus: last week the paws usage alert was easy to clear (everything was in /srv/paws/files-to-remove/) but this week the fix is less obvious; can you have a look in the next couple days? [14:38:29] andrewbogott: sure, although I'm not sure I will find what's going on :D [14:38:43] I think there's some advice in the runbook [14:48:25] dcaro: what do you think about this suggestion to increase loki's timeout to 3h (or less, but more than 10min at least)? do we foresee any risks? (as a temporary solution before we implement proper session auto-refresh) https://phabricator.wikimedia.org/T417225#12066137 [14:50:04] I think it's ok, we have to see if it can be limited to the streaming endpoint only, if so no problem, if it can't then we have to keep an eye on the number of open requests (I think it'll be ok, but well, at least keep it in mind if it spikes) [14:53:44] andrewbogott: where's the runbook? [14:53:59] good point [14:54:45] there's this, but it's not all that helpful https://wikitech.wikimedia.org/wiki/PAWS/Admin#NFS_cleanup [17:15:12] * dcaro off [17:15:14] cya tomorrow!