[10:43:20] <arturo>	 !log admin [codfw1dev] rebooting cloudgw200[12]-dev for T291813
[10:43:23] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[10:54:15] <arturo>	 !log admin rebooting cloudnet1004/1003 for T291813
[10:54:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL
[11:27:09] <arturo>	 !log admin rebooting cloudgw1001/1002 for T291813
[11:44:19] <arturo>	 !log tools.stashbot restarted bot after network hiccup
[12:33:52] <Kelson>	 Hi, not sure if someone is already working on it, but in case this went unoticed, all our mwoffliner systems seems to have massive network problems. For example, on mwoffliner1, a simple $curl -I http://security.debian.org just fails because of network.
[12:37:20] <musikanimal>	 the status for this channel says "Network hiccups" so I'm guessing it's being looked at. In my case our cron jobs that ping GitHub are all timing out
[12:38:58] <dcaro>	 Kelson: yep, we are working on it 👍
[12:39:13] <Kelson>	 dcaro: thx!
[12:41:38] <dcaro>	 we'll update here when able
[12:50:27] <dcaro>	 Kelson: can you elaborate on the error you are seeing?
[12:50:54] <Kelson>	 $ curl -I http://security.debian.org
[12:50:54] <Kelson>	 curl: (56) Recv failure: Connection reset by peer
[12:51:02] <dcaro>	 thanks
[13:01:37] <magnusmanske>	 In case no one said yet, tools-static throws 502 errors. Many of my tools are broken because of eg https://tools-static.wmflabs.org/cdnjs/ajax/libs/leaflet/1.3.1/leaflet.css
[13:02:12] <majavah>	 we're aware and investigating some major networking issues, thanks
[13:25:53] <majavah>	 things should start recovering now
[13:29:26] <wm-bb>	 <JeanFred> Oh − fine by me I guess − thanks! Although it’s actually not my blog − it belongs to User:PierreSelim and User:Léna and I am just the long-term guest :) (re @wmtelegram_bot: <legoktm> @JeanFred: if it's OK with you, I proposed adding your blog to Planet: https://gerrit.wikimedia.org/r/c/operations/puppet/+/736037)
[14:28:19] <Kelson>	 majavah: dcaro Thank you, all our workers are back oneline now.
[23:35:04] <dschwen>	 Hey folks. I'm getting `requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='commons.wikimedia.org', port=443): Read timed out. (read timeout=45)` mid bot run (pywikibot based commons:User:QICBot) and it hasn't recovered in the last half hour. Any idea if this will recover. Or what went wrong in the first place?
[23:35:36] <dschwen>	 more like an hour now
[23:48:38] <AntiComposite>	 hrm, I'm not seeing anything. do you know which URL is causing problems?