[05:18:42] btullis: you have puppet changes pending to merge [05:19:02] btullis: looks like this is it: https://gerrit.wikimedia.org/r/c/labs/private/+/713535 [05:19:34] (I haven't merged them and just merged mine) [08:45:48] marostegui: Thanks. Apologies for that. I didn't realise that I had to merge in on the puppetmasters as well, because it worked for my pcc test immediately. Merging it now. [08:49:46] no problem! [16:49:08] Amir1: is everything all done for the dispatch migration to timers? [16:49:44] legoktm: I have some patches for clean up of logging directory and log rotation [16:50:20] and there is rz*l's patch to clean up the switchover script [16:50:48] https://gerrit.wikimedia.org/r/q/bug:T288175+status:open [16:50:48] T288175: Migrate wikibase-dispatch-changes crons to systemd timers - https://phabricator.wikimedia.org/T288175 [16:50:58] woot, I'll take a look [16:59:51] legoktm: yeah, welcome back! sorry to snag that one, I've just been wanting to do it for A Very Long Time [17:00:18] I just wanted to give less work to Kunal :D [17:19:55] rzl: :D I'm not complaining! [17:43:08] legoktm: thanks for the reviews 👍 I'll wait for v.olans to review before merging, since we need him to do the release anyway [17:46:27] ack [17:56:57] rzl: could you take a look at https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/707457 ? [18:01:57] ✅ [19:42:13] jayme: regarding blocking egress (great to hear btw, I wasn't sure whether we were ready for that), do you know if we have something in place for 'truly external' use cases like those that connect with Flickr API and machine translation services? [19:42:37] (context https://phabricator.wikimedia.org/T288848#7287882 ) [19:53:07] is there a timeout on custom-scripted Icinga checks? how long are they allowed to take? [19:54:16] (going to run httpbb hourly or so, against a couple of appservers -- debating between writing a custom check for it, vs. just making it a systemd timer and monitoring that) [20:10:16] Krinkle: those should already be using the webproxy/urldownloader [20:11:44] legoktm: sure, and I assume they do. [20:11:59] ... which means it won't be egress from k8s side, I guess, assuming there's an envoy port to that as well. [20:12:13] perfect [20:12:45] there's no envoy for it, it just hits the url-downloader host directly (over HTTP but in the same DC) https://noc.wikimedia.org/conf/highlight.php?file=ProductionServices.php [20:13:00] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/deployment-charts/+/refs/heads/master/common_templates/0.3/default-network-policy-conf.yaml#13 is the egress for it [20:13:00] but I wonder in that case, have we disallowed our own domains from webproxy, or is use of that opt-in? I would imagine that those external use cases not all have their own configurable ot-in proxy hardcoded [20:14:03] https://github.com/wikimedia/mediawiki-extensions-UploadWizard/blob/b1f76ebf0f9909150b975855dc451d743a8aa614/includes/UploadWizardFlickrBlacklist.php#L138 [20:14:40] https://gerrit.wikimedia.org/g/mediawiki/core/+/dc699eec4de4de861696ae63575246b7386d791d/includes/http/MWHttpRequest.php#232 [20:14:58] Right, so we basically solved this already. [20:14:59] MW uses the proxy if it's not a "local URL", per $wgLocalVirtualHosts [20:15:25] that is, we already have recognised it and distingiushed it for all relevant cases [20:15:31] yeah, we just need a different proxy for local URLs [20:15:32] but would presuambly need to do something slighty differnet for k8s