[06:02:58] 10Mail, 10Infrastructure-Foundations, 10MediaWiki-Email, 10PHP 8.0 support, and 2 others: Incorrect email headers using PHP mail in PHP 8.0 - https://phabricator.wikimedia.org/T320864 (10tstarling) Hi @Anomie, how's it going? I couldn't reproduce this with the ssmtp variant of sendmail. I think the commen... [07:34:01] XioNoX, topranks: it seems not only Azure... CF too :) https://blog.cloudflare.com/making-peering-easy-with-the-new-cloudflare-peering-portal/ [07:35:43] Cool. Makes sense I guess, but me a real burden managing for those massive networks [07:35:50] volans: funnily we just got a few emails here and there from them asking us if we want to peer at X IXP [07:36:16] It’d be great if there was some “openconfig” effort to define a standard [07:36:53] I mean standard api that everyone could implement [07:36:58] do you think they are running the sre.network.peering too? :-P [07:37:10] clearly [07:38:06] there are two competing ones to manage the IX side of things https://ix-api.net/ [07:38:50] I forgot what the other one was, so maybe this one is winning [07:38:55] lol [07:39:14] https://www.cloudflare.com/partners/peering-portal/ [07:39:23] "In submitting this form, you agree to receive information from Cloudflare related to our products, events, and special offers. You can unsubscribe from such messages at any time." [07:39:52] that's not GDPR compliant :) [07:40:01] Hmm cool (ix api not spam) [07:42:18] Seems it wouldn’t be hard to define the semantics for a direct peering one. Add peer, remove peer, update peer, suspend peer. Could use peering db IDs for exchanges. [07:45:16] MS have one: https://learn.microsoft.com/en-us/rest/api/peering/peerings [07:46:15] yeah looks like the REST version of the ones I shared yesterday [07:46:53] I found the powershell version before the Portal one :) [07:50:12] Heh yeah. Good to see there is an open API too. I thought the PowerShell library was a bit sneaky trying to make people use PowerShell [07:50:21] If that’s the only thing they offered [07:56:03] I think longer term peering-manager is the most promising, but still lacks a public API/portal feature [08:03:08] hmm, I always thought of that as a way to manage peering internally [08:03:37] have they defined a standard API interface for ASNs to expose? [08:08:37] not that I'm aware of, so any popular open source one could make a difference, or we should create an IETF working group [08:09:14] haha yeah that latter thought did occur to me :P [08:09:35] If peering-manager added a public-API and defined it well it could become the "de facto" standard [08:10:38] yeah exactly [08:12:24] mmmh am I reading it wrong or they have an API and support also ix-api? [08:12:27] https://demo.peering-manager.net/api/docs/ [08:13:06] they do [08:13:21] but from what I can tell it doesn't support operations similar to the Microsoft one above [08:13:33] i.e. that an external ASN can connect to and request a peering session with [08:14:40] you can add a peering with it - but you wouldn't expose that whole API publically [08:14:49] ack [08:15:48] and probably the "request peering" function would be gated somehow, maybe not, but you could imagine say for us a request to add a session would generate something in peering-manager that me or Arzhel would manually review and click "ok" on, rather than external parties able to directly add stuff that gets pushed out to routers [08:16:24] sure sure [08:16:39] you know... it's open source... patches welcome :D [08:17:05] The MS API looks great, but that got me thinking of the pain of having to support every single ASN's own custom-defined API [08:17:10] indeed :) [08:28:22] yeah, that's where standards should kick in :) [08:29:50] the other main missing feature (but is present in https://github.com/paravoid/peerassist/ ) is to display peering opportunities (eg. we're peering with an AS in X but not in Y while're we're both present) [08:30:18] and maybe notifications on new peers at IXs [08:53:59] hmm yeah, doesn't sound impossible to add [09:02:29] looks like the cloudflare portal is not working as expected "GraphQL error: Access to ASN 14907 is not allowed with current account with email: ayounsi@..." but only when I click the "peer" button. Of course they don't have a logout button [09:05:59] are you logged in with the CF account or the peeringdb one? [09:06:08] I tried peeringdb [09:06:17] and now tried with google [09:06:19] and it works [09:13:40] I keep being impressed, Microsoft doesn't want me to login or create an account using my corp email. And if I force it I get redirected to okta and that says I'm not allowed. [09:14:38] you're trying to login from the the wrong OS :-P [09:16:40] that's quite the captcha: pick a box that contains two identical objects, and I have to do it 5 times... https://usercontent.irccloud-cdn.com/file/rOmx5JHX/Screenshot%202022-10-20%20at%2011-15-17%20Ajouter%20des%20informations%20de%20s%C3%A9curit%C3%A9.png [09:20:15] 10netbox, 10Infrastructure-Foundations, 10Observability-Alerting, 10User-fgiunchedi: Investigate longer run time for hiera_export netbox script - https://phabricator.wikimedia.org/T319299 (10fgiunchedi) My understanding is that all scripts were (are?) taking longer to execute and not only this one, is that... [09:27:14] some captcha alright... next up "spot 10 differences in these pictures" [09:27:37] the Azure dashboard is driving me crazy [09:28:34] XioNoX: maybe it knows that only robots have enough patience to do that, so everyone who answers all five will be a robot [09:38:26] you did it! [09:41:13] the steps are so wild [09:41:31] and so far it's done with a personal email, dunno if/how they can approve it [09:41:34] let's see [09:45:30] hmmm... that's actually an interesting part of the api discussion, how to authorize who is requesting a session [09:46:12] PeeringDB oauth [09:46:24] you can link your account to the main Wikimedia element [09:46:28] I guess if you could publish a public key via the issuing RIR for the ASN/prefix. I guess rpki has some of that infra but lots of ASNs just let RIR handle it [09:46:34] ... and we're announcing PeeringSEC the new extension for Peering sessions... [09:46:36] or that yeah [09:46:43] lol [09:47:21] it would just take a decade or so to get all the network devices upgraded to support it [09:47:40] but then can peeringdb become too big to fail [09:47:51] I like the RIR way [09:48:09] yeah. and you turn it from "informational" to authoritative [09:48:17] it could also be an email with a confirmation link to the noc or peering email address [09:48:21] but I guess right now it's done on email so easy to spoof [09:48:30] heh [09:48:58] given you need to be at an IX, paying for service, registered company the surface for attacks is probably low [09:49:35] or leverage the IX-API [09:49:48] yeah exactly [09:49:49] fetch from the IX who is authoritative [09:49:49] or you just require that the requester already set their side, so if you get a request for AS# you try to connect, if that works fine, if it doesn't you discard it and ofc check that the IPs are part of AS# [09:49:55] authorize at that level for the IX [09:50:04] good idea [09:50:28] or just do it all via BGP [09:50:33] with a new attribute :) [09:50:44] 100% that is the way it will happen in the end :D [09:53:14] during that time... "for i in `cat interesting-sv1-networks.txt`;do sudo cookbook sre.network.peering email $i ;done" [10:08:47] 10netops, 10Infrastructure-Foundations, 10SRE: Ramp up SV1 IXP - https://phabricator.wikimedia.org/T321193 (10ayounsi) > I highlighted some noticeable SV1 peers as well in T280202#7766440 so we should reach out to them. 14 peering requests sent to those noticeable in SV1 but not in SV8 networks [12:32:50] 10Mail, 10Infrastructure-Foundations, 10Znuny, 10serviceops-collab, and 2 others: Incident: 20221017 - mx and vrts - https://phabricator.wikimedia.org/T321135 (10LSobanski) [13:34:16] Wow, Sprint agreed to peer at all 5 IXPs we have in common [13:46:07] that's amazing [13:47:21] paravoid: we're receiving 64566 prefixes from them in eqiad, about 2000 active [13:47:42] curious to know how/if that's going to be visible on our transit/peering graphs [14:03:57] wow :O cool [14:13:41] nice :) [14:15:39] What's the set of prefixes they are sending I wonder? [14:16:00] are they all direct downstream customers of theirs? [14:26:04] 10SRE-tools, 10Icinga, 10Infrastructure-Foundations: get-raid-status-perccli should allow for commands to return non-zero exit code - https://phabricator.wikimedia.org/T320998 (10jcrespo) FYI, For the rebuilt after a disk change, the utility is working: ` sudo /usr/local/lib/nagios/plugins/get-raid-status-p... [14:42:01] topranks: https://w.wiki/5qdB looks like it, yeah. Main one is Telmex. [15:48:18] 10Mail, 10Data-Engineering-Operations, 10Data-Engineering-Planning, 10SRE: Change the analytics-alerts email alias to a mailman distribution list - https://phabricator.wikimedia.org/T315486 (10BTullis) [15:49:38] 10Mail, 10Data-Engineering-Operations, 10Data-Engineering-Planning, 10SRE: Change the analytics-alerts email alias to a mailman distribution list - https://phabricator.wikimedia.org/T315486 (10BTullis) > Mostly, don't worry about sudo, I can create it for you. Just noting that it will be "whatever-name-ale... [15:50:22] 10Mail, 10Data-Engineering-Operations, 10Data-Engineering-Planning, 10SRE: Change the analytics-alerts email alias to a mailman distribution list - https://phabricator.wikimedia.org/T315486 (10BTullis) 05Resolved→03Open [15:50:48] 10Mail, 10Data-Engineering-Operations, 10Data-Engineering-Planning, 10SRE: Change the analytics-alerts email alias to a mailman distribution list - https://phabricator.wikimedia.org/T315486 (10Dzahn) Thank you very much for doign this, @BTullis [15:52:33] 10Mail, 10Data-Engineering-Operations, 10Data-Engineering-Planning, 10SRE: Change the analytics-alerts email alias to a mailman distribution list - https://phabricator.wikimedia.org/T315486 (10Dzahn) @BTullis Done! Please check your mail. ` [lists1001:~] $ sudo mailman-wrapper create --owner btullis@wik... [16:13:45] 10netops, 10Infrastructure-Foundations, 10SRE: Ramp up SV1 IXP - https://phabricator.wikimedia.org/T321193 (10ayounsi) 05Open→03Resolved a:03ayounsi Mass emailing is done: * SV8 peers that are only in SV8 -> Told them they can delete the SV8 sessions * SV8 peers that are also in other IXPs but all sess... [16:24:24] FYI I'm rebooting netbox-dev2002 [16:29:22] 10Mail, 10Data-Engineering-Operations, 10Data-Engineering-Planning, 10SRE: Change the analytics-alerts email alias to a mailman distribution list - https://phabricator.wikimedia.org/T315486 (10Dzahn) The very last step would then be to remove the line from the puppetized exim aliases in the private repo. [20:47:38] 10SRE-tools, 10Infrastructure-Foundations: Netbox accounting report: exclude removed hosts - https://phabricator.wikimedia.org/T320955 (10wiki_willy) Hi @Volans - we have one thought (and we're totally open to feedback, pros, cons, etc) that's somewhat tied to T310594. What if we were to change the status of... [20:59:11] 10netbox, 10DC-Ops, 10Infrastructure-Foundations: Netbox: investigate custom status - https://phabricator.wikimedia.org/T310594 (10wiki_willy) I think the naming convention of the current states are fine the way they are. In terms of adding additional states, we were wondering if it makes sense to create a...