[07:58:24] 10Puppet, 10Beta-Cluster-Infrastructure, 10Infrastructure-Foundations, 10Release-Engineering-Team, 10Scap: Fatal error: Uncaught ConfigException: Failed to load configuration from etcd - https://phabricator.wikimedia.org/T296125 (10Majavah) >>! In T296125#7518549, @AlexisJazz wrote: >>>! In T296125#75183... [10:04:55] 10puppet-compiler, 10Infrastructure-Foundations, 10User-dcaro: PCC: create the output index once the first host is finished - https://phabricator.wikimedia.org/T295029 (10dcaro) 05Open→03In progress a:03dcaro [10:31:50] 10puppet-compiler, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-dcaro: PCC: add a fast fail option to boil out when the first error is encountered - https://phabricator.wikimedia.org/T295028 (10dcaro) 05In progress→03Open [10:31:53] 10puppet-compiler, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-dcaro: PCC: create the output index once the first host is finished - https://phabricator.wikimedia.org/T295029 (10dcaro) 05In progress→03Open [12:23:06] 10Puppet, 10Beta-Cluster-Infrastructure, 10Infrastructure-Foundations, 10Release-Engineering-Team, 10Scap: Fatal error: Uncaught ConfigException: Failed to load configuration from etcd - https://phabricator.wikimedia.org/T296125 (10Lucas_Werkmeister_WMDE) [13:08:41] topranks: looks like forcing a master change did the trick for B7 [13:08:56] oh wow [13:08:59] nice :) [13:11:12] but now fpc2 is stuck in a line-card role [13:14:48] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Can't commit on asw-b-codfw - https://phabricator.wikimedia.org/T295118 (10ayounsi) The above command doesn't commit on a pre-provisioned VC. I did this instead: ` [edit virtual-chassis member 2] - role routing-engine; +... [13:17:54] alright, kicking it harder did the job [13:21:03] lol.... when you say "kicking it harder" you mean.... requesting switchover angrily? [13:21:21] it didn't need a reboot or anything drastic I assume? [13:21:34] just a bigger hammer :D [13:21:35] I re-did the "set fpc2 as linecard then back as RE" dance [13:22:05] ah ok yeah. [13:22:10] https://repairkit.juniper.mx/800px-Juniper_repair_kit.jpg [13:22:31] to some extent being a network engineer is the art of knowing all the various ways to switch something on and off again ;D [13:22:43] lol [13:23:20] Nice work XioNoX fair play :) [14:58:58] fyi cname at the apex seems i was wrong they are still discussing things. https://datatracker.ietf.org/doc/html/draft-ietf-dnsop-svcb-https-08 seems to be the most recent direction people are heading (with the following expired/abandoned ones https://datatracker.ietf.org/doc/html/draft-ietf-dnsop-svcb-httpssvc-03, https://datatracker.ietf.org/doc/html/draft-ietf-dnsop-aname-04, ... [14:59:04] ... https://datatracker.ietf.org/doc/html/draft-bellis-dnsext-multi-qtypes) [15:00:18] 10puppet-compiler, 10Infrastructure-Foundations, 10Cloud-Services-Origin-Team, 10Patch-For-Review, 10User-dcaro: PCC: create the output index once the first host is finished - https://phabricator.wikimedia.org/T295029 (10dcaro) [15:01:43] also fund: https://blog.cloudflare.com/zone-apex-naked-domain-root-domain-cname-supp/ [15:12:08] thanks John will have a look! [15:12:48] 10Puppet, 10Beta-Cluster-Infrastructure, 10Infrastructure-Foundations, 10Release-Engineering-Team, 10Scap: Fatal error: Uncaught ConfigException: Failed to load configuration from etcd - https://phabricator.wikimedia.org/T296125 (10Jdforrester-WMF) p:05Triage→03Unbreak! Within the context of the Beta... [15:15:19] I guess that SVCB record stuff has the disadvantage that it'll take a long time before all clients support it. Although they're in wide usage due to Cloudflare + Apple supporting AFAIK. [15:16:35] 10Puppet, 10Beta-Cluster-Infrastructure, 10Infrastructure-Foundations, 10Release-Engineering-Team, 10Scap: Fatal error: Uncaught ConfigException: Failed to load configuration from etcd - https://phabricator.wikimedia.org/T296125 (10Urbanecm) >>! In T296125#7520450, @Jdforrester-WMF wrote: > Within the co... [15:18:44] 10Puppet, 10Beta-Cluster-Infrastructure, 10Infrastructure-Foundations, 10Release-Engineering-Team, 10Scap: Fatal error: Uncaught ConfigException: Failed to load configuration from etcd - https://phabricator.wikimedia.org/T296125 (10Jdforrester-WMF) [15:24:18] 10puppet-compiler, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-dcaro: PCC: add a fast fail option to boil out when the first error is encountered - https://phabricator.wikimedia.org/T295028 (10dcaro) 05Open→03in_progress [15:24:54] 10puppet-compiler, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-dcaro: PCC: add a fast fail option to boil out when the first error is encountered - https://phabricator.wikimedia.org/T295028 (10dcaro) 05in_progress→03Open [15:27:01] 10puppet-compiler, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-dcaro: PCC: add a fast fail option to boil out when the first error is encountered - https://phabricator.wikimedia.org/T295028 (10dcaro) 05Open→03In progress [15:27:35] 10puppet-compiler, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-dcaro: PCC: add a fast fail option to boil out when the first error is encountered - https://phabricator.wikimedia.org/T295028 (10dcaro) 05In progress→03Open [15:28:20] 10puppet-compiler, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-dcaro: PCC: add a fast fail option to boil out when the first error is encountered - https://phabricator.wikimedia.org/T295028 (10dcaro) 05Open→03Stalled [15:28:38] 10puppet-compiler, 10Infrastructure-Foundations, 10Patch-For-Review, 10User-dcaro: PCC: add a fast fail option to boil out when the first error is encountered - https://phabricator.wikimedia.org/T295028 (10dcaro) 05Stalled→03Open [15:29:02] topranks: yes thats correct (and from memory would have been for all the proposals) however from a practical PoV theses things happen much quicker we you just have to worry about real humans. as 1) stub resolveres genrally get updated pretty quickly and more users then ever use automatic updates 2) many people use open caches like 8.8.8.8 (which genrally update quickly) 3) as DoH and TRR [15:29:08] become more populer and on by default most useres ... [15:29:10] ... wont even use there stub or configuered resolver. but regardless of all that yes it will take some time to roll out once standardised so not something we can use now [15:30:14] fyi this draft is also geared towards enabling or helping to enable ech (encyprted client hello) so i think manywill want to push it out quickly once its avalible [15:30:42] But the OS needs to support the new record types no? Resolvers should support them as raw "type 64" or "type 65" records without changes. [15:31:02] Ofc these days we have browsers doing their own DNS, so perhaps that'll get adopted quicker. [15:31:25] And indeed as you say given the motivation is at least partly ECH maybe the OS vendors will also roll it out sooner rather than later. [15:35:37] topranks: there are specific on how a cache caches so i dont think caches will support them out the box but yes they may do some dumb proxying, however the big ones will lkley have this implmented before it even makes it as an RFC. In relation to the stub yes he will need updates for the new types but they are normally quick (most of the code will allready exist in a branch somewhere for [15:35:43] testing PoC). [15:36:38] previous experince has shown that if enough of the major players want something then code will start hitting systems before the RFC is published. EDNS client subnet is a good example of how it was working in some forms for a few years before being an RFC (although that had no stub implications) [15:37:16] "fc these days we have browsers doing their own DNS" yes this is on of the big things that will make DNS targeted at https get quicker adoption [15:39:30] In my previous place our bind resolver worked fine when Cloudflare switched on the HTTPS RR type last year, and iOS started using it. [15:39:55] Reported them as "type 65" and the bundled dig couldn't parse the RR data, but it was able to query them upstream and serve clients ok. [15:40:58] 10Puppet, 10Beta-Cluster-Infrastructure, 10Infrastructure-Foundations, 10Release-Engineering-Team, 10Scap: Fatal error: Uncaught ConfigException: Failed to load configuration from etcd - https://phabricator.wikimedia.org/T296125 (10Majavah) >>! In T296125#7520534, @Stashbot wrote: > {nav icon=file, name=... [15:43:31] topranks: indeed bind/unbound/knot resolver i think will all dumb forward unknown types (in fact i think they have to to support enterprise extensions) however that dosn;t neccesarily mean they support it i.e. take into account all the additional caching rules e.g. https://datatracker.ietf.org/doc/html/draft-ietf-dnsop-svcb-httpssvc-03#section-4.2. [15:44:20] indeed that makes sense. [15:45:31] also in relation to bind they are the refrence implmentation daemon, they implment i think every official RFC and often have drafts implmented way before they are RFC's esspecially if they look like they will make it to be one (in some way) unbound at least is much more conservative and dont implment them (i.e. release code) untill the rfc has been published. further the earlier draft of [15:45:37] this was authored by Paul Vixie so can almost ... [15:45:40] ... garuntee his original implmentation was in bind [15:46:41] also unbund only implment standard track + once they like vs all of them i.e. experimental, informational best practice etc [15:47:20] although i know there was some talk about dropping RFC' that make no senses and reliving tention on the camels (https://powerdns.org/dns-camel/) back [15:54:18] XioNoX, moritzm, jbond: can any of you merge https://gerrit.wikimedia.org/r/c/operations/puppet/+/740626/? [15:55:37] majavah: sure merging :) [15:57:34] I've updated https://wikitech.wikimedia.org/wiki/SRE/LDAP/Groups to drop the group from there [15:58:06] thx moritzm [15:58:43] majavah: deployed [15:59:24] and I added orchestrator and librenms to wmf/nda on that same page :P [15:59:40] confirmed that I can log in [16:00:05] majavah: awesome thanks [16:01:10] I also don't see any obvious edit buttons, which is good [16:13:51] 10Puppet, 10Beta-Cluster-Infrastructure, 10Infrastructure-Foundations, 10Release-Engineering-Team, 10Scap: Fatal error: Uncaught ConfigException: Failed to load configuration from etcd - https://phabricator.wikimedia.org/T296125 (10Majavah) [16:14:35] 10Puppet, 10Beta-Cluster-Infrastructure, 10Infrastructure-Foundations, 10Release-Engineering-Team, 10Scap: Fatal error: Uncaught ConfigException: Failed to load configuration from etcd - https://phabricator.wikimedia.org/T296125 (10Majavah) 05Open→03Resolved a:03Majavah [16:15:15] knobs: [16:16:23] either I should not use my phone to try and send messages.. or that should be a catch-all for the whole team :P [16:17:53] jbond: meant to say thanks for the DNS camels link, hadn't seen it before got a good laugh from the name :) [16:30:08] topranks: may find https://www.ietf.org/blog/herding-dns-camel/ intresting as well, im sure there must be a vidoe from bert somewhere as well (and he is a good presenter so worth watchin) but i couldn;t find one unfortunatly [16:31:04] i also came accross https://github.com/ahupowerdns/protocol-camel which allows one to generate DGP and NTP camels as well however i couldn;t find a live page [16:31:14] s/DGP/BGP/ [16:32:13] ahh here is a video :) https://www.youtube.com/watch?v=AquYPF1hvH0&t=2705s [16:35:03] awesome John thanks! [16:35:13] always lap up anything from Bert, never less than entertaining [16:36:15] +1 [16:42:06] his Twitter account is a great follow for vaccine-related knowledge as well :) [17:58:33] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Use next-hop-self for iBGP sessions - https://phabricator.wikimedia.org/T295672 (10cmooney) So we had some unexpected consequences over the weekend following this change. Example mail from ISP below: ` > Cc'ing Wikimedia NOC. > > We have... [18:35:14] 10netops, 10Infrastructure-Foundations, 10SRE, 10SRE-swift-storage, 10ops-codfw: Can't commit on asw-b-codfw - https://phabricator.wikimedia.org/T295118 (10ayounsi) 05Open→03Resolved Codfw repooled, everything is back to normal. [18:47:40] \o/