[08:08:36] test log [08:19:14] jobo, XioNoX, topranks: bblack had reached out with various addressing needs (XioNoX and I had discussed in the past as well) -- for everyone's visibility, I asked him to file a task, and https://phabricator.wikimedia.org/T283612 is the result of that [08:19:20] it likely needs your input [08:20:16] relatedly, I've added a couple of lines to the budget for $$$ to spend on additional IP space (for something like an additional RIPE LIR like we did last year) [08:20:38] that's placed in next fiscal, so around August or so [08:21:26] it seemed to me through a superficial search that there are likely other options like ARIN's waiting list ( https://www.arin.net/resources/guide/ipv4/waiting_list/ ), but I haven't researched it fully [08:22:15] this is not super urgent, but given some of these options have either a waiting list period or require a level of bureaucracy with contracts etc., it'd be good to come up with a plan sooner rather than later [08:23:16] by a plan I mean for "we need X addresses, we have Y, we need to acquire X-Y new ones -- here are the options for doing so" [08:23:49] Ok. Certainly no harm in putting ourselves on the ARIN waiting list, it opens every now and then we’d be fairly certain to get a /24 at some stage. [08:24:25] are we meeting the requirements? that'd be also good to know [08:24:42] I remember from the distant past some requirements like % of used space etc. [08:25:57] next FY could eat up all our existing v4 space, so it would be great to secure some for the years after (eg. new POPs) [08:26:42] FYI this channel is now logged as before (see topic), I'm going through all the other settings [08:26:43] Depends also how they count used space, our anycast ranges are not a great use of that :) [08:27:08] can you all come up with an addressing plan for the next 18-24 months or so, and our options going forward? [08:28:21] we have some money to spend (and we can reliquinsh some of it if it's free, or find more if it's more expensive) + I know traffic and others in the past have had requests, so it'd be good to have a path forward [08:28:45] Thanks Faidon, we'll produce some plan. Any rough timeline when that would be needed by? [08:31:53] good question :) ultimately I think the timeline depends a lot on the needs of the various stakeholders like traffic (and possibly wmcs, XioNoX?) + the delays that we expect by these processes [08:32:16] for the budget I've allocated, I've put provisionally a line item for spending it on Aug 2021 [08:33:32] from what I can tell with what I know so far, I think having a plan in the next ~2 months would work [08:38:07] Thanks :) [08:40:41] I guess it makes sense to open a Phabricator task of our own about this? And link Brandon's one to it? [08:56:12] yeah that makes sense :) wanted to talk to you all first [08:59:29] paravoid: wrt this chan settings, mostly done [08:59:56] volans: thanks :) [09:02:10] thanks volans [09:02:28] np :) [09:03:11] I made the whole team with full permissions and 4 of us founders (that's the limit), I'm trying to set para.void as successor but failing, it did work the other day on another private channel [09:16:10] ack [09:41:17] volans: perhaps we should have wikibugs in here :) [09:41:32] as you like, I can do that sure [09:41:46] I'm not convinced on the closure on the freenode side in light of recent events [09:41:57] https://mastodon.sdf.org/@kline/106299403921451814 [09:42:18] what do you mean by not convinced [09:42:56] that if we "close" it as per our docs we se thte topic that we moved here and it might be taken over [09:43:09] also I'm not an op in the freenode one, so I can't do that ;) [09:43:42] noone is an op in the freenode one now other then freenodeop [09:44:00] nothing we can do about that now unfortunatly [09:44:44] yeah, just it RIP, with noone from SRE around there, every hijack attempt seems futile to begin with :-) [09:44:47] paravoid is still the founder, he might be able to [09:45:03] there are still few people, like grant :D [09:45:28] oh yes your right io thuoght they had done all of the wikimedia* space [09:45:45] no apaprently they went over all the chan where libera was mentioned [09:45:56] hence I'm not sure we want to "close" it as per our docs [09:46:01] or maybe do something different [09:46:14] yes i saw that i thought perhaps they had also targeted wikimedia, as i didn;t think -sre-private had libera in the topic [09:46:40] might have had it to tell people to migrate, I don't recall [09:46:49] yes not sure either [10:12:54] jbond: which tags though? [10:12:59] all our tags? for wikibugs [10:14:05] volans: yes i would say all foundations tags (although im unsure i know what they all are) [10:14:46] mos tof them are listed in the SRE tag description [10:14:51] I've updated that list recently [10:14:54] but might be missing some yet [10:16:04] ahh yes that looks good, although its missing cas-sso and pki-cfssl (how do we update that) [10:17:12] jbond: https://phabricator.wikimedia.org/project/edit/1025/ [10:18:04] thanks updated [10:19:36] also do we want just bugs here or also gerrit changes? [10:19:50] I still need to understand one thing that mentions an exemption on libera side [10:21:05] volans: maybe start with just bugs? [10:21:11] ack [10:30:13] jbond: sorry to keep bothering you, so we really want all the tasks updates here in this chan, not just the bot so that it gives use the link to T12345 [10:32:38] volans: well i should say its more of an `i` then a `we` :) but for me i would say yes all task updates, simlilar to service ops and traffic [10:33:06] I'm ok to try, we can always turn it down [10:33:14] sgtm [10:37:56] hello stashbot [10:37:59] T2001 [10:37:59] T2001: [DO NOT USE] Documentation is out of date, incomplete (tracking) [superseded by #Documentation] - https://phabricator.wikimedia.org/T2001 [10:38:02] might be useful too [10:59:24] is the mail tag on phabricator ours? [11:00:21] topranks, XioNoX: should we drop netops from traffic at this point as it will be also here? [11:00:39] volans: sure [11:01:08] ack [11:05:44] volans: i think we agreeded mail probably does belong in foundation however currently its a bit unowned with a default 301 -> herron [11:06:01] ack, added anyway [11:06:10] thx [11:59:58] hi all i have created a new patch for the contacts metadata puppet work https://gerrit.wikimedia.org/r/c/operations/puppet/+/695230/ feedback welcome [12:00:26] I'll have a look in bit [12:01:01] thanks (althugh you allready did ;)) [12:29:31] ack [13:34:38] moritzm, jbond: can I reimage one of the sretest hosts to test the reimage script? any preference? [13:36:23] sretest1001 runs the cuminunpriv setup, better use 1002 [13:36:39] ack [13:36:50] it's configured for bullseye, it might need a refreshed d-i image due to latest kernel changes [13:37:13] you can also use theemin [13:37:29] whatever is easier/less disruptive [13:38:13] maybe use 1002 and if there's an issue with the kernel udebs, I'll refresh the image and we simply re-try? [13:40:04] ack [14:12:00] moritzm: d-i gets stuck at the main menu, with download installer components selected [14:16:00] with no kernel modules were found as you were suggesting might happen [14:18:36] ok, let me import the latest installer images, then. there must have been another kernel ABI bump, then [14:18:54] ack, thanks, sorry for the trouble [14:19:26] np at all: :-) [14:31:56] volans: updated the image and ran puppet on install*/apt [14:35:28] thanks, let me retry [14:39:29] lol jbond you changed everything while I was reviewing :D [14:40:29] should I wait upcoming patches? [14:41:05] volans: sorry i think it should be good nwo [14:41:16] ack [15:53:30] 10SRE-tools, 10Wikibugs: wikibugs test bug part II - https://phabricator.wikimedia.org/T90594 (10Legoktm) test [15:53:52] ^^ volans, jbond [15:54:19] legoktm: thanks a lot! great [15:54:48] 10Puppet, 10GitLab (Initialization), 10Patch-For-Review, 10Release-Engineering-Team (Radar), 10User-brennen: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10jbond) @brennen sounds find to me, its possible when the play book is run that there may still be some puppetised b... [15:54:53] yay thanks volans legoktm [15:56:01] 10Puppet, 10GitLab (Initialization), 10Patch-For-Review, 10Release-Engineering-Team (Radar), 10User-brennen: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10brennen) > @brennen sounds fine to me, its possible when the play book is run that there may still be some puppetis... [16:10:03] legoktm: FYI there is one left reference to free node at the bottom of channels.yaml ;) [16:10:24] gahhh [16:10:31] thanks [16:55:11] 10netops, 10Wikibugs: wikibugs test bug part II - https://phabricator.wikimedia.org/T90594 (10Volans) [16:59:11] 10netops, 10SRE, 10Traffic: Please configure the routers for Wikidough's anycasted IP - https://phabricator.wikimedia.org/T283503 (10cmooney) Merged and pushed with homer to cr1-codfw and cr2-codfw, working ok with the first VM (Bird being enabled on others shortly): ` cmooney@re0.cr2-codfw> show route rece... [18:34:57] paravoid, topranks, jobo: https://phabricator.wikimedia.org/T283750 [18:38:33] XioNox: thanks [19:03:11] XioNoX: <3 [19:03:39] can I also flag https://phabricator.wikimedia.org/T267714 to this group's attention? [19:04:44] it wasn't (and still isn't) super high prio, but it's been almost 3 months now :) [19:07:51] ah, I've been meaning to get back to that. last I remember all the docs about anchor configuration had disappeared from RIPE's site [19:09:20] maybe it's because of all the account weirdness? I just found https://atlas.ripe.net/docs/anchor-installation/ through my account [19:09:52] i think that might have re-appeared in the meantime [19:09:56] nod [19:39:46] 10netops, 10SRE: Netbox has incorrect email address for GTT - https://phabricator.wikimedia.org/T246564 (10wiki_willy) Sure, no prob. I just sent out an email, for our rep to loop in the Customer Success Manager. You're copied on it, so feel free to chime in on the reply. Thanks, Willy >>! In T246564#711608... [22:35:15] 10netops, 10DC-Ops: allow mgmt network to access tftp servers for firmware updates - https://phabricator.wikimedia.org/T283771 (10RobH) p:05Triage→03Medium [22:36:35] 10netops, 10DC-Ops: allow mgmt network to access tftp servers for firmware updates - https://phabricator.wikimedia.org/T283771 (10RobH) Oh, I added Arzhel so he is aware of any network asks and can potentially point out any hard blockers that are proposed. I plan to bring this up in our next DC ops meeting, j... [22:46:39] 10netops, 10DC-Ops, 10SRE: allow mgmt network to access tftp servers for firmware updates - https://phabricator.wikimedia.org/T283771 (10Dzahn) As one who worked on the installserver puppet roles in the past I'd say the cons of B aren't so bad. We should be able to reuse existing profiles and just combine th... [22:49:03] 10netops, 10DC-Ops, 10SRE: allow mgmt network to access tftp servers for firmware updates - https://phabricator.wikimedia.org/T283771 (10RobH) >>! In T283771#7117937, @Dzahn wrote: > As one who worked on the installserver puppet roles in the past I'd say the cons of B aren't so bad. We should be able to reus... [22:57:31] 10netops, 10DC-Ops, 10SRE: allow mgmt network to access tftp servers for firmware updates - https://phabricator.wikimedia.org/T283771 (10Dzahn) But wouldn't the dedicated ganeti server need hardware then instead? [23:02:22] 10netops, 10DC-Ops, 10SRE: allow mgmt network to access tftp servers for firmware updates - https://phabricator.wikimedia.org/T283771 (10RobH) >>! In T283771#7117961, @Dzahn wrote: > But wouldn't the dedicated ganeti server need hardware then instead? My initial task description is stating ganeti instance no... [23:04:16] 10netops, 10DC-Ops, 10SRE: allow mgmt network to access tftp servers for firmware updates - https://phabricator.wikimedia.org/T283771 (10RobH) [23:05:04] 10netops, 10DC-Ops, 10SRE: allow mgmt network to access tftp servers for firmware updates - https://phabricator.wikimedia.org/T283771 (10Dzahn) ACK, sorry, I read " create a ganeti server" as a server running ganeti that then hosts VMs on it. Yea, so then the biggest part of this would be the "make the mgmt...