[08:49:05] 10Mail, 10Infrastructure-Foundations, 10Patch-For-Review: Upgrade MXes to Bullseye - https://phabricator.wikimedia.org/T286911 (10ayounsi) > To prevent this test server from accidentally messing with our existing production mail infrastructure, I'd like to also filter port 25 for mx2002.wikimedia.org on the... [08:59:32] 10Puppet, 10Infrastructure-Foundations, 10MW-on-K8s, 10Kubernetes: Add a fact holding the type of a disk (spinning/ssd) - https://phabricator.wikimedia.org/T288509 (10JMeybohm) [11:02:03] 10Mail, 10Infrastructure-Foundations: Upgrade MXes to Bullseye - https://phabricator.wikimedia.org/T286911 (10MoritzMuehlenhoff) >>! In T286911#7271558, @ayounsi wrote: >> To prevent this test server from accidentally messing with our existing production mail infrastructure, I'd like to also filter port 25 for... [12:59:25] 10netbox, 10Infrastructure-Foundations: Debug Netbox Disconnection issues - https://phabricator.wikimedia.org/T253358 (10ayounsi) 05Open→03Resolved a:03ayounsi Feel free to reopen if you're experiencing this issue again. [13:03:45] hey folks - I saw some emails flying by about the eqiad-esams wave [13:04:02] some outages and stuff, as well as something about the wave going back and forth through georgia? [13:04:22] can I help? [13:11:02] paravoid: I summarized everything in https://phabricator.wikimedia.org/T288503 [13:11:49] not sure if you're still in discussions with Lumen, and if the new path doesn't have that issue [13:12:53] 10netbox, 10Infrastructure-Foundations: Import row information into Netbox for Ganeti instances - https://phabricator.wikimedia.org/T262446 (10ayounsi) @MoritzMuehlenhoff @Volans : Instead of adding a custom field and machinery to keep it up to date, what do you think of reorganizing the existing data: At leas... [13:14:24] thanks [13:15:53] 10netbox, 10Infrastructure-Foundations: Import row information into Netbox for Ganeti instances - https://phabricator.wikimedia.org/T262446 (10MoritzMuehlenhoff) >>! In T262446#7272106, @ayounsi wrote: > @MoritzMuehlenhoff @Volans : > Instead of adding a custom field and machinery to keep it up to date, what d... [13:19:49] XioNoX: what KML is this task referring to? [13:20:13] is that of the new path? [13:22:02] paravoid: I'm glad you asked, I have a Google Drive with almost all the KMLs! :) I got the KML in march, so it's after the latency increase, but before that new order (not in service yet) [13:22:36] :D [13:22:37] ok [13:22:42] where was that communication with the NOC? [13:22:48] what was it in context of? an outage? [13:24:19] yep, on a NOC ticket after an outage, https://phabricator.wikimedia.org/T287469#7262689 [13:28:36] paravoid: note that the same circuit is down again since early morning [13:40:50] ack [13:40:57] emailed the vendor to complain about wasting our time like that [13:43:46] thanks :) [13:44:23] (I saw the other task flying by above - when we converted racktables to netbox, rows were effectively documented as rack groups in netbox) [13:44:31] 10netbox, 10Infrastructure-Foundations: Homer daily diff stacktrace - https://phabricator.wikimedia.org/T265032 (10ayounsi) 05Open→03Invalid Haven't seen it in a while. Will re-open if needed. [16:18:19] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): Join ARIN waiting list to request additional IPv4 resources. - https://phabricator.wikimedia.org/T288342 (10nskaggs) @aborrero can you ensure our future needs are expressed here? [16:20:13] 10netops, 10DC-Ops, 10SRE, 10ops-codfw, 10Wikimedia-Incident: asw-a2-codfw unresponsive - https://phabricator.wikimedia.org/T286787 (10Papaul) 05Open→03Resolved Received the replacement switch. Rack in C1 U43. setup the mgmt password same as the server mgmt password. Update Netbox with new serial num... [16:24:44] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (Kanban): Join ARIN waiting list to request additional IPv4 resources. - https://phabricator.wikimedia.org/T288342 (10cmooney) @nskaggs @aborrero might be better to add that to the parent task thanks. [17:02:21] 10netops, 10DC-Ops, 10Infrastructure-Foundations, 10SRE: (Need By: TBD) rack/setup/install atlas-codfw.wikimedia.org - https://phabricator.wikimedia.org/T273114 (10RobH) Can the setup of this device please be noted on https://wikitech.wikimedia.org/wiki/SRE/Dc-operations/Platform-specific_documentation/Atl... [17:05:03] 10SRE-tools, 10Infrastructure-Foundations, 10Spicerack, 10cloud-services-team (Kanban): wmcs.spicerack: Setup a host to run cookbooks from prod network - https://phabricator.wikimedia.org/T276440 (10dcaro) p:05Medium→03Triage a:05dcaro→03None [18:08:05] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10Spicerack: Spicerack downtime methods fail when the admin reason includes an apostrophe - https://phabricator.wikimedia.org/T288558 (10RLazarus) [18:08:15] 10SRE-tools, 10Infrastructure-Foundations, 10SRE, 10Spicerack: Spicerack downtime methods fail when the admin reason includes an apostrophe - https://phabricator.wikimedia.org/T288558 (10RLazarus) p:05Triage→03High [19:42:34] hi! as per T257324 and T284246#7180201, we have used bast5001's IP for doh5002. given that we just have on instance of Wikidough in ulsfo, no free public IPs, and I plan to deploy doh4002, would it be fine to decom bast4002 and use its IP for this purpose? [19:42:34] T257324: Consolidate edge bastion server into ganeti - https://phabricator.wikimedia.org/T257324 [19:42:35] T284246: Please create two Ganeti VMs for Wikidough in eqsin - https://phabricator.wikimedia.org/T284246 [19:43:50] probably a discussion for tomorrow morning, now that I think about the timezones but yeah :) [19:45:11] essentially we have two hosts of Wikidough per PoP and ulsfo is the only one with just one host so I wanted to resolve that. it's not urgent but yeah [20:28:05] sukhe: yeah, please go ahead and decom4002, if the hw box gets repurposed as another Ganeti host or something else, it can get readdde/installed under a new internal IP/name [20:29:46] moritzm: thanks for confirming! :) [20:29:51] last statement from Brandon was https://phabricator.wikimedia.org/T257324#6751667 [20:30:15] and given that you're in the process of setting up new doh hosts, that seems to fall into these plans in the wider scheme of things [20:30:42] thanks, yeah, I just wanted to make sure I understood it correctly that it was OK to decom it and then reuse the IP [20:31:32] and also: bast4002 was bought in May 2017, it's nearing an age where it seems increasingly unlikely that we'll find a productive use for it's remaining lifespan anywy [20:33:23] ah I see [20:36:00] when you decom bast4002 using the standard Phab template, there's a step anyway where DC ops make a call whether to reclaim the server as spare or decom/unrack right away [20:36:25] so we can simply rely on their judgment there [20:38:45] thanks! I will work on the decom tomorrow and leave that to dcops [21:45:35] 10netops, 10Infrastructure-Foundations, 10SRE, 10decommission-hardware, 10ops-eqiad: Decommission asw-c-eqiad - https://phabricator.wikimedia.org/T208734 (10Jclark-ctr) [21:46:10] 10netops, 10Infrastructure-Foundations, 10SRE, 10decommission-hardware, 10ops-eqiad: Decommission asw-c-eqiad - https://phabricator.wikimedia.org/T208734 (10Jclark-ctr) Preformed factory reset removed from rack, updated netbox [21:46:26] 10netops, 10Infrastructure-Foundations, 10SRE, 10decommission-hardware, 10ops-eqiad: Decommission asw-c-eqiad - https://phabricator.wikimedia.org/T208734 (10Jclark-ctr) 05Open→03Resolved [22:04:49] 10SRE-tools, 10Infrastructure-Foundations, 10Spicerack, 10cloud-services-team (Kanban): wmcs.spicerack: Setup a host to run cookbooks from prod network - https://phabricator.wikimedia.org/T276440 (10nskaggs) p:05Triage→03Medium