[06:59:04] slyngs: https://phabricator.wikimedia.org/T299560#10094807 looks like it does :) [06:59:37] jayme: haha yeah I should have specified that it was about 2 years ago :) [07:00:15] I wonder what the "to 8.4" comment was all about then [07:00:18] Seems fine [07:00:32] slyngs: do you have a link? [07:00:46] Oh I don't have a dashboard [07:01:00] I was just looking at the exporter output [07:01:08] Aaah, the version comment [07:02:08] https://github.com/prometheus/node_exporter under "disabled by default" [07:04:09] ganeti2033:~$ cat /proc/drbd [07:04:09] version: 8.4.11 [07:04:57] I haven't tested bullseye too, we have 69 ganeti hosts according to https://os-reports.wikimedia.org/bullseye.html [07:05:41] Okay, sp drbd has it's own versioning scheme separate from drbd-utils [07:07:37] Bullseye is 8.4.11 as well [07:07:50] yeah, https://github.com/LINBIT/drbd/tags?after=drbd-9.0.0pre4 [07:08:14] they have a v9 branch as well [07:08:22] Seems feature complete :-) [07:08:22] and they still maintain 8.x [07:09:33] eh, they're working on a v10 it seems [07:15:02] slyngs: now that we have the data, should we do a dashboard? :) [07:18:15] We properly should... What do we want to show? Disks out of sync seems useful [08:09:34] do we have a drbd expert in house? who wants to become one ? :) [09:05:31] slyngs: I did a little something https://grafana-rw.wikimedia.org/d/f_tZtVlMz/drbd [09:06:34] That looks nice [09:08:10] slyngs: should I sent a patch to deploy it to all the ganeti hosts? Probably testing it manually on a random bullseye host too [09:09:02] Might as well, it seems like one of those things that are nice to have, if drbd breaks [09:30:10] slyngs: https://grafana-rw.wikimedia.org/d/f_tZtVlMz/drbd?forceLogin&orgId=1&refresh=1m&from=now-3h&to=now&var-instance=ganeti1009:9100 seems good! [09:38:51] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095106 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [09:40:45] XioNoX: You're pretty quick with Grafana :-) [09:45:02] slyngs (and whoever feels like reviewing it) https://gerrit.wikimedia.org/r/c/operations/puppet/+/1067302 [09:45:09] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095118 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [09:45:16] Done [09:45:19] thx ! [09:52:27] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095141 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [10:14:29] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095259 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w... [10:24:58] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095281 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [10:29:56] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095291 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [10:41:30] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095368 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [10:46:10] FIRING: SystemdUnitFailed: generate_vrts_aliases.service on mx2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:01:21] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095415 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik... [11:43:40] RESOLVED: SystemdUnitFailed: generate_vrts_aliases.service on mx2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:55:59] jobo XioNoX topranks: fyi - just got the news I have passed RP2 :-) [11:56:09] \o/ [11:56:16] awesome, congrats!!! [11:56:21] All the hard work and supervision has been very successful [11:56:48] woot! [12:01:03] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095590 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by akosiaris@cumin1002 from mw2292 to... [12:01:51] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095592 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host... [12:04:12] And as a result, /me suddenly has to think about what to do in post-education life [12:05:00] haha - no need to rush anything man you've plenty of time :) [12:05:37] Ha true. It'll be sorted out eventually [12:12:43] Southparkfan: nice!!! [12:56:28] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10095765 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wiki... [13:24:55] Well done! [13:30:19] slyngs: how do I got to create a test-idp VM ? [13:30:38] 10SRE-tools, 06Infrastructure-Foundations: Allow debmonitor to store the Debian version-id in the OS field - https://phabricator.wikimedia.org/T368744#10095911 (10elukey) Today I cleaned up some db nodes reported as debmonitor client failures while I was on holiday: ` >>> spicerack.debmonitor().host_delete('d... [13:31:02] Oh, I though you would just move one [13:32:29] slyngs: it's an independent cluster, like eqiad is from codfw, so we can't move them over [13:57:44] 10netbox, 06DC-Ops, 06Infrastructure-Foundations, 06SRE, 13Patch-For-Review: sre.hardware.upgrade-firmware cookbook: product slug parsing - https://phabricator.wikimedia.org/T348036#10096042 (10ayounsi) 05Open→03Resolved Deployed! let me know if any issue. [14:15:05] 10homer, 06Infrastructure-Foundations: micro-CI for homer-private - https://phabricator.wikimedia.org/T259182#10096212 (10ayounsi) p:05Medium→03Low [15:06:15] topranks: regarding https://phabricator.wikimedia.org/T370630 am I correct in assuming the row C/D migration will be the same as A/B, with the old vlan configured on the ports and then we move the servers over to the new vlan and peering model? [15:06:56] claime: correct, it's just a simple cable swap and short interruption for each host [15:07:11] cool thank you <3 [15:07:18] we can then tackle the vlan moves / re-ip'ing at a pace that works [15:07:31] btw major kudos for the work you are doing on that for the other hosts!! [15:07:39] major props <3 [15:08:55] topranks: A fire was lit under my arse with the realization of the upcoming switchover, and not wanting to do this on the active datacentre if I can avoid it x) [15:09:51] of course we'll probably have to do it anyways for rows C/D, I don't think we want to wait 6 months to move to the new vlan, but I'd rather exercise the procedures and cookbooks on the secondary :D [15:10:37] well we don't have to rush into anything. waiting 6 months isn't ideal, but there is no point taking risks or making life hard for ourselves either [15:20:25] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10096550 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by kamila@cumin1002 from kubernetes20... [15:22:59] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10096557 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by kamila@cumin1002 for host wik... [15:43:23] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10096619 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by akosiaris@cumin1002 from mw2293 to... [15:44:01] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10096620 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by akosiaris@cumin1002 for host... [16:05:39] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10096726 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by kamila@cumin1002 for host wikikub... [17:09:15] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10097092 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by akosiaris@cumin1002 for host wiki... [17:12:49] 10netops, 06Infrastructure-Foundations: Apply egress Source Address Validation on the Wikimedia core routers - https://phabricator.wikimedia.org/T372158#10097098 (10Southparkfan) >>! In T372158#10056948, @ayounsi wrote: >> However, in reality, it should be possible to reject all IP packets where the source IP... [21:13:14] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10098130 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.rename started by swfrench@cumin2002 from kubernetes... [21:15:08] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10098132 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by swfrench@cumin2002 for host w... [22:02:06] 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10098199 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by swfrench@cumin2002 for host wikik... [22:57:13] 10netops, 06Infrastructure-Foundations: Publish, and maintain ASPA records for valid AS14907 upstreams - https://phabricator.wikimedia.org/T372161#10098371 (10Southparkfan) >>! In T372161#10056965, @ayounsi wrote: > [...] >> However, the ASPA record is yet another duplicate of the transit_provider list in Hom...