[00:34:26] <jinxer-wm>	 FIRING: [2x] SystemdUnitFailed: sync-puppet-ca.service on puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[00:44:26] <jinxer-wm>	 RESOLVED: [2x] SystemdUnitFailed: sync-puppet-ca.service on puppetserver1003:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[08:05:17] <topranks>	 I've to nip out for doctors appt.
[08:05:19] <topranks>	 back in about 90 mins 
[09:39:30] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10087179 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w...
[09:43:01] <wikibugs>	 10netbox, 06Infrastructure-Foundations: Evaluate usage of Kubernetes/Wikikube Tags in netbox and replace them with something if possible - https://phabricator.wikimedia.org/T354169#10087192 (10ayounsi) a:03akosiaris Upgrade is done, I had a bit more time to look into that.  Taking this list as baseline https...
[10:01:00] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10087262 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w...
[10:01:37] <wikibugs>	 10netbox, 06Infrastructure-Foundations, 13Patch-For-Review: netbox: drop profile::netbox::active_server parameter - https://phabricator.wikimedia.org/T309034#10087257 (10ayounsi) 05Open→03Stalled p:05Medium→03Low
[10:14:35] <wikibugs>	 10netbox, 06Infrastructure-Foundations, 06SRE, 06Traffic-Icebox: Make Netbox Active/Active - https://phabricator.wikimedia.org/T234997#10087280 (10ayounsi) 05Stalled→03Declined An active/active Netbox is not really doable for now. For both Redis and Postgres the extra cross-DC latency makes it prac...
[10:17:40] <wikibugs>	 10netbox, 06Infrastructure-Foundations: Improve Netbox active/passive failover process - https://phabricator.wikimedia.org/T330883#10087327 (10ayounsi) We should focus our efforts on improving the active/passive failover process.
[10:17:40] <wikibugs>	 10netbox, 06Infrastructure-Foundations: Improve Netbox active/passive failover process - https://phabricator.wikimedia.org/T330883#10087328 (10ayounsi)
[10:25:24] <wikibugs>	 10netbox, 06Infrastructure-Foundations, 13Patch-For-Review: netbox: drop profile::netbox::active_server parameter - https://phabricator.wikimedia.org/T309034#10087334 (10ayounsi) 05Stalled→03Declined The active server parameter now control rq-netbox as well, so it's unlikely we get rid of it (see {T3...
[10:31:08] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10087361 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik...
[10:36:30] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10087370 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w...
[10:42:03] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10087374 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w...
[10:45:51] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10087380 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik...
[11:21:59] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10087458 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik...
[11:23:33] <wikibugs>	 10netbox, 06Infrastructure-Foundations, 06Traffic: gdnsd failures when converting services from active/passive to active/active - https://phabricator.wikimedia.org/T330084#10087460 (10ayounsi) 05Stalled→03Declined Not going active/active (see {T234997})
[11:28:56] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10087482 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik...
[11:29:19] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10087484 (10Clement_Goubert)
[11:31:20] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10087486 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w...
[12:13:46] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10087551 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik...
[13:30:51] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10087782 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w...
[13:51:28] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10087860 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cgoubert@cumin1002 for host w...
[14:16:40] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10087989 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik...
[14:37:34] <wikibugs>	 10netops, 06Infrastructure-Foundations, 06serviceops, 06SRE, 13Patch-For-Review: Re-IP wikikube servers in codfw row A/B moving to per-rack subnets - https://phabricator.wikimedia.org/T372878#10088060 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cgoubert@cumin1002 for host wikik...