[04:53:42] 10netops, 06DC-Ops, 06Infrastructure-Foundations: Take advantage of 10Gb NICs in the new network stack - https://phabricator.wikimedia.org/T360297#10275555 (10Papaul) I think we have another issue if we go with "move all 4 ports at once" if servers connected for example to port 0,1 and 3 are moving to 10G a... [08:34:25] FIRING: SystemdUnitFailed: prometheus-ganeti-exporter.service on ganeti2044:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:44:25] RESOLVED: SystemdUnitFailed: prometheus-ganeti-exporter.service on ganeti2044:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:38:55] 10CAS-SSO, 06Infrastructure-Foundations, 13Patch-For-Review: Enable Redis backed Ticket Registry for CAS / IDP - https://phabricator.wikimedia.org/T377728#10276150 (10SLyngshede-WMF) [10:38:57] 10CAS-SSO, 06Infrastructure-Foundations: Create Redis database for IDP-Test - https://phabricator.wikimedia.org/T377937#10276152 (10SLyngshede-WMF) 05Open→03Resolved [11:12:52] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 10Spicerack: Upload redfish licenses to supermicro hosts - https://phabricator.wikimedia.org/T376121#10276267 (10elukey) [11:30:41] 10netops, 06DC-Ops, 06Infrastructure-Foundations: Take advantage of 10Gb NICs in the new network stack - https://phabricator.wikimedia.org/T360297#10276313 (10ayounsi) @cmooney @Papaul What do you think of: # Keeping the new script presented previously for the "easy" usecases # Introducing an optional "Serve... [11:34:48] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 10Spicerack: Upload redfish licenses to supermicro hosts - https://phabricator.wikimedia.org/T376121#10276326 (10elukey) [11:53:10] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 10Spicerack: Upload redfish licenses to supermicro hosts - https://phabricator.wikimedia.org/T376121#10276344 (10elukey) [12:00:32] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 10Spicerack: Upload redfish licenses to supermicro hosts - https://phabricator.wikimedia.org/T376121#10276380 (10elukey) 05Open→03Resolved a:03elukey Finally all the hosts without the license, that were manually configured, should be ok. The on... [12:28:28] 10netbox, 06Infrastructure-Foundations, 13Patch-For-Review: Netbox: basic change rollback - https://phabricator.wikimedia.org/T310589#10276466 (10ayounsi) 05Open→03Resolved a:03ayounsi Script deployed, I don't think it will be extremely useful, but let's see how it goes. [12:38:31] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Testing liberica with ncredir@eqiad - https://phabricator.wikimedia.org/T378453#10276553 (10ayounsi) Actually, a 2nd look at https://wikitech.wikimedia.org/wiki/IP_and_AS_allocations shows that `14907:11` is a bit better. But it doesn't matter much... [13:03:29] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Testing liberica with ncredir@eqiad - https://phabricator.wikimedia.org/T378453#10276682 (10Vgutierrez) nice, do we need to add lvs1013 to any ACLs? [13:32:50] 10netops, 06Infrastructure-Foundations, 13Patch-For-Review: Testing liberica with ncredir@eqiad - https://phabricator.wikimedia.org/T378453#10276789 (10ayounsi) It doesn't, but once it's ready to receive traffic we need to : 1/ review then deploy to all the eqiad switches/routers https://gerrit.wikimedia.org... [13:35:05] 10netops, 06DC-Ops, 06Infrastructure-Foundations: Take advantage of 10Gb NICs in the new network stack - https://phabricator.wikimedia.org/T360297#10276817 (10Papaul) @ayounsi i think that is a good idea. Also when @Volans did this task back in March the count of servers was at 54 since we have decommissione... [13:51:48] FIRING: PuppetDisabled: Puppet disabled on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [15:22:25] FIRING: SystemdUnitFailed: sync-puppet-ca.service on puppetserver2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:41:38] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 10Spicerack: Upload redfish licenses to supermicro hosts - https://phabricator.wikimedia.org/T376121#10277551 (10elukey) 05Resolved→03Open [15:48:58] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 10Spicerack: Upload redfish licenses to supermicro hosts - https://phabricator.wikimedia.org/T376121#10277606 (10elukey) [15:49:54] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 10Spicerack: Upload redfish licenses to supermicro hosts - https://phabricator.wikimedia.org/T376121#10277617 (10elukey) ETOOSOON :) It seems that ganeti1044+ hosts were already provisioned, and I didn't notice an error when uploading the license to 104... [16:02:25] FIRING: [2x] SystemdUnitFailed: sync-puppet-ca.service on puppetserver2002:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:32:25] FIRING: [3x] SystemdUnitFailed: sync-puppet-ca.service on puppetserver2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:38:47] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 10Spicerack: Upload redfish licenses to supermicro hosts - https://phabricator.wikimedia.org/T376121#10278006 (10elukey) [16:42:47] 10SRE-tools, 06DC-Ops, 06Infrastructure-Foundations, 10Spicerack: Upload redfish licenses to supermicro hosts - https://phabricator.wikimedia.org/T376121#10278021 (10elukey) Run provision on ganeti1045+, and fixed the ADMIN password as well. Last step is to figure out why ganeti1044's license doesn't work... [17:51:48] FIRING: PuppetDisabled: Puppet disabled on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [19:37:25] RESOLVED: [3x] SystemdUnitFailed: sync-puppet-ca.service on puppetserver2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:51:49] FIRING: PuppetDisabled: Puppet disabled on apt-staging2001:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=misc&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled