[00:05:25] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [02:05:25] FIRING: [2x] SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:05:40] FIRING: [2x] SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:30:45] 10SRE-tools, 06collaboration-services, 06Infrastructure-Foundations, 10Puppet-Core, and 4 others: Migrate roles to puppet7 - https://phabricator.wikimedia.org/T349619#11426937 (10MoritzMuehlenhoff) [08:05:25] FIRING: [2x] SystemdUnitFailed: update-ubuntu-mirror.service on mirror1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:05:40] FIRING: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:32:00] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: GDNSd discovery records: balance requests from POPs across core sites - https://phabricator.wikimedia.org/T411617 (10cmooney) 03NEW p:05Triage→03Medium [12:33:51] 10netops, 06Infrastructure-Foundations, 06SRE, 06Traffic: GDNSd discovery records: balance requests from POPs across core sites - https://phabricator.wikimedia.org/T411617#11428021 (10cmooney) [12:35:25] RESOLVED: SystemdUnitFailed: dump_cloud_ip_ranges.service on puppetserver2004:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:41:10] someone knows if that error is because CI is running a too old Python version ? https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1161337 [14:46:58] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/cookbooks/+/refs/heads/master/tox.ini says CI runs on 3.9-3.11 [14:47:53] right, thx [15:11:13] taavi: https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1214532/ if you have an opinion [15:21:17] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: eqiad: rows C/D Upgrade Tracking - https://phabricator.wikimedia.org/T404609#11428720 (10RobH) Day 12 Update (in progress, will edit as day progresses): * alert1002 migration complete * 306 of 308 hosts migrated. * lvs1019 will migrat... [17:15:45] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-ulsfo, 06SRE: ULSFO: New switch configuration - https://phabricator.wikimedia.org/T408892#11429389 (10ssingh) >>! In T408892#11426444, @Papaul wrote: > @ssingh yes we have to depool the site, yes 10 AM CT Thanks, that works. Will send an invite. [18:47:34] Heya, we're having issues with the reimage cookbook: https://paste.debian.net/plainh/020b551e [18:51:51] figured it out, thanks t :) [18:52:15] brett: what was it? it seemed familiar but could not put my finger on it [18:52:18] 10Mail, 06Infrastructure-Foundations, 06SRE: Emails to Google group no-reply@wikimedia.org are not being delivered - SMTP server issue? - https://phabricator.wikimedia.org/T411027#11429823 (10JKelsoteel-WMF) Hey @jhathaway - thanks for your input! I shared these points with Noah as well, and we were able to... [18:52:37] brett: related to "force puppet 7" in Hiera after all? [18:52:47] 10Mail, 06Infrastructure-Foundations, 06SRE: Emails to Google group no-reply@wikimedia.org are not being delivered - SMTP server issue? - https://phabricator.wikimedia.org/T411027#11429824 (10taavi) 05Open→03Declined [18:52:53] I think we have seen the same one not long ago [18:53:05] mutante: It's likely the fact that a recent patch failed puppet compilation. Will report back if that's not the fix [18:53:16] ah, ok, thx [19:00:39] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 3 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11429856 (10BCornwall) [19:07:00] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 3 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11429884 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by bret... [19:51:34] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11430081 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by brett@cu... [20:01:02] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11430098 (10cmooney) [20:06:26] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11430105 (10Jclark-ctr) [20:14:32] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11430132 (10cmooney) [20:43:28] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, and 2 others: lvs1020: move primary uplink from asw2-d7-eqiad to lsw1-d7-eqiad and remove link to asw2-c2-eqiad - https://phabricator.wikimedia.org/T405609#11430232 (10BCornwall)