[03:14:18] (SystemdUnitFailed) firing: wmf_auto_restart_apache-htcacheclean.service Failed on idm1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [06:16:13] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team: Move WMCS servers to 1 single NIC - https://phabricator.wikimedia.org/T319184 (10ayounsi) [07:14:18] (SystemdUnitFailed) firing: wmf_auto_restart_apache-htcacheclean.service Failed on idm1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:22:51] (SystemdUnitFailed) resolved: wmf_auto_restart_apache-htcacheclean.service Failed on idm1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [07:48:17] 10Puppet, 10Infrastructure-Foundations: Investigate why apache-htcacheclean is started - https://phabricator.wikimedia.org/T334577 (10MoritzMuehlenhoff) 05Open→03Resolved a:03MoritzMuehlenhoff This is resolved. [09:44:34] I seem to remember that Sphinx had some fancy pants new way of ensuring that types where documented without you needing to write the type in the docstring [09:47:30] Ah, found it https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/896323 [10:36:34] 10netops, 10Infrastructure-Foundations, 10SRE: Verify and Configure ECMP operation for EVPN switches - https://phabricator.wikimedia.org/T334658 (10cmooney) p:05Triage→03Medium [11:03:05] slyngs: sorry was in a meeting, lmk if you need any followup [11:07:50] 10netops, 10Infrastructure-Foundations, 10SRE, 10cloud-services-team (FY2022/2023-Q4): Configure cloudsw1-b1-codfw and migrate cloud hosts in codfw B1 to it - https://phabricator.wikimedia.org/T327919 (10fnegri) [11:13:48] volans: Thanks, but stealing configuration from spicerack works perfectly :-) [11:23:00] I want the royalties :-{ [11:24:30] for every bitu you sell :D [11:24:44] it's clearly a cryptocurrency from the name :-P [12:02:45] DogeCoin derivative :-) [13:13:00] 10netbox, 10DC-Ops, 10Infrastructure-Foundations, 10Observability-Alerting, and 2 others: validate what we need from the check_eth check - https://phabricator.wikimedia.org/T333007 (10jbond) > Going further there are already a bunch of network related metric exported by node_exporter including duplex, flag... [14:04:21] (SystemdUnitFailed) firing: (2) nic-saturation-exporter.service Failed on ganeti2019:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:08:32] (SystemdUnitFailed) resolved: (2) nic-saturation-exporter.service Failed on ganeti2019:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [14:35:39] 10netops, 10Infrastructure-Foundations, 10SRE: eqiad/codfw virtual-chassis upgrades - https://phabricator.wikimedia.org/T327248 (10ayounsi) [15:34:31] 10puppet-compiler, 10Infrastructure-Foundations: PCC failing for an LVS host (false negative) even after manually updating facts - https://phabricator.wikimedia.org/T334680 (10ssingh) [15:34:49] 10puppet-compiler, 10Infrastructure-Foundations, 10SRE: PCC failing for an LVS host (false negative) even after manually updating facts - https://phabricator.wikimedia.org/T334680 (10ssingh) [16:18:54] 10netops, 10Infrastructure-Foundations, 10SRE, 10ops-codfw: Codfw:row A/B: rack/cable new switches - https://phabricator.wikimedia.org/T332180 (10Papaul) [16:48:01] 10Puppet, 10Cloud-Services, 10Infrastructure-Foundations: WMCS puppet-enc not working with puppetserver 7 - https://phabricator.wikimedia.org/T334686 (10jbond) [16:49:40] 10Puppet, 10Cloud-VPS, 10Infrastructure-Foundations: WMCS puppet-enc not working with puppetserver 7 - https://phabricator.wikimedia.org/T334686 (10taavi) [16:52:02] 10Puppet, 10Cloud-VPS, 10Infrastructure-Foundations: WMCS puppet-enc not working with puppetserver 7 - https://phabricator.wikimedia.org/T334686 (10taavi) You seem to be missing the configuration to use puppet-enc, particularly `/etc/puppet/puppet.conf` needs `external_nodes = /usr/local/bin/puppet-enc`. [18:20:21] 10Puppet, 10Cloud-VPS, 10Infrastructure-Foundations: WMCS puppet-enc not working with puppetserver 7 - https://phabricator.wikimedia.org/T334686 (10jbond) 05Open→03Resolved a:03jbond >>! In T334686#8779836, @taavi wrote: > You seem to be missing the configuration to use puppet-enc, particularly `/etc/p... [18:20:27] 10Puppet, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Next steps for Puppet 7 - https://phabricator.wikimedia.org/T330490 (10jbond)