[01:59:25] (SystemdUnitFailed) firing: (2) debian-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [05:08:26] (SystemdUnitFailed) firing: (2) debian-weekly-rebuild.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:08:26] (SystemdUnitFailed) firing: geoip_update_ipinfo.service Failed on puppetmaster1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [09:58:13] jbond: When you have the time, we found the reason why OIDC didn't work for Gitlab: https://gerrit.wikimedia.org/r/c/operations/puppet/+/941391 [09:59:28] slyngs: still just catching up with emails but will ytake a look in a second. great work btw [10:00:59] Thanks, I'm of to lunch anyway so no rush [10:01:08] ack [10:56:42] 10netops, 10Infrastructure-Foundations, 10SRE: Add per-output queue graphing for Juniper network devices in LibreNMS - https://phabricator.wikimedia.org/T326322 (10ayounsi) I tested gNMI on lsw1-a1-codfw as it's not in production yet. After upgrading it to Junos 22.2. I configured gnmic with a basic: `lang=... [12:14:50] 10netops, 10Infrastructure-Foundations, 10SRE: Add per-output queue graphing for Juniper network devices in LibreNMS - https://phabricator.wikimedia.org/T326322 (10cmooney) Nice work! Bit of work to get it productionized for sure but great to see it working! [13:08:26] (SystemdUnitFailed) firing: geoip_update_ipinfo.service Failed on puppetmaster1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [13:43:33] this alert is caused by an exopired subsciprtion see: T342878 [13:43:33] T342878: GeoIP2-Anonymous-IP Subscription expired - https://phabricator.wikimedia.org/T342878 [15:12:12] 10netops, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 2 others: Investigate improvements to how puppet manages network interfaces - https://phabricator.wikimedia.org/T234207 (10aborrero) [15:16:34] 10netops, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 2 others: Puppet interface:: resources aren't cleaned up - https://phabricator.wikimedia.org/T342899 (10aborrero) [15:23:24] 10CAS-SSO, 10Infrastructure-Foundations, 10SRE, 10collaboration-services, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10Jelto) GitLab test instance has OIDC enabled as well now, thanks @jbond for that! We expect a GitLab security update somewhere ar... [15:59:25] (SystemdUnitFailed) resolved: geoip_update_ipinfo.service Failed on puppetmaster1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:03:28] 10netops, 10Infrastructure-Foundations, 10Puppet-Core, 10SRE, and 2 others: Puppet interface:: resources aren't cleaned up - https://phabricator.wikimedia.org/T342899 (10aborrero) p:05Triage→03Low [16:04:45] taavi: can you take anotgher look at https://gerrit.wikimedia.org/r/c/operations/puppet/+/940365 [16:05:30] jbond: is the '>> file' part implemented by /usr/bin/printf too? [16:06:03] taavi: thats part of POSIX [16:06:45] the default provider on puppet should support posuix but it wont have any built ins or shell expantions [16:07:02] see https://www.puppet.com/docs/puppet/5.5/types/exec.html#exec-providers [16:07:17] ah, TIL, thanks. that explains [16:07:29] no problem [18:00:11] 10SRE-tools, 10Infrastructure-Foundations, 10Spicerack, 10Patch-For-Review: Spicerack: add distributed locking support - https://phabricator.wikimedia.org/T341973 (10bking) [18:12:58] 10SRE-tools, 10Infrastructure-Foundations, 10Spicerack, 10Patch-For-Review: Spicerack: add distributed locking support - https://phabricator.wikimedia.org/T341973 (10bking) Per email discussion with @jbond , adding another locking backend for consideration: Swift - PRO: Well-known/supported at WMF - PRO... [18:16:04] 10SRE-tools, 10Infrastructure-Foundations, 10Spicerack, 10Patch-For-Review: Spicerack: add distributed locking support - https://phabricator.wikimedia.org/T341973 (10bking) [18:16:48] 10SRE-tools, 10Infrastructure-Foundations, 10Spicerack, 10Patch-For-Review: Spicerack: add distributed locking support - https://phabricator.wikimedia.org/T341973 (10bking) [18:17:14] 10SRE-tools, 10Infrastructure-Foundations, 10Spicerack, 10Patch-For-Review: Spicerack: add distributed locking support - https://phabricator.wikimedia.org/T341973 (10bking)