[00:03:46] (SystemdUnitFailed) resolved: dump_cloud_ip_ranges.service Failed on puppetmaster1001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [04:38:46] (SystemdUnitFailed) firing: docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [08:38:46] (SystemdUnitFailed) firing: docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [10:06:49] jbond: What is the command to lookup hiera values manually on a node for troubleshooting? I've seen it in wikitech somewhere but can't remember/find it [10:09:37] jelto: form puppetserver1001.eqiad.wmnet run smething like [10:09:40] sudo puppet lookup --node puppetserver1001.eqiad.wmnet profile::puppet::agent::use_srv_records --compile [10:09:49] ah great thanks! [10:10:09] yes that worked :) [10:10:15] great :) [12:38:46] (SystemdUnitFailed) firing: docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [12:54:05] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Export routes generated from ARP/ND in EVPN - https://phabricator.wikimedia.org/T329369 (10cmooney) We need to apply this configuration before we start moving servers in codfw to the new switches, while connected to existing row-wide vlans.... [12:55:03] 10netops, 10Infrastructure-Foundations, 10SRE, 10Patch-For-Review: Export routes generated from ARP/ND in EVPN - https://phabricator.wikimedia.org/T329369 (10cmooney) >>! In T329369#8717915, @cmooney wrote: > If we do have stretched L2 segments across multiple LEAFs, we may wish to also export the /32 and... [15:33:46] (SystemdUnitFailed) firing: (2) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [16:39:55] 10Packaging, 10Infrastructure-Foundations: wmfbackups packages for Debian Bookworm - https://phabricator.wikimedia.org/T347740 (10fnegri) [16:40:14] 10Packaging, 10Infrastructure-Foundations: wmfbackups packages for Debian Bookworm - https://phabricator.wikimedia.org/T347740 (10fnegri) [16:41:40] 10Packaging, 10Infrastructure-Foundations, 10cloud-services-team (FY2023/2024-Q1): wmfbackups packages for Debian Bookworm - https://phabricator.wikimedia.org/T347740 (10fnegri) [16:41:57] 10Packaging, 10Infrastructure-Foundations, 10cloud-services-team (FY2023/2024-Q1): wmfbackups packages for Debian Bookworm - https://phabricator.wikimedia.org/T347740 (10fnegri) p:05Triage→03High [16:42:05] 10Packaging, 10Infrastructure-Foundations, 10cloud-services-team (FY2023/2024-Q1): wmfbackups packages for Debian Bookworm - https://phabricator.wikimedia.org/T347740 (10fnegri) 05Open→03In progress [19:33:46] (SystemdUnitFailed) firing: (2) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:03:46] (SystemdUnitFailed) firing: (3) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [21:33:46] (SystemdUnitFailed) firing: (3) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [22:03:48] (SystemdUnitFailed) firing: (2) docker-reporter-base-images.service Failed on build2001:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed