[08:27:23] 06Traffic, 06Machine-Learning-Team (Q1 FY2026-27): CDN caching request for TTS v1 audio files - https://phabricator.wikimedia.org/T434046#12194241 (10isarantopoulos) [09:13:56] 06Traffic, 06collaboration-services, 10Continuous-Integration-Infrastructure, 10Gerrit, and 2 others: Get traces/logs from ATS on Gerrit CI errors - https://phabricator.wikimedia.org/T430901#12194523 (10ABran-WMF) p:05High→03Medium [10:51:13] 10netops, 06Infrastructure-Foundations: Publish, and maintain ASPA records for valid AS14907 upstreams - https://phabricator.wikimedia.org/T372161#12194861 (10ayounsi) I forgot how it works for IX peers, but overall I agree that it would be fine to publish our ASPA records now that ARIN's portal supports it. I... [11:00:53] 10netops, 06DC-Ops, 06Infrastructure-Foundations, 06SRE: Netbox report: alert on cables in 'planned' state for more than two months - https://phabricator.wikimedia.org/T432329#12194879 (10ayounsi) 05Open→03Resolved a:03ayounsi All done! [15:13:19] 06Traffic, 06DC-Ops, 10ops-eqsin, 06SRE: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#12195585 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by cdobbins@cumin1003 for host cp5022.eqsin.wmnet with OS trixie [16:06:14] 06Traffic, 06DC-Ops, 10ops-eqsin, 06SRE: cp5022 is unreachable - https://phabricator.wikimedia.org/T414411#12195852 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by cdobbins@cumin1003 for host cp5022.eqsin.wmnet with OS trixie executed with errors: - cp5022 (**FAIL**) - Removed from... [19:48:48] 06Traffic, 10Pywikibot, 06SRE, 10Wikidata, and 3 others: Pywikibot reports maxlag retry error - https://phabricator.wikimedia.org/T421642#12196506 (10Matthias) It seems today is a bad day again for pywikibot trying to make only a single edit. [20:11:48] FIRING: PuppetZeroResources: Puppet has failed generate resources on hcaptcha-proxy4003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:11:49] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on cp2045:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:11:49] FIRING: PuppetZeroResources: Puppet has failed generate resources on ncredir5003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:11:53] FIRING: PuppetZeroResources: Puppet has failed generate resources on acmechief2002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:12:33] FIRING: PuppetZeroResources: Puppet has failed generate resources on cp1103:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:13:08] FIRING: PuppetZeroResources: Puppet has failed generate resources on doh6001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:13:12] FIRING: PuppetZeroResources: Puppet has failed generate resources on lvs7003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:13:21] FIRING: PuppetZeroResources: Puppet has failed generate resources on durum2001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:14:17] FIRING: PuppetZeroResources: Puppet has failed generate resources on lvs2012:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:16:48] FIRING: [8x] PuppetZeroResources: Puppet has failed generate resources on cp1106:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:16:48] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on hcaptcha-proxy4003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:16:49] FIRING: [4x] PuppetZeroResources: Puppet has failed generate resources on ncredir3006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:17:43] FIRING: [13x] PuppetZeroResources: Puppet has failed generate resources on cp1103:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:18:21] FIRING: [4x] PuppetZeroResources: Puppet has failed generate resources on doh3006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:18:26] FIRING: [3x] PuppetZeroResources: Puppet has failed generate resources on lvs6002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:18:30] FIRING: [3x] PuppetZeroResources: Puppet has failed generate resources on durum2001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:19:13] FIRING: [3x] PuppetZeroResources: Puppet has failed generate resources on lvs1018:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:19:54] FIRING: PuppetZeroResources: Puppet has failed generate resources on dns1005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:21:48] FIRING: [4x] PuppetZeroResources: Puppet has failed generate resources on hcaptcha-proxy4003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:21:49] FIRING: [19x] PuppetZeroResources: Puppet has failed generate resources on cp1102:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:21:49] FIRING: [6x] PuppetZeroResources: Puppet has failed generate resources on ncredir3006:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:22:29] FIRING: [17x] PuppetZeroResources: Puppet has failed generate resources on cp1103:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:23:12] FIRING: [7x] PuppetZeroResources: Puppet has failed generate resources on doh1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:23:16] FIRING: [4x] PuppetZeroResources: Puppet has failed generate resources on lvs3009:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:23:25] FIRING: [4x] PuppetZeroResources: Puppet has failed generate resources on durum2001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:24:30] FIRING: PuppetZeroResources: Puppet has failed generate resources on ncmonitor1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:24:39] FIRING: [4x] PuppetZeroResources: Puppet has failed generate resources on dns1005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:26:48] FIRING: [5x] PuppetZeroResources: Puppet has failed generate resources on hcaptcha-proxy3001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:26:48] FIRING: [27x] PuppetZeroResources: Puppet has failed generate resources on cp1100:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:26:49] FIRING: [10x] PuppetZeroResources: Puppet has failed generate resources on ncredir1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:27:07] FIRING: [28x] PuppetZeroResources: Puppet has failed generate resources on cp1103:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:27:48] FIRING: [9x] PuppetZeroResources: Puppet has failed generate resources on doh1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:27:53] FIRING: [8x] PuppetZeroResources: Puppet has failed generate resources on lvs3009:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:27:57] FIRING: [6x] PuppetZeroResources: Puppet has failed generate resources on durum1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:29:20] FIRING: [7x] PuppetZeroResources: Puppet has failed generate resources on dns1005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:31:48] FIRING: [8x] PuppetZeroResources: Puppet has failed generate resources on hcaptcha-proxy1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:31:49] FIRING: [35x] PuppetZeroResources: Puppet has failed generate resources on cp1100:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:31:49] FIRING: [11x] PuppetZeroResources: Puppet has failed generate resources on ncredir1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:32:20] FIRING: [41x] PuppetZeroResources: Puppet has failed generate resources on cp1101:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:32:58] FIRING: [12x] PuppetZeroResources: Puppet has failed generate resources on doh1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:33:03] FIRING: [10x] PuppetZeroResources: Puppet has failed generate resources on lvs3009:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:33:16] FIRING: [10x] PuppetZeroResources: Puppet has failed generate resources on durum1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:34:08] FIRING: [6x] PuppetZeroResources: Puppet has failed generate resources on lvs1017:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:34:35] FIRING: [3x] PuppetZeroResources: Puppet has failed generate resources on lvs1013:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:34:44] FIRING: [10x] PuppetZeroResources: Puppet has failed generate resources on dns1005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:36:48] FIRING: [10x] PuppetZeroResources: Puppet has failed generate resources on hcaptcha-proxy1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:36:49] FIRING: [46x] PuppetZeroResources: Puppet has failed generate resources on cp1100:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:36:49] FIRING: [13x] PuppetZeroResources: Puppet has failed generate resources on ncredir1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:37:29] FIRING: [46x] PuppetZeroResources: Puppet has failed generate resources on cp1101:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:38:03] FIRING: [13x] PuppetZeroResources: Puppet has failed generate resources on doh1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:38:07] FIRING: [13x] PuppetZeroResources: Puppet has failed generate resources on lvs3009:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:38:12] FIRING: [11x] PuppetZeroResources: Puppet has failed generate resources on durum1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:38:54] FIRING: [7x] PuppetZeroResources: Puppet has failed generate resources on lvs1017:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:39:30] FIRING: [4x] PuppetZeroResources: Puppet has failed generate resources on lvs1013:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:39:44] FIRING: [13x] PuppetZeroResources: Puppet has failed generate resources on dns1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:41:48] FIRING: [14x] PuppetZeroResources: Puppet has failed generate resources on hcaptcha-proxy1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:41:49] FIRING: [14x] PuppetZeroResources: Puppet has failed generate resources on ncredir1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:41:49] FIRING: [55x] PuppetZeroResources: Puppet has failed generate resources on cp1100:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:42:02] FIRING: [2x] PuppetZeroResources: Puppet has failed generate resources on acmechief1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:42:52] FIRING: [55x] PuppetZeroResources: Puppet has failed generate resources on cp1101:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:43:21] FIRING: [14x] PuppetZeroResources: Puppet has failed generate resources on doh1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:43:25] FIRING: [15x] PuppetZeroResources: Puppet has failed generate resources on lvs3008:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:43:39] FIRING: [14x] PuppetZeroResources: Puppet has failed generate resources on durum1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:44:26] FIRING: [8x] PuppetZeroResources: Puppet has failed generate resources on lvs1017:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:44:53] FIRING: [4x] PuppetZeroResources: Puppet has failed generate resources on lvs1013:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:45:11] FIRING: [16x] PuppetZeroResources: Puppet has failed generate resources on dns1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:46:48] FIRING: [14x] PuppetZeroResources: Puppet has failed generate resources on hcaptcha-proxy1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:46:49] FIRING: [14x] PuppetZeroResources: Puppet has failed generate resources on ncredir1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:46:49] FIRING: [55x] PuppetZeroResources: Puppet has failed generate resources on cp1100:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:47:25] FIRING: [55x] PuppetZeroResources: Puppet has failed generate resources on cp1101:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:47:48] FIRING: [14x] PuppetZeroResources: Puppet has failed generate resources on doh1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:47:53] FIRING: [15x] PuppetZeroResources: Puppet has failed generate resources on lvs3008:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:47:57] FIRING: [14x] PuppetZeroResources: Puppet has failed generate resources on durum1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:48:49] FIRING: [8x] PuppetZeroResources: Puppet has failed generate resources on lvs1017:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:49:11] FIRING: [16x] PuppetZeroResources: Puppet has failed generate resources on dns1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:51:48] FIRING: [14x] PuppetZeroResources: Puppet has failed generate resources on hcaptcha-proxy1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:51:49] FIRING: [14x] PuppetZeroResources: Puppet has failed generate resources on ncredir1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:51:49] FIRING: [55x] PuppetZeroResources: Puppet has failed generate resources on cp1100:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:52:02] RESOLVED: [2x] PuppetZeroResources: Puppet has failed generate resources on acmechief1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:52:52] FIRING: [55x] PuppetZeroResources: Puppet has failed generate resources on cp1101:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:53:17] FIRING: [14x] PuppetZeroResources: Puppet has failed generate resources on doh1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:53:21] FIRING: [15x] PuppetZeroResources: Puppet has failed generate resources on lvs3008:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:53:30] FIRING: [14x] PuppetZeroResources: Puppet has failed generate resources on durum1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:54:08] FIRING: [8x] PuppetZeroResources: Puppet has failed generate resources on lvs1017:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:54:36] FIRING: [4x] PuppetZeroResources: Puppet has failed generate resources on lvs1013:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:54:49] FIRING: [15x] PuppetZeroResources: Puppet has failed generate resources on dns1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:56:48] FIRING: [14x] PuppetZeroResources: Puppet has failed generate resources on hcaptcha-proxy1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:56:49] FIRING: [13x] PuppetZeroResources: Puppet has failed generate resources on ncredir1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:56:49] FIRING: [54x] PuppetZeroResources: Puppet has failed generate resources on cp1102:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:57:38] FIRING: [55x] PuppetZeroResources: Puppet has failed generate resources on cp1101:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:57:58] FIRING: [13x] PuppetZeroResources: Puppet has failed generate resources on doh1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:58:03] FIRING: [15x] PuppetZeroResources: Puppet has failed generate resources on lvs3008:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:58:16] FIRING: [13x] PuppetZeroResources: Puppet has failed generate resources on durum1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [20:59:00] RESOLVED: [8x] PuppetZeroResources: Puppet has failed generate resources on lvs1017:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:01:48] FIRING: [11x] PuppetZeroResources: Puppet has failed generate resources on hcaptcha-proxy1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:01:49] FIRING: [11x] PuppetZeroResources: Puppet has failed generate resources on ncredir1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:01:49] FIRING: [42x] PuppetZeroResources: Puppet has failed generate resources on cp1102:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:02:34] FIRING: [39x] PuppetZeroResources: Puppet has failed generate resources on cp1103:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:02:54] FIRING: [10x] PuppetZeroResources: Puppet has failed generate resources on doh1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:03:03] FIRING: [9x] PuppetZeroResources: Puppet has failed generate resources on durum1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:04:11] RESOLVED: [2x] PuppetZeroResources: Puppet has failed generate resources on lvs1014:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:04:29] FIRING: [10x] PuppetZeroResources: Puppet has failed generate resources on dns1004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:06:48] RESOLVED: [9x] PuppetZeroResources: Puppet has failed generate resources on hcaptcha-proxy2002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:06:49] FIRING: [9x] PuppetZeroResources: Puppet has failed generate resources on ncredir1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:06:49] FIRING: [33x] PuppetZeroResources: Puppet has failed generate resources on cp1102:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:07:16] FIRING: [33x] PuppetZeroResources: Puppet has failed generate resources on cp1103:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:07:48] RESOLVED: [9x] PuppetZeroResources: Puppet has failed generate resources on doh1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:07:49] RESOLVED: [9x] PuppetZeroResources: Puppet has failed generate resources on lvs3008:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:07:57] FIRING: [8x] PuppetZeroResources: Puppet has failed generate resources on durum1002:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:08:49] RESOLVED: [7x] PuppetZeroResources: Puppet has failed generate resources on dns1005:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:26:49] FIRING: [5x] PuppetZeroResources: Puppet has failed generate resources on cp1106:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:26:53] FIRING: [4x] PuppetZeroResources: Puppet has failed generate resources on cp1105:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:27:49] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on durum4004:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:31:48] FIRING: [5x] PuppetZeroResources: Puppet has failed generate resources on cp1106:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:31:53] RESOLVED: [4x] PuppetZeroResources: Puppet has failed generate resources on cp1105:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:36:48] RESOLVED: [5x] PuppetZeroResources: Puppet has failed generate resources on cp1106:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [21:36:49] RESOLVED: PuppetZeroResources: Puppet has failed generate resources on ncredir7003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources