[00:02:48] (PuppetZeroResources) firing: (7) Puppet has failed generate resources on db1161:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [00:07:48] (PuppetZeroResources) firing: (5) Puppet has failed generate resources on db1161:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [00:11:49] (PuppetZeroResources) firing: Puppet has failed generate resources on db1249:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [00:12:48] (PuppetZeroResources) firing: (5) Puppet has failed generate resources on db1161:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [00:21:49] (PuppetZeroResources) firing: (2) Puppet has failed generate resources on db1249:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [00:32:49] (PuppetZeroResources) firing: (2) Puppet has failed generate resources on db1161:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [00:37:48] (PuppetZeroResources) firing: (2) Puppet has failed generate resources on db1161:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [00:42:48] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on db1161:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [00:51:49] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on db1200:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [00:56:49] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1200:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [00:57:48] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on es1023:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [01:06:50] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1200:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [01:07:49] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on es1023:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [01:22:49] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on es1023:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [01:26:49] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on db1229:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [01:36:49] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1221:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [01:42:49] (PuppetZeroResources) resolved: Puppet has failed generate resources on es1025:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:01:49] (PuppetZeroResources) firing: (2) Puppet has failed generate resources on db1212:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:11:49] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on db1211:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:16:49] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on db1211:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:35:18] (PuppetZeroResources) firing: Puppet has failed generate resources on db1150:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:45:18] (PuppetZeroResources) firing: (2) Puppet has failed generate resources on db1150:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:46:49] (PuppetZeroResources) firing: (2) Puppet has failed generate resources on db1211:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [02:51:49] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on db1211:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [03:00:19] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on db1150:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [03:01:49] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on db1211:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [03:10:18] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on db1172:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [03:11:49] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1222:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [03:20:18] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on db1172:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [03:25:33] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on db1172:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [03:31:49] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1239:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [03:46:49] (PuppetZeroResources) firing: (5) Puppet has failed generate resources on db1207:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [03:50:18] (PuppetZeroResources) firing: (2) Puppet has failed generate resources on db1153:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [03:51:49] (PuppetZeroResources) firing: (5) Puppet has failed generate resources on db1207:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [03:56:49] (PuppetZeroResources) firing: (5) Puppet has failed generate resources on db1207:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:01:49] (PuppetZeroResources) firing: (5) Puppet has failed generate resources on db1207:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:05:18] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on db1153:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:06:49] (PuppetZeroResources) firing: (7) Puppet has failed generate resources on db1207:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:10:18] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1153:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:11:49] (PuppetZeroResources) firing: (8) Puppet has failed generate resources on db1207:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:15:18] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1153:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:16:49] (PuppetZeroResources) firing: (7) Puppet has failed generate resources on db1216:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:20:19] (PuppetZeroResources) firing: (5) Puppet has failed generate resources on db1153:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:21:49] (PuppetZeroResources) firing: (7) Puppet has failed generate resources on db1216:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:22:03] (PuppetZeroResources) firing: (7) Puppet has failed generate resources on db1216:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:26:49] (PuppetZeroResources) firing: (7) Puppet has failed generate resources on db1216:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:30:18] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1153:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:36:49] (PuppetZeroResources) firing: (6) Puppet has failed generate resources on db1216:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:40:19] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1153:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:41:49] (PuppetZeroResources) firing: (5) Puppet has failed generate resources on db1216:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:50:18] (PuppetZeroResources) resolved: (2) Puppet has failed generate resources on db1156:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [04:51:49] (PuppetZeroResources) firing: (5) Puppet has failed generate resources on db1216:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [05:00:19] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on db1156:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [05:10:20] (PuppetZeroResources) resolved: Puppet has failed generate resources on db1169:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [05:21:49] (PuppetZeroResources) resolved: Puppet has failed generate resources on db1217:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [05:28:18] (PuppetZeroResources) firing: (2) Puppet has failed generate resources on db1209:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [05:35:19] (PuppetZeroResources) firing: Puppet has failed generate resources on db1185:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [05:43:18] (PuppetZeroResources) firing: (2) Puppet has failed generate resources on db1209:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [05:45:18] (PuppetZeroResources) resolved: Puppet has failed generate resources on db1185:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [05:48:18] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on db1207:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [05:53:18] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1200:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [05:58:18] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1200:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:01:19] (PuppetZeroResources) firing: (2) Puppet has failed generate resources on db1185:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:08:19] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1200:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:18:18] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on db1200:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:26:33] (PuppetZeroResources) firing: (2) Puppet has failed generate resources on db1175:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:28:19] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on db1200:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:31:18] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on db1175:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:33:18] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1200:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:36:18] (PuppetZeroResources) firing: (3) Puppet has failed generate resources on db1175:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:41:18] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1175:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:43:18] (PuppetZeroResources) firing: (6) Puppet has failed generate resources on db1200:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:46:18] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1167:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:51:18] (PuppetZeroResources) firing: (5) Puppet has failed generate resources on db1167:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:53:18] (PuppetZeroResources) firing: (5) Puppet has failed generate resources on db1215:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:56:18] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1167:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [06:58:18] (PuppetZeroResources) firing: (5) Puppet has failed generate resources on db1215:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:01:18] (PuppetZeroResources) firing: (5) Puppet has failed generate resources on db1167:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:03:18] (PuppetZeroResources) firing: (6) Puppet has failed generate resources on db1215:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:08:18] (PuppetZeroResources) firing: (7) Puppet has failed generate resources on db1215:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:11:18] (PuppetZeroResources) resolved: (4) Puppet has failed generate resources on db1167:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:13:18] (PuppetZeroResources) firing: (9) Puppet has failed generate resources on db1204:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:18:18] (PuppetZeroResources) firing: (8) Puppet has failed generate resources on db1204:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:23:19] (PuppetZeroResources) firing: (9) Puppet has failed generate resources on db1204:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:26:19] (PuppetZeroResources) firing: (2) Puppet has failed generate resources on db1190:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:33:33] (PuppetZeroResources) firing: (8) Puppet has failed generate resources on db1204:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:36:19] (PuppetZeroResources) resolved: Puppet has failed generate resources on es1023:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:41:18] (PuppetZeroResources) firing: (2) Puppet has failed generate resources on db1189:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:43:18] (PuppetZeroResources) firing: (8) Puppet has failed generate resources on db1204:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:51:18] (PuppetZeroResources) firing: (5) Puppet has failed generate resources on db1118:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:53:18] (PuppetZeroResources) firing: (7) Puppet has failed generate resources on db1204:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:53:33] (PuppetZeroResources) firing: (7) Puppet has failed generate resources on db1204:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [07:58:18] (PuppetZeroResources) firing: (7) Puppet has failed generate resources on db1204:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [08:01:18] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1118:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [08:23:18] (PuppetZeroResources) resolved: Puppet has failed generate resources on db1219:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [08:31:19] (PuppetZeroResources) resolved: Puppet has failed generate resources on db1118:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [08:32:46] argh [08:35:48] (PuppetZeroResources) firing: Puppet has failed generate resources on db1225:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [08:36:33] (PuppetZeroResources) firing: (2) Puppet has failed generate resources on db1219:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [08:46:48] (PuppetZeroResources) firing: Puppet has failed generate resources on es1031:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:00:48] (PuppetZeroResources) firing: (2) Puppet has failed generate resources on db1225:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:01:48] (PuppetZeroResources) firing: (2) Puppet has failed generate resources on db1197:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:03:52] PROBLEM - MariaDB sustained replica lag on s6 on db1180 is CRITICAL: 85.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1180&var-port=9104 [09:05:04] PROBLEM - MariaDB sustained replica lag on s8 on db2195 is CRITICAL: 56.75 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2195&var-port=9104 [09:05:49] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1225:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:05:52] RECOVERY - MariaDB sustained replica lag on s6 on db1180 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db1180&var-port=9104 [09:06:48] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1165:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:07:04] RECOVERY - MariaDB sustained replica lag on s8 on db2195 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2195&var-port=9104 [09:10:49] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1225:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:11:48] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1165:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:15:49] (PuppetZeroResources) resolved: (4) Puppet has failed generate resources on db1225:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:16:48] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1165:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:21:49] (PuppetZeroResources) firing: (4) Puppet has failed generate resources on db1165:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:31:48] (PuppetZeroResources) resolved: (3) Puppet has failed generate resources on db1165:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetZeroResources [09:38:53] Hopefully that's that done [09:42:04] 🤞 [09:42:16] the signal/noise ratio was not in favor of signal [09:46:19] yeah, g.odog has just put through a change that will downgrade this to a warning, too [11:43:38] Once I'm back I will need to take a look at the intermittent replag alerts too. We didn't have them until a couple months ago. I think there's a regression in mw [14:24:20] anyone working with db2149? [14:24:56] not me [14:25:40] thanks jaime, let's see if there's someone else [14:25:47] If not I will repool [14:48:19] I think Amir*1 was working on it: https://sal.toolforge.org/production?p=0&q=db2149&d= [14:48:53] yeah I think he was using it to reclone another host [14:49:03] But I think that's probably done [14:49:27] The host looks fine and show processlist is clean to [14:49:28] SAL says it is anyway :D [14:49:53] imho it's safe to repool, also cloning process does not impact the source host as long as cloning is over [14:51:03] yep, I am going to repool then [14:51:05] Thanks :* [15:38:39] Emperor: hey, rack B2 which we are hoping to work on in codfw has thanos-fe2002 as well as ms-fe2010 and ms-fe2014 in it [15:38:59] you noted before swift would have to be depooled for that latter? [15:41:41] topranks: yep, as before will be aiming to do these shortly before 16:00 [15:41:58] [am slightly multitasking with a ceph user/dev meeting] [15:41:59] ok that's great just confirming [15:42:02] thanks! [15:42:22] no pressure just let me know when it's done we can hold off till you're ready [15:54:26] topranks: depooling done, you're GTG from a swift perspective [15:54:41] Emperor: great, thanks! [15:54:47] we'll kick off at the top of the hour [15:55:11] 👍 [16:08:17] Emperor: we're all done, those hosts back responding to pings now [16:09:15] cool, ta, will repool shortly