[08:35:54] Hi inflatador, sorry for the delay. I was taking a look at https://phabricator.wikimedia.org/T412447 [08:38:03] Hi inflatador, sorry for the delay. I was taking a look at T412447. IIUC, you need to monitor your service to verify reachability and TLS certificate validity. I think most of what you need is already in place; you might want to take a look at hieradata/common/services.yaml (e.g. citoid) to see how monitoring is set up. [08:38:04] T412447: OpenSearch on K8s: Monitor and rotate TLS certificates - https://phabricator.wikimedia.org/T412447 [08:44:01] inflatador: Feel free to reach out if you need more details or if I missed the point :) [09:20:34] FIRING: DiskSpace: Disk space titan2001:9100:/srv 0.4794% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=titan2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:21:28] Hey claime, sorry for the delay. I just checked, and any (additional) labels or annotations related to an alert are sent together with the alert itself into the Phab task. [09:25:34] RESOLVED: DiskSpace: Disk space titan2001:9100:/srv 0.4794% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=titan2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:39:34] FIRING: DiskSpace: Disk space titan2001:9100:/srv 2.786% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=titan2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [09:49:34] RESOLVED: DiskSpace: Disk space titan2001:9100:/srv 2.773% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=titan2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [10:26:28] tappof: Cool thanks! [10:40:12] FIRING: ThanosCompactHalted: Thanos Compact has failed to run and is now halted. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/651943d05a8123e32867b4673963f42b/thanos-compact - https://alerts.wikimedia.org/?q=alertname%3DThanosCompactHalted [12:29:47] I wanted to initiate a swap for one day with Arnaud in January in OCO; but I don't see any shifts, neither my own or the ones by others, can someone please look into it? [13:11:27] moritzm: not even in https://app.oncall-optimizer.com/my/swaps ? [13:12:56] they are listed there, but not in the calendar view shown under the "Home" menu button [13:13:16] only those for Dec, but not the coming quarter [13:13:51] ok but from there you can click the 2 arrows button and get to the swap form no? [14:18:10] I don't want to initiate a swap for my hosts, Arnaud took over a day from me last month and now I want to take a Sunday from him in January. and that doesn't work since the shifts are not listed in the calendar [14:40:27] FIRING: ThanosCompactHalted: Thanos Compact has failed to run and is now halted. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/651943d05a8123e32867b4673963f42b/thanos-compact - https://alerts.wikimedia.org/?q=alertname%3DThanosCompactHalted [14:55:12] RESOLVED: ThanosCompactHalted: Thanos Compact has failed to run and is now halted. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/651943d05a8123e32867b4673963f42b/thanos-compact - https://alerts.wikimedia.org/?q=alertname%3DThanosCompactHalted [15:40:20] hey moritzm I see the same thing and raised an issue with OCO, should get looked at today during working hours in NZ [15:48:51] who is in change of the coordination while Hugh is out? I suddently got a totally different shift assigned for Jan 8 which sounds like a bug as well [15:49:06] moritzm: I am around, trying to fix atm [15:49:24] ok, sorry for distburing your time off, I'll send you a mail [15:50:05] Sorry for the noise [17:14:34] FIRING: DiskSpace: Disk space titan2001:9100:/srv 1.123% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=titan2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [17:14:46] ^^ it's me [17:19:34] RESOLVED: DiskSpace: Disk space titan2001:9100:/srv 0.4666% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=titan2001 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [18:00:12] FIRING: ThanosCompactHalted: Thanos Compact has failed to run and is now halted. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/651943d05a8123e32867b4673963f42b/thanos-compact - https://alerts.wikimedia.org/?q=alertname%3DThanosCompactHalted [22:00:27] FIRING: ThanosCompactHalted: Thanos Compact has failed to run and is now halted. - https://wikitech.wikimedia.org/wiki/Thanos#Alerts - https://grafana.wikimedia.org/d/651943d05a8123e32867b4673963f42b/thanos-compact - https://alerts.wikimedia.org/?q=alertname%3DThanosCompactHalted