[06:13:21] <XioNoX>	 came here to report the same issue as above - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale
[07:14:44] <tappof>	 I'll take a look
[07:16:16] <tappof>	 This is related to a check I enabled and then reverted. Sorry for the noise, I’ll clean up the alerts in the morning.
[07:27:59] <vgutierrez>	 thx tappof 
[08:17:20] <tappof>	 All the alerts are now RESOLVED. Sorry once again for all the noise.
[08:18:30] <tappof>	 In short, the notification storm was caused by the patch gerrit.wikimedia.org/r/c/operations/puppet/+/1180501, where I disabled the nrpe2nodexp wrapper on check_disk_space (since it can be natively accomplished through the metrics exported by node_exporter) and enabled it on check_ferm_active. I forgot to clean up the node.d directory, which is why the alerts were triggered.
[08:18:59] <tappof>	 sukhe: XioNoX vgutierrez 
[09:40:30] <XioNoX>	 thx! no pb!
[13:19:21] <sukhe>	 thanks tappof!
[13:34:42] <sukhe>	 hi folks. is anyone working on Grafana? it is down atm
[13:36:23] <tappof>	 I don’t know of any planned activities, I’ll take a look.
[13:36:37] <sukhe>	 it's back 
[13:36:40] <sukhe>	 that was weird
[13:36:44] <tappof>	 yeah
[13:36:46] <sukhe>	 > 09:32:55 <+icinga-wm> PROBLEM - grafana-rw.wikimedia.org requires authentication on grafana1002 is CRITICAL: CRITICAL - Socket 
[14:34:19] <denisse>	 We experienced that issue before, when it can't communicate to the SSO something causes grafana to go down, I'll investigate it further.
[20:46:40] <jinxer-wm>	 FIRING: LogstashClusterStatus: OpenSearch reports cluster status is red. - https://wikitech.wikimedia.org/wiki/Logstash#Unassigned_Shards_and_Cluster_Status - https://grafana.wikimedia.org/d/000000561/logstash?viewPanel=panel-49 - https://alerts.wikimedia.org/?q=alertname%3DLogstashClusterStatus
[20:51:40] <jinxer-wm>	 FIRING: [2x] LogstashClusterStatus: OpenSearch reports cluster status is red. - https://wikitech.wikimedia.org/wiki/Logstash#Unassigned_Shards_and_Cluster_Status - https://grafana.wikimedia.org/d/000000561/logstash?viewPanel=panel-49 - https://alerts.wikimedia.org/?q=alertname%3DLogstashClusterStatus
[20:51:56] <cwhite>	 bah, Imma fix that
[21:01:40] <jinxer-wm>	 RESOLVED: [2x] LogstashClusterStatus: OpenSearch reports cluster status is red. - https://wikitech.wikimedia.org/wiki/Logstash#Unassigned_Shards_and_Cluster_Status - https://grafana.wikimedia.org/d/000000561/logstash?viewPanel=panel-49 - https://alerts.wikimedia.org/?q=alertname%3DLogstashClusterStatus
[21:01:52] <inflatador>	 {◕ ◡ ◕}