[00:17:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:12:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:00:08] RECOVERY - db151 Backups SQL on db151 is OK: FILE_AGE OK: /var/log/sql-backup.log is 7 seconds old and 0 bytes [06:44:00] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [06:57:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:13:39] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:57:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:01:15] PROBLEM - graphite151 NTP time on graphite151 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [13:03:17] PROBLEM - graphite151 NTP time on graphite151 is CRITICAL: NTP CRITICAL: Offset 104.0162099 secs [15:32:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:14:35] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:44:15] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [16:57:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [17:16:02] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:45:41] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [18:28:20] PROBLEM - wiki.kingmingshrimp.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kingmingshrimp.com' expires in 15 day(s) (Tue 28 May 2024 06:22:08 PM GMT +0000). [18:28:33] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/602954161bce...f952f7789acb [18:28:36] [02ssl] 07WikiTideSSLBot 03f952f77 - Bot: Update SSL cert for wiki.kingmingshrimp.com [18:55:39] RECOVERY - Host cp51 is UP: PING OK - Packet loss = 0%, RTA = 163.63 ms [18:55:41] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 163.53 ms [18:55:44] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [18:55:55] RECOVERY - cp51 Nginx Backend for mw162 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8116 [18:55:59] RECOVERY - cp51 HTTPS on cp51 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3780 bytes in 1.226 second response time [18:56:09] RECOVERY - cp51 NTP time on cp51 is OK: NTP OK: Offset -0.002754420042 secs [18:56:15] RECOVERY - cp51 Nginx Backend for swiftproxy161 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8206 [18:56:17] RECOVERY - cp51 PowerDNS Recursor on cp51 is OK: DNS OK: 0.482 seconds response time. wikitide.net returns 109.123.230.163,2400:d320:2161:9775::1 [18:56:24] RECOVERY - cp51 Nginx Backend for reports171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8205 [18:56:25] RECOVERY - cp51 Nginx Backend for mwtask181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8160 [18:56:27] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [18:56:27] RECOVERY - cp51 Nginx Backend for test151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8181 [18:56:47] RECOVERY - cp51 Nginx Backend for mw171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8117 [18:56:47] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 3% [18:56:48] RECOVERY - cp51 Nginx Backend for mon181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8201 [18:56:52] RECOVERY - cp51 Nginx Backend for mw152 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8114 [18:56:52] RECOVERY - cp51 Nginx Backend for mw182 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8120 [18:57:00] RECOVERY - cp51 Nginx Backend for phorge171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8202 [18:57:01] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [18:57:02] RECOVERY - cp51 Nginx Backend for mwtask171 on cp51 is OK: TCP OK - 0.001 second response time on localhost port 8161 [18:57:03] RECOVERY - cp51 Nginx Backend for mw161 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8115 [18:57:06] RECOVERY - cp51 Nginx Backend for matomo151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8203 [18:57:08] RECOVERY - cp51 Nginx Backend for puppet181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8204 [18:57:10] RECOVERY - cp51 Nginx Backend for mw181 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8119 [18:57:11] RECOVERY - cp51 ferm_active on cp51 is OK: OK ferm input default policy is set [18:57:11] RECOVERY - cp51 Current Load on cp51 is OK: LOAD OK - total load average: 0.19, 0.15, 0.06 [18:57:11] RECOVERY - cp51 SSH on cp51 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [18:57:14] RECOVERY - cp51 Nginx Backend for mw151 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8113 [18:57:21] RECOVERY - cp51 Nginx Backend for swiftproxy171 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8207 [18:57:23] RECOVERY - cp51 conntrack_table_size on cp51 is OK: OK: nf_conntrack is 0 % full [18:57:26] RECOVERY - cp51 Nginx Backend for mw172 on cp51 is OK: TCP OK - 0.000 second response time on localhost port 8118 [18:57:26] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:57:31] RECOVERY - cp51 Disk Space on cp51 is OK: DISK OK - free space: / 18366MiB (19% inode=98%); [18:58:02] RECOVERY - wiki.kingmingshrimp.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kingmingshrimp.com' will expire on Sat 10 Aug 2024 05:28:27 PM GMT +0000. [19:18:28] PROBLEM - wiki.oasismp.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.oasismp.com' expires in 15 day(s) (Tue 28 May 2024 07:01:00 PM GMT +0000). [19:18:39] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f952f7789acb...642c3582d5a5 [19:18:41] [02ssl] 07WikiTideSSLBot 03642c358 - Bot: Update SSL cert for wiki.oasismp.com [20:16:57] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:18:23] RECOVERY - wiki.oasismp.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.oasismp.com' will expire on Sat 10 Aug 2024 06:18:33 PM GMT +0000. [21:32:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [21:45:52] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [21:48:27] PROBLEM - cp51 NTP time on cp51 is WARNING: NTP WARNING: Offset 0.4431277514 secs [22:02:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [22:02:27] RECOVERY - cp51 NTP time on cp51 is OK: NTP OK: Offset -0.0001059770584 secs [22:36:53] PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [23:04:58] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures