[00:01:06] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.07, 3.85, 3.18 [00:01:17] RECOVERY - gs.sidem.wiki - reverse DNS on sslhost is OK: SSL OK - gs.sidem.wiki reverse DNS resolves to cp22.miraheze.org - CNAME OK [00:03:02] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.96, 3.63, 3.08 [00:03:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.45, 3.80, 3.25 [00:04:59] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.78, 4.81, 3.72 [00:05:01] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.89, 3.06, 2.94 [00:09:00] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.31, 6.10, 3.23 [00:10:50] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.18, 3.96, 3.70 [00:11:00] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.97, 6.41, 3.73 [00:14:43] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.11, 3.10, 3.37 [00:18:39] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.63, 3.47, 3.52 [00:20:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.60, 2.81, 3.27 [00:24:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.61, 4.11, 3.69 [00:25:00] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.85, 6.67, 4.75 [00:26:28] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.13, 3.65, 3.56 [00:27:07] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1017121375 secs [00:28:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.45, 4.12, 3.76 [00:29:00] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.19, 6.25, 5.06 [00:30:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.32, 3.48, 3.57 [00:30:46] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:32:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.19, 3.63, 3.60 [00:35:43] RECOVERY - cp23 APT on cp23 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [00:36:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.45, 3.72, 3.64 [00:41:49] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.89, 7.81, 6.16 [00:42:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.54, 2.91, 3.30 [00:45:37] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.96, 7.49, 6.45 [00:47:31] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 4.83, 6.71, 6.29 [00:53:07] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.01262813807 secs [00:58:23] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.43, 4.08, 3.08 [01:00:23] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.91, 3.55, 3.00 [01:02:23] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.32, 3.30, 2.96 [01:14:59] PROBLEM - matomo131 Current Load on matomo131 is CRITICAL: CRITICAL - load average: 4.31, 3.25, 2.51 [01:16:59] RECOVERY - matomo131 Current Load on matomo131 is OK: OK - load average: 2.19, 2.94, 2.50 [01:18:43] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.08, 4.72, 3.53 [01:19:32] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:22:36] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.11, 3.93, 3.48 [01:24:32] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.36, 4.89, 3.89 [01:28:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.13, 3.36, 3.48 [01:30:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.61, 3.06, 3.35 [01:48:24] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [02:17:16] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:24:26] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.96, 2.98, 2.36 [02:26:24] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.90, 2.99, 2.44 [02:46:53] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [02:52:44] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1036380827 secs [03:00:44] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09879720211 secs [03:07:25] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.89, 4.38, 3.31 [03:09:21] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.64, 3.90, 3.27 [03:11:18] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.51, 3.33, 3.13 [03:16:52] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [03:31:27] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.66, 3.41, 3.03 [03:33:25] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.86, 3.12, 2.96 [03:46:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.88, 3.97, 2.99 [03:48:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.33, 3.14, 2.79 [04:09:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.84, 9.48, 8.35 [04:11:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.93, 9.32, 8.45 [04:28:53] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.4753462076 secs [04:32:53] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset -0.0006695091724 secs [05:05:38] PROBLEM - matomo131 HTTPS on matomo131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 358 bytes in 0.024 second response time [05:10:15] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 12566 MB (9% inode=98%); [05:21:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.55, 10.23, 9.00 [05:22:15] PROBLEM - db112 Disk Space on db112 is CRITICAL: DISK CRITICAL - free space: / 7808 MB (5% inode=98%); [05:23:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.00, 9.72, 8.96 [05:27:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.60, 10.74, 9.55 [05:28:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.97, 10.78, 9.42 [05:29:59] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.23, 11.19, 9.73 [05:31:56] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.81, 10.70, 9.74 [05:33:54] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.61, 11.36, 10.08 [05:35:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.29, 9.89, 9.66 [05:35:52] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.63, 10.90, 10.08 [05:37:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.83, 10.03, 9.88 [05:48:20] PROBLEM - mail121 webmail.miraheze.org HTTPS on mail121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:59:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.72, 11.45, 9.73 [06:01:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.12, 11.31, 9.91 [06:03:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.42, 12.91, 10.64 [06:06:18] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.39, 12.07, 10.30 [06:10:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.33, 11.50, 10.59 [06:10:15] !log removed binlogs and rebooted db112 [06:10:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:11:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.61, 11.96, 11.09 [06:11:30] [Grafana] !sre FIRING: There has been a rise in the MediaWiki exception rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [06:11:40] RECOVERY - mail121 webmail.miraheze.org HTTPS on mail121 is OK: HTTP OK: HTTP/1.1 200 OK - 1213 bytes in 0.066 second response time [06:12:06] PROBLEM - db112 SSH on db112 is CRITICAL: connect to address 2a10:6740::6:211 and port 22: Connection refused [06:12:17] PROBLEM - db112 APT on db112 is CRITICAL: connect to address 2a10:6740::6:211 port 5666: Connection refusedconnect to host 2a10:6740::6:211 port 5666: Connection refused [06:12:17] PROBLEM - db112 ferm_active on db112 is CRITICAL: connect to address 2a10:6740::6:211 port 5666: Connection refusedconnect to host 2a10:6740::6:211 port 5666: Connection refused [06:12:28] PROBLEM - db112 Backups SQL grafana on db112 is CRITICAL: connect to address 2a10:6740::6:211 port 5666: Connection refusedconnect to host 2a10:6740::6:211 port 5666: Connection refused [06:12:39] PROBLEM - db112 Puppet on db112 is CRITICAL: connect to address 2a10:6740::6:211 port 5666: Connection refusedconnect to host 2a10:6740::6:211 port 5666: Connection refused [06:12:48] PROBLEM - db112 Backups SQL icingaweb2 on db112 is CRITICAL: connect to address 2a10:6740::6:211 port 5666: Connection refusedconnect to host 2a10:6740::6:211 port 5666: Connection refused [06:12:50] PROBLEM - db112 MariaDB Connections on db112 is CRITICAL: PHP Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Warning: mysqli_real_connect(): (HY000/2002): Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66Connection failed: Connection refused [06:12:54] PROBLEM - db112 Backups SQL roundcubemail on db112 is CRITICAL: connect to address 2a10:6740::6:211 port 5666: Connection refusedconnect to host 2a10:6740::6:211 port 5666: Connection refused [06:13:05] PROBLEM - mon141 grafana.miraheze.org HTTPS on mon141 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.013 second response time [06:13:06] PROBLEM - db112 Current Load on db112 is CRITICAL: connect to address 2a10:6740::6:211 port 5666: Connection refusedconnect to host 2a10:6740::6:211 port 5666: Connection refused [06:13:07] PROBLEM - db112 Backups SQL on db112 is CRITICAL: connect to address 2a10:6740::6:211 port 5666: Connection refusedconnect to host 2a10:6740::6:211 port 5666: Connection refused [06:13:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.07, 13.00, 11.57 [06:13:20] PROBLEM - db112 NTP time on db112 is CRITICAL: connect to address 2a10:6740::6:211 port 5666: Connection refusedconnect to host 2a10:6740::6:211 port 5666: Connection refused [06:13:21] PROBLEM - db112 PowerDNS Recursor on db112 is CRITICAL: connect to address 2a10:6740::6:211 port 5666: Connection refusedconnect to host 2a10:6740::6:211 port 5666: Connection refused [06:13:21] PROBLEM - db112 Backups SQL reports on db112 is CRITICAL: connect to address 2a10:6740::6:211 port 5666: Connection refusedconnect to host 2a10:6740::6:211 port 5666: Connection refused [06:13:22] PROBLEM - mon141 HTTPS on mon141 is WARNING: HTTP WARNING: HTTP/1.1 403 Forbidden - 366 bytes in 0.011 second response time [06:13:32] PROBLEM - db112 conntrack_table_size on db112 is CRITICAL: connect to address 2a10:6740::6:211 port 5666: Connection refusedconnect to host 2a10:6740::6:211 port 5666: Connection refused [06:13:38] PROBLEM - phab121 phabricator.miraheze.org HTTPS on phab121 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 4167 bytes in 0.098 second response time [06:13:46] PROBLEM - db112 Backups SQL on db112 is CRITICAL: connect to address 2a10:6740::6:211 port 5666: Connection refusedconnect to host 2a10:6740::6:211 port 5666: Connection refused [06:13:52] PROBLEM - db112 Backups SQL icinga on db112 is CRITICAL: connect to address 2a10:6740::6:211 port 5666: Connection refusedconnect to host 2a10:6740::6:211 port 5666: Connection refused [06:13:54] PROBLEM - phab121 phab.miraheze.wiki HTTPS on phab121 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 500 Internal Server Error [06:13:55] PROBLEM - db112 MariaDB on db112 is CRITICAL: Can't connect to MySQL server on 'db112.miraheze.org' (115) [06:14:08] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.33, 12.57, 11.12 [06:15:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.64, 11.46, 11.19 [06:16:06] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.38, 11.51, 10.90 [06:18:04] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.08, 11.58, 10.98 [06:19:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.58, 12.69, 11.65 [06:21:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.94, 11.40, 11.16 [06:23:00] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.31, 6.27, 3.80 [06:23:56] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.26, 12.37, 11.53 [06:25:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.10, 11.31, 11.46 [06:25:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.86, 11.21, 11.23 [06:27:00] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.24, 6.78, 4.64 [06:27:02] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.44, 3.43, 2.30 [06:27:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.22, 12.93, 12.01 [06:27:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.25, 12.99, 11.89 [06:28:06] RECOVERY - db112 SSH on db112 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [06:28:08] RECOVERY - db112 APT on db112 is OK: APT OK: 31 packages available for upgrade (0 critical updates). [06:28:17] RECOVERY - db112 ferm_active on db112 is OK: OK ferm input default policy is set [06:28:28] RECOVERY - db112 Backups SQL grafana on db112 is OK: FILE_AGE OK: /var/log/sql-grafana-backup-fortnightly.log is 782897 seconds old and 88 bytes [06:28:39] RECOVERY - db112 Puppet on db112 is OK: OK: Puppet is currently enabled, last run 27 minutes ago with 0 failures [06:28:42] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 11.42, 6.40, 3.03 [06:28:48] RECOVERY - db112 Backups SQL icingaweb2 on db112 is OK: FILE_AGE OK: /var/log/sql-icingaweb2-backup-fortnightly.log is 782920 seconds old and 88 bytes [06:28:50] RECOVERY - db112 MariaDB Connections on db112 is OK: OK connection usage: 1.8%Current connections: 9 [06:28:54] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.52, 4.12, 2.68 [06:28:54] RECOVERY - db112 Backups SQL roundcubemail on db112 is OK: FILE_AGE OK: /var/log/sql-roundcubemail-backup-fortnightly.log is 782926 seconds old and 88 bytes [06:29:00] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.18, 3.38, 2.42 [06:29:05] RECOVERY - mon141 grafana.miraheze.org HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 40338 bytes in 0.052 second response time [06:29:07] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.41, 0.86, 0.31 [06:29:07] RECOVERY - db112 Backups SQL on db112 is OK: FILE_AGE OK: /var/log/sql-backup.log is 185344 seconds old and 363 bytes [06:29:17] RECOVERY - db112 NTP time on db112 is OK: NTP OK: Offset -0.0002075731754 secs [06:29:22] RECOVERY - db112 PowerDNS Recursor on db112 is OK: DNS OK: 0.094 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [06:29:22] RECOVERY - db112 Backups SQL reports on db112 is OK: FILE_AGE OK: /var/log/sql-reports-backup-weekly.log is 437352 seconds old and 88 bytes [06:29:22] RECOVERY - mon141 HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.035 second response time [06:29:32] RECOVERY - db112 conntrack_table_size on db112 is OK: OK: nf_conntrack is 0 % full [06:29:38] RECOVERY - matomo131 HTTPS on matomo131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.158 second response time [06:29:38] RECOVERY - phab121 phabricator.miraheze.org HTTPS on phab121 is OK: HTTP OK: HTTP/1.1 200 OK - 18991 bytes in 0.329 second response time [06:29:46] RECOVERY - db112 Backups SQL on db112 is OK: FILE_AGE OK: /var/log/sql-backup.log is 185383 seconds old and 363 bytes [06:29:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.27, 11.76, 11.60 [06:29:52] RECOVERY - db112 Backups SQL icinga on db112 is OK: FILE_AGE OK: /var/log/sql-icinga-backup-fortnightly.log is 782841 seconds old and 89 bytes [06:29:54] RECOVERY - phab121 phab.miraheze.wiki HTTPS on phab121 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17684 bytes in 0.159 second response time [06:29:55] RECOVERY - db112 MariaDB on db112 is OK: Uptime: 93 Threads: 9 Questions: 64906 Slow queries: 798 Opens: 147 Open tables: 141 Queries per second avg: 697.913 [06:30:50] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.71, 3.65, 2.69 [06:31:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.30, 11.11, 11.54 [06:31:30] [Grafana] !sre RESOLVED: MediaWiki Exception Rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [06:32:41] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.25, 3.77, 2.73 [06:32:47] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.11, 3.30, 2.66 [06:34:41] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.86, 2.74, 2.47 [06:35:07] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1311622858 secs [06:35:42] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.24, 8.39, 10.13 [06:41:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.44, 8.53, 9.90 [06:51:41] PROBLEM - cloud13 Puppet on cloud13 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [06:55:07] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.01081183553 secs [07:05:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.21, 10.14, 8.29 [07:07:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.66, 8.66, 7.97 [07:13:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.94, 11.20, 9.35 [07:15:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.56, 9.77, 9.08 [07:18:52] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [07:21:41] RECOVERY - cloud13 Puppet on cloud13 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:47:45] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [08:36:24] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.99, 3.70, 2.09 [08:40:23] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.70, 3.64, 2.56 [08:42:23] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 0.60, 2.54, 2.28 [08:48:25] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.49, 3.08, 2.52 [08:49:49] !log deleted /srv/backups/dbs stored locally on db112 [08:49:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:50:15] RECOVERY - db112 Disk Space on db112 is OK: DISK OK - free space: / 24118 MB (18% inode=98%); [08:50:23] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.19, 2.42, 2.34 [09:22:23] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [09:26:23] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 7 = Critical] [10:02:45] [02mw-config] 07redbluegreenhat opened pull request 03#5165: T10640: Set some wgDefaultUserOptions values on landarwiki - 13https://github.com/miraheze/mw-config/pull/5165 [10:03:49] miraheze/mw-config - redbluegreenhat the build passed. [11:25:31] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 27.54, 10.97, 5.11 [11:27:28] PROBLEM - cp33 APT on cp33 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:29:31] RECOVERY - cp33 APT on cp33 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [11:30:44] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1665341854 secs [11:38:44] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.06711837649 secs [11:42:36] PROBLEM - ecole.science - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.404361963272095 seconds [11:49:15] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.96, 2.60, 3.85 [11:50:09] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [11:55:10] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 9.86, 5.94, 4.70 [12:03:04] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.37, 3.14, 3.87 [12:04:08] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.66, 9.74, 8.28 [12:06:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.50, 9.44, 8.35 [12:07:01] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.04, 2.28, 3.37 [12:09:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.84, 11.40, 9.38 [12:11:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.89, 11.39, 9.62 [12:12:29] RECOVERY - ecole.science - reverse DNS on sslhost is OK: SSL OK - ecole.science reverse DNS resolves to cp23.miraheze.org - NS RECORDS OK [12:15:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.45, 9.17, 9.06 [12:18:48] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:19:24] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.59, 10.72, 9.68 [12:21:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.51, 11.45, 10.10 [12:27:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.84, 9.99, 9.84 [12:37:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.61, 10.14, 9.73 [12:41:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.57, 11.96, 10.47 [12:43:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.03, 10.81, 10.24 [12:43:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.20, 11.03, 9.87 [12:45:00] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.22, 6.48, 3.40 [12:45:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.31, 10.07, 9.63 [12:47:00] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.83, 6.34, 3.75 [12:48:09] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [12:49:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 16.43, 12.65, 11.04 [12:49:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.84, 11.52, 10.29 [12:53:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.94, 11.03, 10.82 [12:53:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.72, 11.49, 10.60 [12:55:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.96, 10.06, 10.18 [13:01:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.70, 10.94, 10.58 [13:03:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.21, 10.22, 10.37 [13:05:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.27, 9.66, 10.15 [13:06:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.72, 12.31, 11.04 [13:09:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.01, 12.68, 11.18 [13:12:13] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.58, 11.79, 11.38 [13:17:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.65, 11.97, 11.70 [13:19:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.33, 12.16, 11.80 [13:21:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.49, 11.50, 11.60 [13:23:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.58, 11.82, 11.69 [13:25:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.95, 11.54, 11.61 [13:29:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.25, 11.29, 11.45 [13:31:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.49, 10.32, 11.09 [13:33:47] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.25, 11.59, 10.91 [13:35:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.40, 11.50, 11.31 [13:35:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.63, 11.33, 10.89 [13:37:42] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.02, 11.67, 11.07 [13:39:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.68, 11.67, 11.14 [13:41:37] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.38, 12.40, 11.47 [13:53:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.73, 11.91, 11.83 [13:55:00] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.09, 7.17, 3.75 [13:57:00] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.12, 5.20, 3.46 [13:57:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.64, 12.31, 11.99 [13:59:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.18, 11.58, 11.77 [14:03:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.27, 10.69, 11.83 [14:03:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.96, 11.50, 11.63 [14:09:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.82, 11.86, 11.82 [14:11:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 17.17, 12.44, 11.91 [14:13:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.54, 12.14, 11.86 [14:15:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.26, 11.79, 11.89 [14:15:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.56, 11.14, 11.54 [14:17:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.56, 12.84, 12.08 [14:25:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.16, 11.40, 11.78 [14:31:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.67, 13.29, 12.30 [14:33:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.89, 11.88, 11.48 [14:37:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.81, 11.47, 11.51 [14:41:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.02, 11.95, 11.67 [14:41:28] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.23, 11.23, 11.82 [14:43:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.51, 10.96, 11.36 [14:51:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.80, 11.06, 11.07 [14:53:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.99, 9.94, 10.66 [14:53:28] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.09, 8.10, 9.82 [15:00:20] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=speculativeevolutionwiki --no-updates --username-prefix=wikia:speculativeevolution /home/reception/ImportDump/speculativeevolutionwiki-20230324143211.xml (START) [15:00:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:01:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.94, 9.25, 10.08 [15:05:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.23, 9.91, 10.20 [15:06:16] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.27, 10.55, 9.85 [15:07:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.04, 9.58, 10.05 [15:08:13] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.97, 9.70, 9.61 [15:10:10] [02mediawiki] 07AgentIsai opened pull request 03#13956: Install NearbyPages - 13https://github.com/miraheze/mediawiki/pull/13956 [15:11:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.13, 11.46, 10.71 [15:12:32] [02puppet] 07AgentIsai opened pull request 03#3170: Add composer for NearbyPages - 13https://github.com/miraheze/puppet/pull/3170 [15:13:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.16, 10.97, 10.63 [15:17:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.00, 9.43, 10.09 [15:18:02] [02mw-config] 07AgentIsai opened pull request 03#5166: Add NearbyPages to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/5166 [15:19:03] miraheze/mw-config - AgentIsai the build passed. [15:22:02] [02mediawiki] 07AgentIsai synchronize pull request 03#13956: Install NearbyPages - 13https://github.com/miraheze/mediawiki/pull/13956 [15:22:58] [02mediawiki] 07AgentIsai synchronize pull request 03#13956: Install NearbyPages - 13https://github.com/miraheze/mediawiki/pull/13956 [15:23:21] [02mediawiki] 07AgentIsai synchronize pull request 03#13956: Install NearbyPages - 13https://github.com/miraheze/mediawiki/pull/13956 [15:26:01] PROBLEM - test131 MediaWiki Rendering on test131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 314 bytes in 0.031 second response time [15:27:07] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1324084997 secs [15:27:50] [02mediawiki] 07AgentIsai synchronize pull request 03#13956: Install NearbyPages - 13https://github.com/miraheze/mediawiki/pull/13956 [15:28:01] RECOVERY - test131 MediaWiki Rendering on test131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.380 second response time [15:28:34] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/a0f54ab42580 [15:28:35] [02miraheze/mw-config] 07Reception123 03a0f54ab - add SRE recruitment sitenotice for Tech NS on meta [15:28:38] [02mw-config] 07Reception123 created branch 03Reception123-patch-3 - 13https://github.com/miraheze/mw-config [15:28:41] [02mw-config] 07Reception123 opened pull request 03#5167: add SRE recruitment sitenotice for Tech NS on meta - 13https://github.com/miraheze/mw-config/pull/5167 [15:29:11] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a0f54ab42580...1fb62544c20a [15:29:13] [02miraheze/mw-config] 07github-actions 031fb6254 - CI: lint code to MediaWiki standards [15:29:14] [02mw-config] 07github-actions[bot] synchronize pull request 03#5167: add SRE recruitment sitenotice for Tech NS on meta - 13https://github.com/miraheze/mw-config/pull/5167 [15:29:43] miraheze/mw-config - Reception123 the build passed. [15:30:49] [02mw-config] 07Reception123 synchronize pull request 03#5167: add SRE recruitment sitenotice for Tech NS on meta - 13https://github.com/miraheze/mw-config/pull/5167 [15:30:50] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/1fb62544c20a...314727169bb9 [15:30:53] [02miraheze/mw-config] 07Reception123 033147271 - correct link [15:32:05] miraheze/mw-config - Reception123 the build passed. [15:33:19] [02mediawiki] 07Reception123 closed pull request 03#13956: Install NearbyPages - 13https://github.com/miraheze/mediawiki/pull/13956 [15:33:20] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_39 [+1/-0/±1] 13https://github.com/miraheze/mediawiki/compare/af88d37bb758...1410447877f0 [15:33:22] [02miraheze/mediawiki] 07AgentIsai 031410447 - Install NearbyPages (#13956) [15:33:24] [02mw-config] 07Reception123 closed pull request 03#5167: add SRE recruitment sitenotice for Tech NS on meta - 13https://github.com/miraheze/mw-config/pull/5167 [15:33:26] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-3 [15:33:27] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-3 - 13https://github.com/miraheze/mw-config [15:33:30] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/91eb24b697d3...76e1408891df [15:33:31] [02miraheze/mw-config] 07Reception123 0376e1408 - add SRE recruitment sitenotice for Tech NS on meta (#5167) [15:33:32] !log [reception@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'world': True} to all [15:33:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:34:32] miraheze/mw-config - Reception123 the build passed. [15:38:01] [02puppet] 07Reception123 closed pull request 03#3170: Add composer for NearbyPages - 13https://github.com/miraheze/puppet/pull/3170 [15:38:03] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/39d93de1fd5e...ab7224c7cd11 [15:38:06] [02miraheze/puppet] 07AgentIsai 03ab7224c - Add composer for NearbyPages (#3170) [15:39:29] [02mw-config] 07AgentIsai synchronize pull request 03#5166: Add NearbyPages to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/5166 [15:40:12] !log [reception@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [15:40:14] !log [reception@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'world': True} to all [15:40:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:40:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:40:30] miraheze/mw-config - AgentIsai the build passed. [15:43:44] PROBLEM - mwtask141 Puppet on mwtask141 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nearbypages_composer] [15:44:12] !log [reception@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [15:44:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:49:07] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09863805771 secs [15:50:09] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [15:57:13] !log [@test131] starting deploy of {'config': True} to all [15:57:14] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [15:57:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:57:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:01:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.47, 10.50, 9.38 [16:01:19] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[nearbypages_composer] [16:03:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.43, 10.50, 9.50 [16:05:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.71, 10.05, 9.45 [16:15:25] PROBLEM - 100acg.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate '100acg.wiki' expires in 15 day(s) (Sun 09 Apr 2023 15:57:29 GMT +0000). [16:17:26] PROBLEM - wiki.maxus.space - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.maxus.space' expires in 15 day(s) (Sun 09 Apr 2023 16:01:16 GMT +0000). [16:18:21] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=speculativeevolutionwiki --no-updates --username-prefix=wikia:speculativeevolution /home/reception/ImportDump/speculativeevolutionwiki-20230324143211.xml (END - exit=0) [16:18:22] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=speculativeevolutionwiki (START) [16:18:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:18:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:18:48] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:22:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a0f6ea33c01e...e7de83766351 [16:22:59] [02miraheze/ssl] 07MirahezeSSLBot 03e7de837 - Bot: Update SSL cert for 100acg.wiki [16:24:42] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e7de83766351...12490bf3e11d [16:24:45] [02miraheze/ssl] 07MirahezeSSLBot 0312490bf - Bot: Update SSL cert for wiki.maxus.space [16:27:59] PROBLEM - wiki.esnmilanostatale.it - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.esnmilanostatale.it' expires in 15 day(s) (Sun 09 Apr 2023 16:03:28 GMT +0000). [16:28:50] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/12490bf3e11d...b11f91c75120 [16:28:53] [02miraheze/ssl] 07MirahezeSSLBot 03b11f91c - Bot: Update SSL cert for wiki.esnmilanostatale.it [16:44:46] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=speculativeevolutionwiki (END - exit=0) [16:44:49] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=speculativeevolutionwiki --active --update (END - exit=0) [16:44:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:44:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:45:20] RECOVERY - 100acg.wiki - LetsEncrypt on sslhost is OK: OK - Certificate '100acg.wiki' will expire on Thu 22 Jun 2023 15:22:49 GMT +0000. [16:46:22] RECOVERY - wiki.maxus.space - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.maxus.space' will expire on Thu 22 Jun 2023 15:24:34 GMT +0000. [16:48:09] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [16:56:59] RECOVERY - wiki.esnmilanostatale.it - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.esnmilanostatale.it' will expire on Thu 22 Jun 2023 15:28:42 GMT +0000. [16:58:00] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.zulaclub.net. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:58:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.47, 3.40, 2.73 [16:59:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.29, 9.12, 8.58 [17:00:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.22, 3.82, 2.96 [17:01:19] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.66, 8.81, 8.53 [17:04:00] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.50, 3.71, 2.90 [17:07:57] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.18, 3.20, 2.87 [17:11:57] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.84, 11.56, 9.94 [17:12:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.23, 3.57, 3.60 [17:13:51] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.44, 10.78, 9.84 [17:16:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.42, 3.00, 3.35 [17:18:10] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:21:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.69, 9.55, 9.72 [17:25:42] !log [reception@mwtask141] starting deploy of {'world': True} to all [17:25:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:26:58] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [17:27:49] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.23, 10.24, 9.12 [17:29:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.25, 9.36, 8.95 [17:30:29] !log [reception@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [17:30:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:30:36] !log [reception@mwtask141] starting deploy of {'pull': 'world'} to all [17:30:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:34:08] !log [reception@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [17:34:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:41:07] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.2127160728 secs [17:56:01] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [18:05:07] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.02098569274 secs [18:05:22] !log [reception@mwtask141] starting deploy of {'world': True} to all [18:05:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:07:15] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.07, 3.81, 2.99 [18:09:12] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.52, 3.63, 3.02 [18:09:57] !log [reception@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [18:10:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:11:08] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.98, 2.93, 2.83 [18:17:01] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.39, 3.36, 2.97 [18:18:09] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [18:18:10] [02miraheze/mediawiki] 07Reception123 pushed 032 commits to 03REL1_39 [+1/-0/±40] 13https://github.com/miraheze/mediawiki/compare/1410447877f0...3480cd29000c [18:18:11] [02miraheze/mediawiki] 07Reception123 03945a824 - switch PageProperties to master/REL139 discontinued [18:18:14] [02miraheze/mediawiki] 07Reception123 033480cd2 - merge conflict [18:18:58] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.90, 3.24, 2.95 [18:20:55] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.36, 3.14, 2.94 [18:46:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.72, 3.85, 3.07 [18:48:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.43, 3.39, 3.01 [18:54:30] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.42, 3.04, 2.37 [18:56:29] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.06, 2.74, 2.34 [19:02:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.95, 4.37, 3.54 [19:04:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.75, 3.72, 3.40 [19:06:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.49, 4.20, 3.60 [19:08:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.85, 3.60, 3.44 [19:10:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.83, 3.03, 3.25 [19:48:02] PROBLEM - ping6 on cp33 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 104.28 ms [19:50:01] RECOVERY - ping6 on cp33 is OK: PING OK - Packet loss = 0%, RTA = 104.49 ms [20:17:02] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.30, 4.39, 3.22 [20:37:09] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 2.78, 4.02, 3.24 [20:39:06] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.49, 3.54, 3.16 [20:40:43] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.71, 2.26, 3.91 [20:42:59] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.40, 3.17, 3.13 [20:44:40] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.40, 1.80, 3.34 [20:50:09] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:51:29] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.19, 9.71, 8.91 [20:53:23] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.10, 8.88, 8.69 [21:48:09] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [21:51:11]