[00:02:45] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.96, 6.86, 6.63 [00:02:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.48, 7.19, 6.59 [00:04:36] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.41, 2.92, 1.51 [00:04:39] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.06, 7.29, 6.82 [00:04:49] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 18.93, 10.70, 4.90 [00:07:16] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 14.29, 11.83, 7.20 [00:08:44] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.88, 6.45, 6.51 [00:09:16] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 9.97, 11.98, 7.87 [00:11:16] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 4.73, 9.32, 7.39 [00:12:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.59, 7.60, 7.25 [00:12:26] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.88, 3.63, 2.74 [00:14:28] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.54, 2.93, 2.59 [00:14:49] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.94, 6.69, 6.18 [00:21:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.60, 8.02, 7.13 [00:23:05] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.51, 3.17, 2.46 [00:23:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.33, 7.05, 6.89 [00:24:37] PROBLEM - db131 Current Load on db131 is CRITICAL: CRITICAL - load average: 14.75, 9.59, 6.51 [00:25:01] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 6.37, 4.25, 2.94 [00:25:03] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.83, 6.33, 6.65 [00:26:12] PROBLEM - os131 PowerDNS Recursor on os131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:26:45] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.15, 4.09, 3.00 [00:27:26] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 13.69, 9.11, 5.72 [00:27:28] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 15.56, 11.05, 7.04 [00:28:18] RECOVERY - os131 PowerDNS Recursor on os131 is OK: DNS OK: 8.346 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [00:28:54] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.33, 6.71, 6.72 [00:29:22] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 11.60, 10.08, 6.50 [00:30:03] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fediverse.kr' expires in 9 day(s) (Mon 04 Dec 2023 20:50:46 GMT +0000). [00:31:19] PROBLEM - mw133 Current Load on mw133 is CRITICAL: CRITICAL - load average: 13.21, 10.80, 7.16 [00:32:18] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.72, 6.42, 6.77 [00:32:43] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.91, 6.65, 6.73 [00:33:15] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 8.71, 10.09, 7.36 [00:33:25] PROBLEM - mw134 Current Load on mw134 is WARNING: WARNING - load average: 9.24, 11.20, 8.57 [00:34:07] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 2 backends are down. mw133 mw134 [00:34:19] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw133 [00:34:30] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw133 [00:36:05] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [00:36:19] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [00:36:27] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.03, 3.54, 2.73 [00:36:30] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [00:37:26] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 7.02, 9.26, 8.40 [00:38:28] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.83, 3.69, 2.89 [00:42:28] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.86, 3.31, 2.93 [00:45:54] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.17, 7.17, 6.90 [00:46:09] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.94, 6.92, 6.80 [00:46:29] PROBLEM - mw134 Current Load on mw134 is CRITICAL: CRITICAL - load average: 12.24, 11.45, 9.54 [00:48:24] RECOVERY - mw134 Current Load on mw134 is OK: OK - load average: 6.83, 9.54, 9.05 [00:50:59] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.65, 3.52, 3.94 [00:51:37] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.09, 7.49, 7.08 [00:51:40] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [00:52:45] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:53:00] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.47, 3.89, 4.03 [00:53:31] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.46, 7.21, 7.04 [00:54:59] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.26, 3.74, 3.96 [00:56:58] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.04, 3.87, 3.98 [00:58:07] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.30, 6.55, 6.74 [00:59:02] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.07, 3.44, 3.80 [01:00:03] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.fediverse.kr and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:00:55] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 2.04, 3.17, 3.84 [01:02:57] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:03:03] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.00, 7.78, 7.24 [01:03:11] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.11, 3.73, 3.83 [01:04:52] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.312 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [01:04:55] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 0.85, 2.06, 3.24 [01:04:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.37, 7.35, 7.13 [01:05:10] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.57, 3.63, 3.78 [01:07:09] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.02, 3.88, 3.86 [01:09:13] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.67, 3.41, 3.69 [01:14:37] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 3.79, 5.73, 7.74 [01:15:10] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.75, 2.81, 3.32 [01:21:45] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.37, 7.19, 6.82 [01:22:37] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 4.47, 5.28, 6.73 [01:23:28] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [01:23:40] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.47, 6.67, 6.68 [01:30:03] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fediverse.kr' expires in 9 day(s) (Mon 04 Dec 2023 20:50:46 GMT +0000). [01:31:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.18, 7.28, 6.88 [01:33:16] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.97, 7.67, 7.08 [01:33:16] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.18, 3.70, 3.10 [01:33:28] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 6.76, 6.94, 6.82 [01:35:15] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.87, 3.60, 3.13 [01:35:26] PROBLEM - db131 Current Load on db131 is CRITICAL: CRITICAL - load average: 8.17, 7.37, 6.99 [01:37:14] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.93, 3.00, 2.98 [01:37:20] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 2.49, 5.59, 6.39 [01:37:46] PROBLEM - mw134 MediaWiki Rendering on mw134 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.018 second response time [01:38:19] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.022 second response time [01:38:19] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [01:38:23] PROBLEM - mw143 MediaWiki Rendering on mw143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:38:30] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [01:38:44] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:38:56] PROBLEM - mw133 MediaWiki Rendering on mw133 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.020 second response time [01:39:00] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.58, 6.05, 6.57 [01:39:10] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [01:39:11] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.028 second response time [01:39:16] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [01:39:21] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 5903 bytes in 0.016 second response time [01:40:18] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.96, 6.18, 6.72 [01:48:19] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [01:48:30] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [01:49:01] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [01:49:16] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [01:49:17] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - 981 bytes in 0.376 second response time [01:49:18] PROBLEM - db131 MariaDB Connections on db131 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db131.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections. [01:49:18] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db131.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [01:49:27] PROBLEM - db131 MariaDB on db131 is CRITICAL: Can't connect to MySQL server on 'db131.miraheze.org' (115) [01:50:24] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ec825d999c22...10143ee09a3d [01:50:25] [02miraheze/puppet] 07paladox 0310143ee - Lower parsercache to 5 days [01:50:43] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/88f94aebaa56...b3a88f7bfb06 [01:50:45] [02miraheze/mw-config] 07paladox 03b3a88f7 - Lower parsercache to 5 days [01:51:03] RECOVERY - mw133 MediaWiki Rendering on mw133 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.759 second response time [01:51:06] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 2.173 second response time [01:51:08] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.724 second response time [01:51:17] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.648 second response time [01:51:18] RECOVERY - db131 MariaDB Connections on db131 is OK: OK connection usage: 5.4%Current connections: 27 [01:51:27] RECOVERY - db131 MariaDB on db131 is OK: Uptime: 77 Threads: 27 Questions: 11669 Slow queries: 0 Opens: 572 Open tables: 566 Queries per second avg: 151.545 [01:51:41] miraheze/mw-config - paladox the build passed. [01:51:42] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [01:51:43] RECOVERY - mw134 MediaWiki Rendering on mw134 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.745 second response time [01:51:56] RECOVERY - mw143 MediaWiki Rendering on mw143 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.595 second response time [01:52:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:52:24] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.226 second response time [01:52:40] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 58s [01:52:43] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.533 second response time [01:52:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:52:52] !log [paladox@mwtask141] starting deploy of {'config': True} to all [01:52:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:52:59] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 6s [01:53:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:55:52] PROBLEM - db131 Disk Space on db131 is WARNING: DISK WARNING - free space: / 25796 MB (10% inode=96%); [01:56:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.46, 6.71, 6.23 [01:56:38] !log [@test131] starting deploy of {'config': True} to all [01:56:39] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [01:56:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:56:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:58:10] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.65, 6.59, 6.24 [02:19:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.49, 6.88, 6.65 [02:21:16] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.36, 6.55, 6.55 [02:24:37] PROBLEM - db131 Current Load on db131 is CRITICAL: CRITICAL - load average: 8.17, 6.63, 5.47 [02:25:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.37, 7.16, 6.81 [02:26:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.08, 7.02, 6.60 [02:26:37] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 6.02, 6.34, 5.50 [02:28:19] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.10, 7.44, 6.78 [02:28:57] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.16, 6.80, 6.76 [02:30:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.57, 7.06, 6.72 [02:32:48] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.34, 7.07, 6.87 [02:34:43] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.57, 6.12, 6.55 [02:36:19] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.46, 6.56, 6.73 [02:40:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.69, 6.84, 6.69 [02:44:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.55, 7.40, 6.93 [02:46:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.20, 7.32, 6.95 [02:48:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.86, 6.67, 6.75 [02:50:45] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.72, 7.68, 6.92 [02:52:40] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.77, 7.47, 6.93 [02:55:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [02:56:28] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.78, 6.79, 6.78 [03:00:03] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.fediverse.kr and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [03:00:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.53, 7.43, 7.01 [03:01:21] RECOVERY - mon141 Backups Grafana on mon141 is OK: FILE_AGE OK: /var/log/grafana-backup.log is 59 seconds old and 93 bytes [03:02:37] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.28, 7.22, 6.97 [03:04:32] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.11, 6.49, 6.74 [03:08:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.08, 8.10, 7.19 [03:10:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 7.44, 8.09, 7.42 [03:12:13] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.85, 7.80, 7.40 [03:12:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.81, 7.73, 7.22 [03:20:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.18, 7.83, 7.38 [03:22:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.69, 7.97, 7.51 [03:24:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.23, 7.37, 7.35 [03:26:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.97, 7.33, 7.33 [03:36:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.80, 7.79, 7.47 [03:36:18] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.97, 6.24, 6.76 [03:38:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.42, 7.22, 7.30 [03:42:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.28, 7.75, 7.48 [03:42:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.49, 6.73, 6.83 [03:44:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.00, 7.37, 7.36 [03:46:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 7.52, 7.41 [03:48:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.20, 6.90, 7.19 [03:48:18] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.39, 6.52, 6.75 [03:52:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.07, 6.76, 6.82 [03:52:20] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [03:52:45] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [03:54:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.77, 7.62, 7.35 [03:54:19] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.65, 6.40, 6.68 [03:58:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.76, 7.73, 7.46 [04:00:03] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fediverse.kr' expires in 9 day(s) (Mon 04 Dec 2023 20:50:46 GMT +0000). [04:00:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.03, 6.74, 6.74 [04:02:18] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.33, 6.54, 6.66 [04:09:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-13 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/1b14b34bfc82 [04:09:08] [02miraheze/puppet] 07paladox 031b14b34 - mariadb: Increase binlog_cache_size to 10M [04:09:09] [02puppet] 07paladox created branch 03paladox-patch-13 - 13https://github.com/miraheze/puppet [04:09:12] [02puppet] 07paladox opened pull request 03#3520: mariadb: Increase binlog_cache_size to 10M - 13https://github.com/miraheze/puppet/pull/3520 [04:15:06] !log db131: set global binlog_cache_size = 10485760; [04:15:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:17:53] !log db142: set global binlog_cache_size = 10485760; [04:17:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:19:25] !log db101: set global binlog_cache_size = 10485760; [04:19:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:20:34] [02puppet] 07paladox closed pull request 03#3520: mariadb: Increase binlog_cache_size to 10M - 13https://github.com/miraheze/puppet/pull/3520 [04:20:36] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-13 [04:20:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/10143ee09a3d...57ae33fee450 [04:20:40] [02miraheze/puppet] 07paladox 0357ae33f - mariadb: Increase binlog_cache_size to 10M (#3520) [04:20:41] [02puppet] 07paladox deleted branch 03paladox-patch-13 - 13https://github.com/miraheze/puppet [04:22:46] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [04:24:03] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [04:36:07] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.78, 6.44, 6.79 [04:36:36] !log db112: set global binlog_cache_size = 10485760; [04:36:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:36:43] !log db121: set global binlog_cache_size = 10485760; [04:36:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:40:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.64, 6.87, 6.92 [04:40:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.42, 7.19, 6.65 [04:42:18] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.09, 6.70, 6.53 [04:50:07] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.76, 6.42, 6.72 [04:54:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.24, 6.82, 6.82 [05:02:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.65, 7.43, 7.06 [05:04:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.93, 6.91, 6.92 [05:04:34] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/c662b9959d27 [05:04:36] [02miraheze/mw-config] 07paladox 03c662b99 - Use db121 for mysql-multiwrite [05:04:38] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [05:04:40] [02mw-config] 07paladox opened pull request 03#5379: Use db121 for mysql-multiwrite - 13https://github.com/miraheze/mw-config/pull/5379 [05:05:08] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.09, 7.11, 6.77 [05:05:31] miraheze/mw-config - paladox the build passed. [05:07:02] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.23, 6.47, 6.57 [05:09:01] [02mw-config] 07paladox closed pull request 03#5379: Use db121 for mysql-multiwrite - 13https://github.com/miraheze/mw-config/pull/5379 [05:09:03] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [05:09:04] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b3a88f7bfb06...a6bddf5df4bb [05:09:07] [02miraheze/mw-config] 07paladox 03a6bddf5 - Use db121 for mysql-multiwrite (#5379) [05:09:09] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [05:09:18] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [05:09:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:09:31] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a6bddf5df4bb...8a7f896f1bd0 [05:09:32] [02miraheze/mw-config] 07paladox 038a7f896 - Revert "Lower parsercache to 5 days" [05:09:38] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 20s [05:09:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:09:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/57ae33fee450...3fde070c48ea [05:09:44] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [05:09:46] [02miraheze/puppet] 07paladox 033fde070 - Revert "Lower parsercache to 5 days" [05:09:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:09:56] miraheze/mw-config - paladox the build passed. [05:10:07] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.59, 6.62, 6.79 [05:10:22] miraheze/mw-config - paladox the build passed. [05:10:23] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 39s [05:10:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:10:45] !log [paladox@mwtask141] starting deploy of {'config': True} to all [05:10:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:10:53] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [05:10:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:12:23] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [05:14:16] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.46, 7.12, 6.96 [05:15:02] RECOVERY - db131 Disk Space on db131 is OK: DISK OK - free space: / 58946 MB (24% inode=96%); [05:16:39] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.23, 7.29, 6.80 [05:18:36] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.14, 7.23, 6.84 [05:23:44] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.70, 6.52, 6.74 [05:26:00] !log [@test131] starting deploy of {'config': True} to all [05:26:01] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [05:26:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:26:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:26:25] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.23, 7.37, 7.00 [05:28:22] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.18, 6.52, 6.73 [05:40:32] !log db121: set global innodb_io_capacity = 5000; [05:40:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:40:37] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [05:51:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3fde070c48ea...9508246d75d9 [05:51:59] [02miraheze/puppet] 07paladox 039508246 - Update runner.pp [05:53:32] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9508246d75d9...847174270746 [05:53:34] [02miraheze/puppet] 07paladox 038471742 - runner: use run.php for some scripts [05:54:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/847174270746...44562468a039 [05:54:22] [02miraheze/puppet] 07paladox 034456246 - runner: Temporarily remove update_sfs_database cron [05:55:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.74, 6.94, 6.74 [05:59:04] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.19, 6.69, 6.69 [06:12:57] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.35, 9.67, 8.64 [06:14:57] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 8.86, 9.25, 8.60 [06:26:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.10, 7.08, 6.72 [06:30:33] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.86, 6.75, 6.69 [06:42:17] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.04, 7.38, 6.73 [06:42:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.65, 6.69, 6.53 [06:44:14] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.93, 6.84, 6.61 [06:44:24] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.19, 6.57, 6.51 [06:46:11] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.19, 6.74, 6.61 [06:53:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.79, 6.83, 6.67 [06:56:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.72, 7.69, 7.03 [07:00:01] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [07:00:45] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.94, 6.75, 6.50 [07:02:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.92, 7.68, 7.22 [07:06:37] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.14, 6.56, 6.52 [07:22:00] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [07:22:24] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.96, 6.44, 6.76 [07:33:53] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.89, 6.75, 6.41 [07:37:47] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.44, 6.52, 6.42 [07:41:44] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.12, 6.88, 6.74 [07:43:39] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.54, 6.79, 6.72 [07:50:00] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [07:51:48] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [07:54:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.83, 7.20, 6.86 [08:04:24] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.43, 6.34, 6.70 [08:07:53] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.09, 7.44, 6.86 [08:09:51] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.79, 6.71, 6.66 [08:13:42] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.50, 6.91, 6.78 [08:14:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.22, 7.58, 7.04 [08:15:39] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.40, 7.26, 6.91 [08:16:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.60, 7.13, 6.94 [08:17:39] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.13, 6.77, 6.77 [08:20:24] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.91, 6.53, 6.74 [08:21:51] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [08:30:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.82, 6.54, 6.56 [08:32:24] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.12, 6.15, 6.42 [08:52:43] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.fediverse.kr and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:53:28] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [08:56:58] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.05, 7.35, 6.81 [09:02:41] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.68, 6.72, 6.67 [09:12:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.61, 7.02, 6.81 [09:14:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.26, 7.45, 6.99 [09:16:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.70, 6.80, 6.81 [09:18:24] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.48, 6.41, 6.67 [09:21:40] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fediverse.kr' expires in 9 day(s) (Mon 04 Dec 2023 20:50:46 GMT +0000). [09:45:26] PROBLEM - elephants.ggreig.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'elephants.ggreig.com' expires in 15 day(s) (Mon 11 Dec 2023 09:29:43 GMT +0000). [09:45:48] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9bcc23172386...7b8a225809f2 [09:45:51] [02miraheze/ssl] 07MirahezeSSLBot 037b8a225 - Bot: Update SSL cert for elephants.ggreig.com [09:57:58] PROBLEM - studio.niuboss123.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'studio.niuboss123.com' expires in 15 day(s) (Mon 11 Dec 2023 09:43:36 GMT +0000). [10:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [10:14:09] PROBLEM - dodgecircle.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dodgecircle.com' expires in 15 day(s) (Mon 11 Dec 2023 09:46:04 GMT +0000). [10:14:23] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7b8a225809f2...3ddaf1aa45f0 [10:14:24] [02miraheze/ssl] 07MirahezeSSLBot 033ddaf1a - Bot: Update SSL cert for dodgecircle.com [10:15:21] RECOVERY - elephants.ggreig.com - LetsEncrypt on sslhost is OK: OK - Certificate 'elephants.ggreig.com' will expire on Fri 23 Feb 2024 08:45:39 GMT +0000. [10:19:52] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 13.51, 11.45, 7.17 [10:21:52] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 5.92, 9.15, 6.84 [10:42:49] RECOVERY - dodgecircle.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dodgecircle.com' will expire on Fri 23 Feb 2024 09:14:15 GMT +0000. [11:09:52] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 22.64, 14.48, 8.39 [11:10:01] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 5.32, 2.74, 1.44 [11:12:00] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 1.46, 2.36, 1.47 [11:13:52] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 8.38, 11.21, 8.48 [11:15:52] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 4.79, 8.88, 7.95 [11:27:44] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query franchise.franchising.org.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:51:45] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [11:57:41] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp25.miraheze.org - CNAME OK [12:06:49] PROBLEM - index.pdcommunity.ir - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.411 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [12:23:23] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [12:35:23] RECOVERY - index.pdcommunity.ir - reverse DNS on sslhost is OK: SSL OK - index.pdcommunity.ir reverse DNS resolves to cp25.miraheze.org - CNAME OK [13:19:40] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [13:49:29] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [13:50:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.95, 6.68, 6.21 [13:52:24] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.00, 6.37, 6.15 [14:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [14:50:19] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.68, 6.72, 6.04 [14:52:16] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.11, 6.53, 6.06 [15:03:42] PROBLEM - swiftobject111 APT on swiftobject111 is WARNING: APT WARNING: 0 packages available for upgrade (0 critical updates). warnings detected, errors detected. [15:05:43] PROBLEM - swiftobject111 APT on swiftobject111 is CRITICAL: APT CRITICAL: 84 packages available for upgrade (30 critical updates). [15:41:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.54, 6.81, 6.42 [15:45:17] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.13, 6.31, 6.29 [16:00:27] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [16:07:03] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.03, 3.31, 2.24 [16:09:03] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.68, 3.20, 2.34 [16:13:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.94, 7.55, 6.73 [16:16:57] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.04, 6.79, 6.59 [16:30:24] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp25.miraheze.org - CNAME OK [16:49:55] PROBLEM - matomo121 Current Load on matomo121 is CRITICAL: LOAD CRITICAL - total load average: 4.10, 3.86, 3.39 [16:51:52] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 2.98, 3.54, 3.32 [16:53:49] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 3.13, 3.34, 3.27 [17:06:23] RECOVERY - wiki.anglish.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.anglish.info' will expire on Mon 18 Dec 2023 21:25:31 GMT +0000. [17:07:26] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.12, 3.43, 3.22 [17:09:23] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 3.34, 3.40, 3.22 [17:15:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.35, 6.72, 6.26 [17:16:59] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.82, 6.42, 6.21 [17:20:27] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [17:27:03] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.43, 3.40, 3.21 [17:29:03] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.20, 3.01, 3.09 [17:38:23] PROBLEM - wiki.anglish.info - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.anglish.info and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [18:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [18:23:39] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.03, 7.16, 6.57 [18:29:39] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.69, 6.47, 6.49 [18:36:27] RECOVERY - wiki.anglish.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.anglish.info' will expire on Mon 18 Dec 2023 21:25:31 GMT +0000. [18:51:28] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:08:19] PROBLEM - wiki.anglish.info - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.anglish.info and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:15:11] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.89, 6.97, 6.34 [19:17:06] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.91, 6.80, 6.38 [19:17:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [19:17:39] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.09, 7.07, 6.36 [19:19:39] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.64, 6.43, 6.21 [19:21:11] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [19:46:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.42, 6.72, 6.33 [19:47:10] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [19:48:35] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.65, 6.01, 6.12 [19:52:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.17, 6.90, 6.46 [19:52:48] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [19:56:25] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.50, 6.39, 6.36 [20:04:39] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.84, 6.99, 6.54 [20:08:33] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.45, 6.63, 6.54 [20:16:58] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [20:22:30] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [20:54:13] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [20:56:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.92, 6.78, 6.26 [20:58:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.81, 8.03, 6.74 [21:02:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.20, 6.81, 6.55 [21:03:01] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.60, 7.00, 6.47 [21:04:25] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.32, 6.78, 6.58 [21:08:53] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.22, 6.58, 6.46 [21:14:09] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importImages.php --wiki=rlabwiki /home/macfan/images --search-recursively (START) [21:14:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:18:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.78, 7.22, 6.78 [21:20:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.45, 7.52, 6.94 [21:20:32] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.55, 7.17, 6.62 [21:22:24] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.59, 7.16, 6.88 [21:28:21] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.77, 7.59, 6.98 [21:28:24]