[00:00:47] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.20, 6.55, 5.81 [00:02:41] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.17, 5.97, 5.69 [00:03:58] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 14.27, 8.00, 3.73 [00:06:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.23, 10.97, 10.77 [00:12:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [00:12:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.39, 11.69, 11.19 [00:13:57] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 2.83, 7.23, 6.33 [00:15:57] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.40, 5.72, 5.88 [00:16:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.55, 7.21, 6.70 [00:19:01] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.10, 6.94, 6.65 [00:19:44] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [00:20:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.04, 11.79, 11.43 [00:22:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.85, 11.78, 11.47 [00:24:52] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.93, 6.65, 6.66 [00:37:55] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.21, 6.41, 6.06 [00:39:50] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.26, 6.27, 6.04 [00:42:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [00:45:37] PROBLEM - sdiy.info - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.wikitide.net.', 'ns2.wikitide.net.'], 'CNAME': None} [00:47:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.35, 6.92, 6.60 [00:49:12] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.99, 6.46, 6.46 [00:51:14] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:52:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.25, 11.24, 10.97 [01:00:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.31, 11.75, 11.44 [01:04:41] PROBLEM - jobchron121 JobChron Service on jobchron121 is CRITICAL: PROCS CRITICAL: 2 processes with args 'redisJobChronService' [01:08:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 13.64, 11.91, 11.48 [01:12:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [01:14:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 6.28, 10.70, 11.30 [01:15:38] RECOVERY - sdiy.info - reverse DNS on sslhost is OK: SSL OK - sdiy.info reverse DNS resolves to cp25.miraheze.org - NS RECORDS OK [01:16:03] [Grafana] !sre FIRING: There has been a rise in the MediaWiki exception rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [01:16:08] PROBLEM - db131 Current Load on db131 is CRITICAL: CRITICAL - load average: 57.27, 25.64, 10.85 [01:16:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.92, 7.56, 10.07 [01:16:41] PROBLEM - mw134 MediaWiki Rendering on mw134 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:16:41] RECOVERY - jobchron121 JobChron Service on jobchron121 is OK: PROCS OK: 1 process with args 'redisJobChronService' [01:17:38] PROBLEM - mw131 HTTPS on mw131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [01:17:46] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [01:17:48] PROBLEM - mw133 MediaWiki Rendering on mw133 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:18:19] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [01:18:21] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:18:24] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5020071566 secs [01:18:36] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [01:18:44] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 8 backends are down. mw131 mw132 mw141 mw142 mw133 mw134 mw143 mediawiki [01:18:59] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [01:19:23] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.513 second response time [01:19:33] RECOVERY - mw131 HTTPS on mw131 is OK: HTTP OK: HTTP/2 301 - 345 bytes in 0.056 second response time [01:19:43] RECOVERY - mw133 MediaWiki Rendering on mw133 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.537 second response time [01:19:45] PROBLEM - test131 Current Load on test131 is CRITICAL: LOAD CRITICAL - total load average: 4.17, 2.91, 1.33 [01:19:51] PROBLEM - prometheus131 Current Load on prometheus131 is CRITICAL: LOAD CRITICAL - total load average: 5.33, 4.55, 2.07 [01:20:16] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [01:20:17] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.350 second response time [01:20:30] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [01:20:38] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [01:20:39] RECOVERY - mw134 MediaWiki Rendering on mw134 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.331 second response time [01:21:36] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [01:21:41] RECOVERY - test131 Current Load on test131 is OK: LOAD OK - total load average: 0.82, 2.10, 1.22 [01:21:50] RECOVERY - prometheus131 Current Load on prometheus131 is OK: LOAD OK - total load average: 0.96, 3.18, 1.87 [01:23:19] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 0.52, 3.81, 2.80 [01:23:35] [Grafana] !sre RESOLVED: MediaWiki Exception Rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [01:25:14] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 0.35, 2.69, 2.50 [01:26:55] RECOVERY - mw132 Disk Space on mw132 is OK: DISK OK - free space: / 8871 MB (37% inode=79%); [01:28:07] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.68, 9.14, 9.09 [01:28:48] RECOVERY - www.sdiy.info - reverse DNS on sslhost is OK: SSL OK - www.sdiy.info reverse DNS resolves to cp24.miraheze.org - NS RECORDS OK [01:30:03] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 10.02, 9.34, 9.16 [01:33:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.55, 10.38, 9.62 [01:37:51] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 8.42, 9.92, 9.64 [01:41:32] PROBLEM - mw131 Disk Space on mw131 is WARNING: DISK WARNING - free space: / 1758 MB (7% inode=79%); [01:42:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [01:48:51] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.31, 7.08, 6.48 [01:50:45] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.49, 6.55, 6.36 [01:52:41] PROBLEM - graylog131 PowerDNS Recursor on graylog131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [01:54:40] RECOVERY - graylog131 PowerDNS Recursor on graylog131 is OK: DNS OK: 0.203 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [01:58:50] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 3.34, 5.88, 7.84 [01:59:24] PROBLEM - www.sdiy.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for www.sdiy.info could not be found [02:00:47] PROBLEM - db131 Current Load on db131 is CRITICAL: CRITICAL - load average: 8.07, 6.76, 7.93 [02:03:35] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 17.25, 10.59, 6.97 [02:05:31] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 7.94, 9.51, 7.02 [02:08:45] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 4.47, 6.11, 7.62 [02:08:56] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.66, 9.91, 8.93 [02:12:45] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 3.47, 4.78, 6.74 [02:15:29] [02miraheze/mw-config] 07paladox pushed 031 commit to 03revert-5296-paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/5c73dbd5a3d8 [02:15:30] [02miraheze/mw-config] 07paladox 035c73dbd - Revert "Unset wgUseLocalMessageCache (#5296)" [02:15:32] [02mw-config] 07paladox created branch 03revert-5296-paladox-patch-1 - 13https://github.com/miraheze/mw-config [02:15:35] [02mw-config] 07paladox opened pull request 03#5373: Revert "Unset wgUseLocalMessageCache (#5296)" - 13https://github.com/miraheze/mw-config/pull/5373 [02:16:24] [02mw-config] 07paladox closed pull request 03#5373: Revert "Unset wgUseLocalMessageCache (#5296)" - 13https://github.com/miraheze/mw-config/pull/5373 [02:16:27] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/63481546523a...938eb0faf945 [02:16:30] [02miraheze/mw-config] 07paladox 03938eb0f - Revert "Unset wgUseLocalMessageCache (#5296)" (#5373) [02:16:31] [02miraheze/mw-config] 07paladox deleted branch 03revert-5296-paladox-patch-1 [02:16:32] miraheze/mw-config - paladox the build passed. [02:16:33] [02mw-config] 07paladox deleted branch 03revert-5296-paladox-patch-1 - 13https://github.com/miraheze/mw-config [02:16:50] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [02:16:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:16:59] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [02:17:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:17:16] PROBLEM - sdiy.info - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - sdiy.info reverse DNS resolves to vps-c10c02c8.vps.ovh.net [02:17:20] miraheze/mw-config - paladox the build passed. [02:18:18] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/938eb0faf945...86cd5141d53d [02:18:21] [02miraheze/mw-config] 07paladox 0386cd514 - Set wgResourceLoaderUseObjectCacheForDeps to true [02:18:22] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [02:18:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:18:38] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 8.91, 10.13, 9.58 [02:18:48] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 25s [02:18:57] !log [paladox@mwtask141] starting deploy of {'config': True} to all [02:19:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:19:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:19:12] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 14s [02:19:13] miraheze/mw-config - paladox the build passed. [02:20:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:20:52] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [02:20:55] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 2s [02:20:58] !log [paladox@test131] starting deploy of {'config': True} to all [02:20:59] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [02:21:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:21:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:21:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:21:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:22:45] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 6.74, 7.44, 6.94 [02:24:11] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [02:24:45] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 5.00, 6.43, 6.62 [02:25:02] PROBLEM - mw131 Disk Space on mw131 is CRITICAL: DISK CRITICAL - free space: / 1060 MB (4% inode=79%); [02:26:06] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/86cd5141d53d...540d23a64db2 [02:26:08] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [02:26:09] [02miraheze/mw-config] 07paladox 03540d23a - Set wgMaxExecutionTimeForExpensiveQueries to 30000 [02:26:13] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [02:26:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:26:14] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [02:26:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:26:19] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [02:26:21] !log [paladox@test131] starting deploy of {'config': True} to all [02:26:22] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [02:26:23] !log [paladox@mwtask141] starting deploy of {'config': True} to all [02:26:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:26:31] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [02:26:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:26:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:26:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:26:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:26:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:27:02] PROBLEM - mw131 Disk Space on mw131 is WARNING: DISK WARNING - free space: / 1745 MB (7% inode=79%); [02:27:02] miraheze/mw-config - paladox the build passed. [02:28:11] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [02:31:19] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 6 backends are down. mw131 mw132 mw141 mw142 mw133 mw143 [02:31:24] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 6 backends are down. mw131 mw132 mw141 mw142 mw133 mw143 [02:31:27] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/bd04d5b41f2e [02:31:30] [02miraheze/mw-config] 07paladox 03bd04d5b - Set wgCdnMaxAge to 5 days [02:31:32] PROBLEM - mw141 nutcracker process on mw141 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [02:31:33] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [02:31:35] PROBLEM - mw141 Disk Space on mw141 is CRITICAL: DISK CRITICAL - free space: / 657 MB (2% inode=80%); [02:31:36] [02mw-config] 07paladox opened pull request 03#5374: Set wgCdnMaxAge to 5 days - 13https://github.com/miraheze/mw-config/pull/5374 [02:31:37] [02mw-config] 07paladox closed pull request 03#5374: Set wgCdnMaxAge to 5 days - 13https://github.com/miraheze/mw-config/pull/5374 [02:31:38] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [02:31:38] PROBLEM - mw141 php-fpm on mw141 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [02:31:38] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/540d23a64db2...3a7f3f9af15e [02:31:40] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [02:31:40] [02miraheze/mw-config] 07paladox 033a7f3f9 - Set wgCdnMaxAge to 5 days (#5374) [02:31:41] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [02:31:41] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [02:31:43] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [02:32:02] PROBLEM - mw141 JobRunner Service on mw141 is CRITICAL: CRITICAL - Plugin timed out after 10 seconds [02:32:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:32:26] miraheze/mw-config - paladox the build passed. [02:32:28] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:32:28] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.02, 3.49, 3.91 [02:32:40] miraheze/mw-config - paladox the build passed. [02:32:42] PROBLEM - mw141 SSH on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:32:48] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 4 backends are down. mw141 mw142 mw133 mw143 [02:33:03] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 4 backends are down. mw141 mw142 mw133 mw143 [02:33:12] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:33:50] PROBLEM - bast141 Current Load on bast141 is CRITICAL: LOAD CRITICAL - total load average: 3.03, 1.88, 0.82 [02:33:57] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 8.86, 3.87, 1.94 [02:34:23] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.290 second response time [02:34:35] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:34:36] PROBLEM - mw141 Puppet on mw141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:34:40] PROBLEM - mw141 conntrack_table_size on mw141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:34:59] PROBLEM - ldap141 Current Load on ldap141 is CRITICAL: LOAD CRITICAL - total load average: 6.13, 2.87, 1.17 [02:35:20] PROBLEM - mw141 ferm_active on mw141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:35:27] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 9.84, 5.75, 2.51 [02:36:21] PROBLEM - mon141 icinga.miraheze.org HTTPS on mon141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:36:26] PROBLEM - mw143 Current Load on mw143 is CRITICAL: CRITICAL - load average: 20.67, 10.89, 5.85 [02:36:33] PROBLEM - mon141 grafana.miraheze.org HTTPS on mon141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:36:33] PROBLEM - mon141 HTTPS on mon141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10003 milliseconds [02:36:41] PROBLEM - db142 MariaDB on db142 is CRITICAL: Too many connections [02:37:03] PROBLEM - cloud14 Current Load on cloud14 is CRITICAL: CRITICAL - load average: 157.77, 101.27, 53.53 [02:37:25] PROBLEM - mw142 Current Load on mw142 is CRITICAL: CRITICAL - load average: 18.09, 11.14, 6.79 [02:37:54] RECOVERY - mw141 nutcracker process on mw141 is OK: PROCS OK: 1 process with UID = 115 (nutcracker), command name 'nutcracker' [02:38:05] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.68, 6.59, 3.29 [02:38:17] RECOVERY - mw141 JobRunner Service on mw141 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [02:38:39] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 13.44, 10.30, 9.17 [02:39:02] PROBLEM - db142 MariaDB Connections on db142 is UNKNOWN: [02:39:40] PROBLEM - os141 SSH on os141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:39:48] RECOVERY - mw141 php-fpm on mw141 is OK: PROCS OK: 22 processes with command name 'php-fpm7.4' [02:39:48] PROBLEM - mw143 MediaWiki Rendering on mw143 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:41:24] PROBLEM - os141 PowerDNS Recursor on os141 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:41:31] PROBLEM - os141 conntrack_table_size on os141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [02:41:57] PROBLEM - db142 PowerDNS Recursor on db142 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:42:05] !log [paladox@mwtask141] DEPLOY ABORTED: Canary check failed for publictestwiki.com@localhost [02:42:27] RECOVERY - mon141 icinga.miraheze.org HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 302 Found - 297 bytes in 0.018 second response time [02:42:28] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.73, 2.40, 3.18 [02:42:31] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 37 minutes ago with 0 failures [02:42:37] RECOVERY - db142 MariaDB on db142 is OK: Uptime: 742537 Threads: 9 Questions: 120478166 Slow queries: 8035608 Opens: 146080 Open tables: 145565 Queries per second avg: 162.252 [02:42:39] RECOVERY - mon141 HTTPS on mon141 is OK: HTTP OK: HTTP/2 200 - 336 bytes in 0.030 second response time [02:42:40] RECOVERY - mon141 grafana.miraheze.org HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 43410 bytes in 0.723 second response time [02:42:42] RECOVERY - mw141 conntrack_table_size on mw141 is OK: OK: nf_conntrack is 0 % full [02:42:52] RECOVERY - mw141 SSH on mw141 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u2 (protocol 2.0) [02:43:10] RECOVERY - mw141 ferm_active on mw141 is OK: OK ferm input default policy is set [02:43:24] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [02:43:29] RECOVERY - os141 conntrack_table_size on os141 is OK: OK: nf_conntrack is 0 % full [02:43:33] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.895 second response time [02:43:48] RECOVERY - os141 SSH on os141 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u1 (protocol 2.0) [02:43:52] RECOVERY - db142 MariaDB Connections on db142 is OK: OK connection usage: 0.2%Current connections: 1 [02:43:55] RECOVERY - mw143 MediaWiki Rendering on mw143 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.280 second response time [02:43:56] RECOVERY - db142 PowerDNS Recursor on db142 is OK: DNS OK: 0.240 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [02:44:17] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 2.93, 7.48, 5.14 [02:44:38] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.83, 11.58, 10.25 [02:44:43] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [02:44:48] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [02:45:03] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [02:45:25] PROBLEM - mw142 Current Load on mw142 is WARNING: WARNING - load average: 5.22, 11.57, 9.77 [02:45:33] RECOVERY - os141 PowerDNS Recursor on os141 is OK: DNS OK: 0.170 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [02:45:42] RECOVERY - bast141 Current Load on bast141 is OK: LOAD OK - total load average: 0.26, 1.59, 1.49 [02:46:17] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 0.65, 5.09, 4.54 [02:46:26] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 2.44, 10.14, 9.47 [02:46:55] PROBLEM - cloud14 Current Load on cloud14 is WARNING: WARNING - load average: 17.73, 70.44, 71.61 [02:47:25] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 3.66, 8.69, 8.92 [02:47:26] PROBLEM - mw141 Current Load on mw141 is WARNING: WARNING - load average: 4.18, 8.76, 11.27 [02:48:54] RECOVERY - cloud14 Current Load on cloud14 is OK: OK - load average: 16.05, 52.96, 65.10 [02:49:29] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [02:49:30] RECOVERY - mw141 Disk Space on mw141 is OK: DISK OK - free space: / 9412 MB (39% inode=80%); [02:49:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:49:36] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [02:49:38] !log [paladox@mwtask141] starting deploy of {'config': True} to all [02:49:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:49:43] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 5s [02:49:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:49:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:51:23] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 2.64, 5.43, 9.33 [02:52:59] PROBLEM - ldap141 Current Load on ldap141 is WARNING: LOAD WARNING - total load average: 0.08, 0.83, 1.80 [02:53:57] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 0.65, 2.00, 3.92 [02:54:57] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 1.62, 1.87, 3.75 [02:54:59] RECOVERY - ldap141 Current Load on ldap141 is OK: LOAD OK - total load average: 0.83, 0.82, 1.67 [02:55:48] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3a7f3f9af15e...eb8f0bc1a2e7 [02:55:50] [02miraheze/mw-config] 07paladox 03eb8f0bc - Set wgSearchSuggestCacheExpiry to 10800 [02:55:52] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [02:55:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:56:28] !log [@test131] starting deploy of {'config': True} to all [02:56:29] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [02:56:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:56:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:56:43] PROBLEM - cp35 Varnish Backends on cp35 is CRITICAL: 1 backends are down. mw141 [02:56:44] miraheze/mw-config - paladox the build passed. [02:56:48] PROBLEM - cp25 Varnish Backends on cp25 is CRITICAL: 1 backends are down. mw141 [02:56:54] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.88, 2.74, 3.83 [02:57:03] PROBLEM - cp24 Varnish Backends on cp24 is CRITICAL: 1 backends are down. mw141 [02:57:16] PROBLEM - mw141 Current Load on mw141 is CRITICAL: CRITICAL - load average: 14.30, 10.12, 9.93 [02:57:24] PROBLEM - cp34 Varnish Backends on cp34 is CRITICAL: 1 backends are down. mw141 [02:57:31] PROBLEM - mw141 HTTPS on mw141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10002 milliseconds [02:57:40] PROBLEM - mw141 MediaWiki Rendering on mw141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [02:58:10] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 18.91, 12.19, 7.15 [02:58:39] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.42, 11.21, 10.59 [02:58:59] PROBLEM - ldap141 Current Load on ldap141 is CRITICAL: LOAD CRITICAL - total load average: 2.14, 1.56, 1.78 [02:59:32] PROBLEM - bast141 Current Load on bast141 is CRITICAL: LOAD CRITICAL - total load average: 4.69, 2.21, 1.50 [02:59:57] PROBLEM - mon141 Current Load on mon141 is CRITICAL: LOAD CRITICAL - total load average: 4.29, 2.93, 3.63 [03:00:53] PROBLEM - cloud14 Current Load on cloud14 is CRITICAL: CRITICAL - load average: 123.25, 80.02, 65.02 [03:01:10] PROBLEM - mw141 Puppet on mw141 is CRITICAL: connect to address 2a10:6740::6:502 port 5666: No route to hostconnect to host 2a10:6740::6:502 port 5666: No route to host [03:01:10] PROBLEM - mw141 JobRunner Service on mw141 is CRITICAL: connect to address 2a10:6740::6:502 port 5666: No route to hostconnect to host 2a10:6740::6:502 port 5666: No route to host [03:01:15] PROBLEM - mw141 PowerDNS Recursor on mw141 is CRITICAL: connect to address 2a10:6740::6:502 port 5666: No route to hostconnect to host 2a10:6740::6:502 port 5666: No route to host [03:01:26] PROBLEM - mw143 Current Load on mw143 is CRITICAL: connect to address 2a10:6740::6:513 port 5666: No route to hostconnect to host 2a10:6740::6:513 port 5666: No route to host [03:01:32] PROBLEM - mw141 conntrack_table_size on mw141 is CRITICAL: connect to address 2a10:6740::6:502 port 5666: No route to hostconnect to host 2a10:6740::6:502 port 5666: No route to host [03:01:32] PROBLEM - mw141 Disk Space on mw141 is CRITICAL: connect to address 2a10:6740::6:502 port 5666: No route to hostconnect to host 2a10:6740::6:502 port 5666: No route to host [03:01:45] PROBLEM - mw141 ferm_active on mw141 is CRITICAL: connect to address 2a10:6740::6:502 port 5666: No route to hostconnect to host 2a10:6740::6:502 port 5666: No route to host [03:01:52] PROBLEM - mw141 NTP time on mw141 is CRITICAL: connect to address 2a10:6740::6:502 port 5666: No route to hostconnect to host 2a10:6740::6:502 port 5666: No route to host [03:01:59] PROBLEM - mw143 MediaWiki Rendering on mw143 is CRITICAL: connect to address 2a10:6740::6:513 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [03:02:02] PROBLEM - Host mw141 is DOWN: CRITICAL - Host Unreachable (2a10:6740::6:502) [03:02:13] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: connect to address 2a10:6740::6:503 and port 443: No route to hostHTTP CRITICAL - Unable to open TCP socket [03:02:21] PROBLEM - mw142 conntrack_table_size on mw142 is CRITICAL: connect to address 2a10:6740::6:503 port 5666: No route to hostconnect to host 2a10:6740::6:503 port 5666: No route to host [03:02:27] PROBLEM - mw142 Current Load on mw142 is CRITICAL: connect to address 2a10:6740::6:503 port 5666: No route to hostconnect to host 2a10:6740::6:503 port 5666: No route to host [03:02:28] PROBLEM - mw142 JobRunner Service on mw142 is CRITICAL: connect to address 2a10:6740::6:503 port 5666: No route to hostconnect to host 2a10:6740::6:503 port 5666: No route to host [03:02:28] PROBLEM - mon141 grafana.miraheze.org HTTPS on mon141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:02:28] PROBLEM - mon141 HTTPS on mon141 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Operation timed out after 10004 milliseconds with 0 bytes received [03:02:35] PROBLEM - mw143 ferm_active on mw143 is CRITICAL: connect to address 2a10:6740::6:513 port 5666: No route to hostconnect to host 2a10:6740::6:513 port 5666: No route to host [03:02:38] PROBLEM - mw142 nutcracker process on mw142 is CRITICAL: connect to address 2a10:6740::6:503 port 5666: No route to hostconnect to host 2a10:6740::6:503 port 5666: No route to host [03:02:41] PROBLEM - mw142 php-fpm on mw142 is CRITICAL: connect to address 2a10:6740::6:503 port 5666: No route to hostconnect to host 2a10:6740::6:503 port 5666: No route to host [03:02:41] PROBLEM - mw142 HTTPS on mw142 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw142.miraheze.org port 443 after 3066 ms: Couldn't connect to server [03:02:45] PROBLEM - mw143 HTTPS on mw143 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to mw143.miraheze.org port 443 after 3066 ms: Couldn't connect to server [03:02:56] PROBLEM - ping6 on mw142 is CRITICAL: CRITICAL - Host Unreachable (2a10:6740::6:503) [03:02:56] PROBLEM - mw143 NTP time on mw143 is CRITICAL: connect to address 2a10:6740::6:513 port 5666: No route to hostconnect to host 2a10:6740::6:513 port 5666: No route to host [03:03:04] PROBLEM - mw143 nutcracker process on mw143 is CRITICAL: connect to address 2a10:6740::6:513 port 5666: No route to hostconnect to host 2a10:6740::6:513 port 5666: No route to host [03:03:12] PROBLEM - mw143 SSH on mw143 is CRITICAL: connect to address 2a10:6740::6:513 and port 22: No route to host [03:03:13] PROBLEM - mw142 ferm_active on mw142 is CRITICAL: connect to address 2a10:6740::6:503 port 5666: No route to hostconnect to host 2a10:6740::6:503 port 5666: No route to host [03:03:13] PROBLEM - mw142 NTP time on mw142 is CRITICAL: connect to address 2a10:6740::6:503 port 5666: No route to hostconnect to host 2a10:6740::6:503 port 5666: No route to host [03:03:19] PROBLEM - mw142 PowerDNS Recursor on mw142 is CRITICAL: connect to address 2a10:6740::6:503 port 5666: No route to hostconnect to host 2a10:6740::6:503 port 5666: No route to host [03:03:19] PROBLEM - mw142 Puppet on mw142 is CRITICAL: connect to address 2a10:6740::6:503 port 5666: No route to hostconnect to host 2a10:6740::6:503 port 5666: No route to host [03:03:19] PROBLEM - mw142 Disk Space on mw142 is CRITICAL: connect to address 2a10:6740::6:503 port 5666: No route to hostconnect to host 2a10:6740::6:503 port 5666: No route to host [03:03:24] PROBLEM - Host mw142 is DOWN: CRITICAL - Host Unreachable (2a10:6740::6:503) [03:03:25] PROBLEM - mw143 Puppet on mw143 is CRITICAL: connect to address 2a10:6740::6:513 port 5666: No route to hostconnect to host 2a10:6740::6:513 port 5666: No route to host [03:03:29] PROBLEM - mw143 PowerDNS Recursor on mw143 is CRITICAL: connect to address 2a10:6740::6:513 port 5666: No route to hostconnect to host 2a10:6740::6:513 port 5666: No route to host [03:03:29] PROBLEM - mw143 conntrack_table_size on mw143 is CRITICAL: connect to address 2a10:6740::6:513 port 5666: No route to hostconnect to host 2a10:6740::6:513 port 5666: No route to host [03:03:36] PROBLEM - Host mw143 is DOWN: CRITICAL - Host Unreachable (2a10:6740::6:513) [03:06:43] PROBLEM - os141 Puppet on os141 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [03:06:46] PROBLEM - ping6 on mwtask141 is CRITICAL: CRITICAL - Host Unreachable (2a10:6740::6:504) [03:06:49] PROBLEM - mwtask141 SSH on mwtask141 is CRITICAL: connect to address 2a10:6740::6:504 and port 22: No route to host [03:06:49] PROBLEM - mwtask141 mathoid on mwtask141 is CRITICAL: connect to address 2a10:6740::6:504 and port 10044: No route to host [03:07:33] PROBLEM - Host mwtask141 is DOWN: CRITICAL - Host Unreachable (2a10:6740::6:504) [03:07:33] PROBLEM - mwtask141 NTP time on mwtask141 is CRITICAL: connect to address 2a10:6740::6:504 port 5666: No route to hostconnect to host 2a10:6740::6:504 port 5666: No route to host [03:07:34] RECOVERY - Host mw142 is UP: PING OK - Packet loss = 0%, RTA = 1.51 ms [03:07:34] RECOVERY - mw142 JobRunner Service on mw142 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [03:07:34] RECOVERY - mw142 Current Load on mw142 is OK: OK - load average: 0.78, 0.16, 0.05 [03:07:34] RECOVERY - mw142 php-fpm on mw142 is OK: PROCS OK: 25 processes with command name 'php-fpm7.4' [03:07:35] RECOVERY - mw142 nutcracker process on mw142 is OK: PROCS OK: 1 process with UID = 115 (nutcracker), command name 'nutcracker' [03:07:35] RECOVERY - Host mw143 is UP: PING OK - Packet loss = 0%, RTA = 2.10 ms [03:07:38] RECOVERY - mon141 grafana.miraheze.org HTTPS on mon141 is OK: HTTP OK: HTTP/1.1 200 OK - 43418 bytes in 0.066 second response time [03:07:39] RECOVERY - mon141 HTTPS on mon141 is OK: HTTP OK: HTTP/2 200 - 336 bytes in 0.024 second response time [03:07:43] RECOVERY - mw142 ferm_active on mw142 is OK: OK ferm input default policy is set [03:07:43] RECOVERY - mw143 Current Load on mw143 is OK: OK - load average: 1.68, 0.37, 0.12 [03:07:44] RECOVERY - ping6 on mw142 is OK: PING OK - Packet loss = 0%, RTA = 1.31 ms [03:07:46] RECOVERY - mw143 SSH on mw143 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u2 (protocol 2.0) [03:07:46] PROBLEM - mw142 NTP time on mw142 is WARNING: NTP WARNING: Offset 0.3951113224 secs [03:07:46] RECOVERY - mw143 conntrack_table_size on mw143 is OK: OK: nf_conntrack is 0 % full [03:07:47] RECOVERY - mw142 PowerDNS Recursor on mw142 is OK: DNS OK: 5.750 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [03:07:48] RECOVERY - mw142 Disk Space on mw142 is OK: DISK OK - free space: / 9365 MB (39% inode=79%); [03:07:49] RECOVERY - mw142 Puppet on mw142 is OK: OK: Puppet is currently enabled, last run 37 minutes ago with 0 failures [03:07:49] RECOVERY - mw143 Puppet on mw143 is OK: OK: Puppet is currently enabled, last run 34 minutes ago with 0 failures [03:07:49] RECOVERY - mw143 PowerDNS Recursor on mw143 is OK: DNS OK: 0.229 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [03:07:53] PROBLEM - db142 MariaDB Connections on db142 is UNKNOWN: PHP Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db142.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections. [03:07:53] n line 66Fatal error: Uncaught mysqli_sql_exception: Connection refused in /usr/lib/nagios/plugins/check_mysql_connections.php:66Stack trace:#0 /usr/lib/nagios/plugins/check_mysql_connections.php(66): mysqli_real_connect(Object(mysqli), 'db142.miraheze....', 'icinga', Object(SensitiveParameterValue), NULL, NULL, NULL, true)#1 {main} thrown in /usr/lib/nagios/plugins/check_mysql_connections.php on line 66 [03:08:08] RECOVERY - mw142 conntrack_table_size on mw142 is OK: OK: nf_conntrack is 0 % full [03:08:09] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.994 second response time [03:08:39] PROBLEM - os141 Puppet on os141 is UNKNOWN: NRPE: Unable to read output [03:08:53] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [03:09:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:09:24] RECOVERY - mw143 HTTPS on mw143 is OK: HTTP OK: HTTP/2 301 - 345 bytes in 0.012 second response time [03:09:29] PROBLEM - db142 MariaDB on db142 is CRITICAL: Can't connect to MySQL server on 'db142.miraheze.org' (115) [03:09:29] RECOVERY - Host mw141 is UP: PING OK - Packet loss = 0%, RTA = 1.13 ms [03:09:29] RECOVERY - mw142 HTTPS on mw142 is OK: HTTP OK: HTTP/2 301 - 345 bytes in 0.018 second response time [03:09:31] RECOVERY - mw143 ferm_active on mw143 is OK: OK ferm input default policy is set [03:09:31] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 38s [03:09:32] RECOVERY - mw143 nutcracker process on mw143 is OK: PROCS OK: 1 process with UID = 115 (nutcracker), command name 'nutcracker' [03:09:33] RECOVERY - Host mwtask141 is UP: PING OK - Packet loss = 0%, RTA = 0.25 ms [03:09:33] RECOVERY - mwtask141 NTP time on mwtask141 is OK: NTP OK: Offset -0.0006466805935 secs [03:09:33] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 9.91, 3.48, 1.25 [03:09:35] RECOVERY - mw141 PowerDNS Recursor on mw141 is OK: DNS OK: 0.104 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [03:09:36] RECOVERY - mw141 Puppet on mw141 is OK: OK: Puppet is currently enabled, last run 25 minutes ago with 0 failures [03:09:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:09:36] PROBLEM - cloud14 Current Load on cloud14 is WARNING: WARNING - load average: 17.95, 70.34, 75.36 [03:09:38] !log [paladox@mwtask141] starting deploy of {'config': True} to all [03:09:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:09:43] RECOVERY - mw141 HTTPS on mw141 is OK: HTTP OK: HTTP/2 301 - 345 bytes in 0.169 second response time [03:09:44] RECOVERY - mw141 JobRunner Service on mw141 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [03:09:45] RECOVERY - mw143 NTP time on mw143 is OK: NTP OK: Offset -0.0002294182777 secs [03:09:46] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [03:09:46] RECOVERY - mw142 NTP time on mw142 is OK: NTP OK: Offset -0.007281452417 secs [03:09:49] RECOVERY - mw141 ferm_active on mw141 is OK: OK ferm input default policy is set [03:09:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:09:51] RECOVERY - mw141 conntrack_table_size on mw141 is OK: OK: nf_conntrack is 0 % full [03:09:52] RECOVERY - mw141 Disk Space on mw141 is OK: DISK OK - free space: / 9443 MB (39% inode=80%); [03:09:53] PROBLEM - mwtask141 MediaWiki Rendering on mwtask141 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [03:09:53] RECOVERY - mw141 MediaWiki Rendering on mw141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.899 second response time [03:09:54] RECOVERY - mw141 NTP time on mw141 is OK: NTP OK: Offset -4.801154137e-05 secs [03:09:57] RECOVERY - mw143 MediaWiki Rendering on mw143 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.407 second response time [03:10:18] RECOVERY - mw141 Current Load on mw141 is OK: OK - load average: 5.69, 1.95, 0.71 [03:10:43] RECOVERY - mwtask141 SSH on mwtask141 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u2 (protocol 2.0) [03:10:48] RECOVERY - mwtask141 mathoid on mwtask141 is OK: TCP OK - 0.000 second response time on 2a10:6740::6:504 port 10044 [03:10:53] RECOVERY - ping6 on mwtask141 is OK: PING OK - Packet loss = 0%, RTA = 1.04 ms [03:11:20] RECOVERY - cp25 Varnish Backends on cp25 is OK: All 15 backends are healthy [03:11:22] RECOVERY - cp35 Varnish Backends on cp35 is OK: All 15 backends are healthy [03:11:27] RECOVERY - db142 MariaDB on db142 is OK: Uptime: 243 Threads: 12 Questions: 12219 Slow queries: 498 Opens: 1183 Open tables: 1176 Queries per second avg: 50.283 [03:11:31] RECOVERY - cp34 Varnish Backends on cp34 is OK: All 15 backends are healthy [03:11:33] RECOVERY - cp24 Varnish Backends on cp24 is OK: All 15 backends are healthy [03:11:44] RECOVERY - mwtask141 MediaWiki Rendering on mwtask141 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.378 second response time [03:11:47] RECOVERY - db142 MariaDB Connections on db142 is OK: OK connection usage: 1.2%Current connections: 6 [03:11:57] PROBLEM - mon141 Current Load on mon141 is WARNING: LOAD WARNING - total load average: 0.52, 2.72, 3.96 [03:13:33] RECOVERY - cloud14 Current Load on cloud14 is OK: OK - load average: 17.46, 41.43, 62.36 [03:14:06] PROBLEM - swiftproxy111 Puppet on swiftproxy111 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:15:29] PROBLEM - mail121 Puppet on mail121 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:15:57] RECOVERY - mon141 Current Load on mon141 is OK: LOAD OK - total load average: 0.56, 1.58, 3.20 [03:16:13] PROBLEM - puppet141 Puppet on puppet141 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:16:31] PROBLEM - ldap141 Current Load on ldap141 is WARNING: LOAD WARNING - total load average: 0.03, 0.88, 1.86 [03:16:41] PROBLEM - sdiy.info - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns1.wikitide.net.', 'ns2.wikitide.net.'], 'CNAME': None} [03:17:26] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 0.03, 1.36, 3.68 [03:18:29] RECOVERY - ldap141 Current Load on ldap141 is OK: LOAD OK - total load average: 0.05, 0.60, 1.63 [03:19:26] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.17, 0.98, 3.26 [03:19:41] PROBLEM - bast141 Current Load on bast141 is WARNING: LOAD WARNING - total load average: 0.00, 0.59, 1.89 [03:20:41] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/169f0d8cbbb4 [03:20:44] [02miraheze/mw-config] 07paladox 03169f0d8 - Set $wgChronologyProtectorStash [03:20:46] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 0.35, 2.44, 7.36 [03:20:47] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [03:20:50] [02mw-config] 07paladox opened pull request 03#5375: Set $wgChronologyProtectorStash - 13https://github.com/miraheze/mw-config/pull/5375 [03:21:09] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/169f0d8cbbb4...d8ea00fed1f6 [03:21:10] [02miraheze/mw-config] 07github-actions 03d8ea00f - CI: lint code to MediaWiki standards [03:21:11] [02mw-config] 07github-actions[bot] synchronize pull request 03#5375: Set $wgChronologyProtectorStash - 13https://github.com/miraheze/mw-config/pull/5375 [03:21:36] [02mw-config] 07paladox closed pull request 03#5375: Set $wgChronologyProtectorStash - 13https://github.com/miraheze/mw-config/pull/5375 [03:21:37] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [03:21:38] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [03:21:40] RECOVERY - bast141 Current Load on bast141 is OK: LOAD OK - total load average: 0.03, 0.40, 1.66 [03:21:40] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/eb8f0bc1a2e7...c24b5cd535da [03:21:40] !log [paladox@test131] starting deploy of {'config': True} to all [03:21:42] [02miraheze/mw-config] 07paladox 03c24b5cd - Set $wgChronologyProtectorStash (#5375) [03:21:42] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [03:21:42] miraheze/mw-config - paladox the build passed. [03:21:43] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [03:21:43]