[00:00:39] RECOVERY - mw131 Disk Space on mw131 is OK: DISK OK - free space: / 2946 MB (12% inode=82%); [00:02:50] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 5.00, 4.37, 2.17 [00:04:50] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 0.98, 3.07, 1.96 [00:05:33] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [00:11:15] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=purcalwiki (END - exit=0) [00:11:16] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=purcalwiki --active --update (END - exit=0) [00:11:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:11:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:12:50] PROBLEM - cp22 Current Load on cp22 is CRITICAL: LOAD CRITICAL - total load average: 36.95, 11.85, 5.29 [00:13:48] PROBLEM - cp23 Current Load on cp23 is CRITICAL: LOAD CRITICAL - total load average: 4.67, 5.95, 3.36 [00:23:26] RECOVERY - cp23 Current Load on cp23 is OK: LOAD OK - total load average: 0.41, 2.94, 3.34 [00:24:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 1.93, 2.02, 3.86 [00:30:58] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 1.80, 1.98, 3.25 [00:35:30] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:38:20] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=piesekmaksiowiki --no-updates --username-prefix=wikia:piesekmaksio /home/macfan/ImportDump/piesekmaksiowiki-20230808220156.xml (START) [00:38:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:40:50] PROBLEM - cp22 Current Load on cp22 is WARNING: LOAD WARNING - total load average: 0.46, 0.68, 3.88 [00:42:36] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=piesekmaksiowiki --no-updates --username-prefix=wikia:piesekmaksio /home/macfan/ImportDump/piesekmaksiowiki-20230808220156.xml (END - exit=0) [00:42:36] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=piesekmaksiowiki (START) [00:42:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:42:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:44:50] RECOVERY - cp22 Current Load on cp22 is OK: LOAD OK - total load average: 0.29, 0.51, 3.08 [00:44:55] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=piesekmaksiowiki (END - exit=0) [00:44:56] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=piesekmaksiowiki --active --update (END - exit=0) [00:44:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:45:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:48:32] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=purcalwiki --no-updates --username-prefix=w /home/macfan/ImportDump/purcalwiki-20230810201318.xml (START) [00:48:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:51:34] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=purcalwiki --no-updates --username-prefix=w /home/macfan/ImportDump/purcalwiki-20230810201318.xml (END - exit=0) [00:51:35] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=purcalwiki (START) [00:51:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:51:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:54:35] PROBLEM - graylog131 NTP time on graylog131 is WARNING: NTP WARNING: Offset -0.1288007498 secs [00:56:58] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 8.16, 4.88, 3.45 [01:05:30] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [01:18:15] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [01:20:35] RECOVERY - graylog131 NTP time on graylog131 is OK: NTP OK: Offset -0.09832033515 secs [01:22:47] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=purcalwiki (END - exit=0) [01:22:48] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=purcalwiki --active --update (END - exit=0) [01:22:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:22:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:32:41] PROBLEM - graylog131 NTP time on graylog131 is WARNING: NTP WARNING: Offset -0.1701547205 secs [01:38:41] RECOVERY - graylog131 NTP time on graylog131 is OK: NTP OK: Offset -0.006300657988 secs [01:42:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.51, 2.43, 3.84 [01:48:15] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [02:01:38] RECOVERY - swiftproxy111 NTP time on swiftproxy111 is OK: NTP OK: Offset -0.08973905444 secs [02:08:58] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.57, 3.68, 3.56 [02:12:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.86, 3.95, 3.71 [02:20:58] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.12, 3.12, 3.38 [03:38:11] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:07:30] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [04:09:57] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:33:02] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query zulaclub.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:36:49] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:39:27] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [05:02:20] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [05:06:07] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [05:31:39] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [05:54:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/120e000ccbd0 [05:54:22] [02miraheze/puppet] 07paladox 03120e000 - reload-vcl: fix support for varnish 6.2+ in auto discard [05:54:25] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [05:54:26] [02puppet] 07paladox opened pull request 03#3377: reload-vcl: fix support for varnish 6.2+ in auto discard - 13https://github.com/miraheze/puppet/pull/3377 [05:54:34] [02puppet] 07paladox edited pull request 03#3377: reload-vcl: fix support for varnish 6.2+ in auto discard - 13https://github.com/miraheze/puppet/pull/3377 [05:55:54] [02puppet] 07paladox closed pull request 03#3377: reload-vcl: fix support for varnish 6.2+ in auto discard - 13https://github.com/miraheze/puppet/pull/3377 [05:55:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7c2479b67604...4042a7315acc [05:55:58] [02miraheze/puppet] 07paladox 034042a73 - reload-vcl: fix support for varnish 6.2+ in auto discard (#3377) [05:56:00] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-10 [05:56:01] [02puppet] 07paladox deleted branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [06:00:01] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4042a7315acc...13aaf9cb5c51 [06:00:04] [02miraheze/puppet] 07paladox 0313aaf9c - reload-vcl: semi update documentation to new format in auto discard [06:05:02] PROBLEM - cp22 Varnish Backends on cp22 is WARNING: No backends detected. If this is an error, see readme.txt [06:06:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/13aaf9cb5c51...bdcef8a51e9c [06:06:03] [02miraheze/puppet] 07paladox 03bdcef8a - Update reload-vcl.py [06:11:02] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 15 backends are healthy [06:17:02] PROBLEM - cp22 Varnish Backends on cp22 is WARNING: No backends detected. If this is an error, see readme.txt [06:21:02] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 15 backends are healthy [06:22:33] [02miraheze/phabricator-extensions] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/compare/eb71d83e2066...74f097b98a80 [06:22:34] [02miraheze/phabricator-extensions] 07paladox 0374f097b - Add missing PhabricatorMediaWikiAuthProvider to map [06:23:30] miraheze/phabricator-extensions - paladox the build passed. [06:27:03] PROBLEM - cp22 Varnish Backends on cp22 is WARNING: No backends detected. If this is an error, see readme.txt [06:32:34] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.66, 3.54, 3.17 [06:32:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-10 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/43020ecec152 [06:32:38] [02miraheze/puppet] 07paladox 0343020ec - reload-vcl: discard reload_ vcl as well [06:32:39] [02puppet] 07paladox created branch 03paladox-patch-10 - 13https://github.com/miraheze/puppet [06:32:40] [02puppet] 07paladox opened pull request 03#3378: reload-vcl: discard reload_ vcl as well - 13https://github.com/miraheze/puppet/pull/3378 [06:33:02] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 15 backends are healthy [06:34:32] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.47, 3.90, 3.35 [06:38:27] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.35, 3.74, 3.42 [06:42:23] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.61, 3.29, 3.32 [06:50:12] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 4.00, 3.70, 3.47 [06:54:08] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.51, 3.31, 3.39 [07:10:04] PROBLEM - www.pyramidgames.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pyramidgames.wiki' expires in 15 day(s) (Sun 27 Aug 2023 07:04:54 GMT +0000). [07:12:15] PROBLEM - pyramidgames.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pyramidgames.wiki' expires in 15 day(s) (Sun 27 Aug 2023 07:04:54 GMT +0000). [08:11:03] RECOVERY - wiki.altilunium.my.id - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.altilunium.my.id' will expire on Wed 11 Oct 2023 05:49:31 GMT +0000. [08:43:18] PROBLEM - wiki.altilunium.my.id - LetsEncrypt on sslhost is CRITICAL: connect to address wiki.altilunium.my.id and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:17:36] PROBLEM - wiki.wisrail.de - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.wisrail.de All nameservers failed to answer the query. [10:27:26] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.altilunium.my.id All nameservers failed to answer the query. [10:46:42] PROBLEM - wiki.wisrail.de - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wisrail.de. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [10:57:26] PROBLEM - wiki.altilunium.my.id - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.altilunium.my.id could not be found [11:15:48] RECOVERY - wiki.wisrail.de - reverse DNS on sslhost is OK: SSL OK - wiki.wisrail.de reverse DNS resolves to cp23.miraheze.org - CNAME OK [12:10:58] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.37, 3.73, 3.32 [12:12:58] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.66, 3.40, 3.25 [12:18:15] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pt.eu.org. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:18:58] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.06, 3.70, 3.41 [12:20:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.51, 3.55, 3.39 [12:22:58] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.81, 3.39, 3.35 [12:28:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.25, 3.54, 3.43 [12:32:58] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.03, 3.63, 3.49 [12:44:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.67, 3.96, 3.83 [12:56:53] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 122.93 ms [12:57:23] PROBLEM - ping6 on cp32 is WARNING: PING WARNING - Packet loss = 0%, RTA = 123.32 ms [12:58:05] PROBLEM - ping6 on cp33 is WARNING: PING WARNING - Packet loss = 0%, RTA = 122.56 ms [13:02:58] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.65, 2.94, 3.39 [13:13:00] [02MirahezeMagic] 07amire80 opened pull request 03#428: Fix punctuation in an English message - 13https://github.com/miraheze/MirahezeMagic/pull/428 [13:19:59] miraheze/MirahezeMagic - amire80 the build has errored. [13:49:06] PROBLEM - wiki.danvs.net - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query danvs.net. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:18:09] RECOVERY - wiki.danvs.net - reverse DNS on sslhost is OK: SSL OK - wiki.danvs.net reverse DNS resolves to cp22.miraheze.org - CNAME OK [15:23:19] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/be00da53c8fc...e64496f43ed2 [15:23:21] [02miraheze/dns] 07paladox 03e64496f - Depool cp22 temporarily [15:25:03] [02MirahezeMagic] 07paladox closed pull request 03#428: Fix punctuation in an English message - 13https://github.com/miraheze/MirahezeMagic/pull/428 [15:25:05] [02miraheze/MirahezeMagic] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/bbdda5e1ef58...9136a84d955f [15:25:08] [02miraheze/MirahezeMagic] 07amire80 039136a84 - Fix punctuation in an English message (#428) [15:30:00] miraheze/MirahezeMagic - paladox the build has errored. [15:41:02] PROBLEM - cp22 Varnish Backends on cp22 is WARNING: No backends detected. If this is an error, see readme.txt [15:44:11] PROBLEM - cp22 Puppet on cp22 is WARNING: WARNING: Puppet is currently disabled, message: paladox, last run 31 minutes ago with 0 failures [15:48:15] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [15:49:02] RECOVERY - cp22 Varnish Backends on cp22 is OK: All 15 backends are healthy [15:53:47] PROBLEM - cp22 HTTPS on cp22 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.108 second response time [15:55:47] RECOVERY - cp22 HTTPS on cp22 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3713 bytes in 0.137 second response time [15:58:48] PROBLEM - uk.religiononfire.mar.in.ua - LetsEncrypt on sslhost is CRITICAL: connect to address uk.religiononfire.mar.in.ua and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:18:15] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [16:22:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/629e4c537ecf [16:22:18] [02miraheze/puppet] 07paladox 03629e4c5 - varnish:Remove duplicate ExecReload from systemd unit [16:22:20] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [16:22:21] [02puppet] 07paladox opened pull request 03#3379: varnish:Remove duplicate ExecReload from systemd unit - 13https://github.com/miraheze/puppet/pull/3379 [16:22:25] [02puppet] 07paladox edited pull request 03#3379: varnish: Remove duplicate ExecReload from systemd unit - 13https://github.com/miraheze/puppet/pull/3379 [16:22:38] [02puppet] 07paladox closed pull request 03#3379: varnish: Remove duplicate ExecReload from systemd unit - 13https://github.com/miraheze/puppet/pull/3379 [16:22:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bdcef8a51e9c...7509b36e7210 [16:22:41] [02miraheze/puppet] 07paladox 037509b36 - varnish: Remove duplicate ExecReload from systemd unit (#3379) [16:22:42] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11 [16:22:44] [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [16:24:11] RECOVERY - cp22 Puppet on cp22 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [16:28:23] [02miraheze/dns] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/e64496f43ed2...eead6608b0be [16:28:24] [02miraheze/dns] 07paladox 03eead660 - Revert "Depool cp22 temporarily" [16:28:37] RECOVERY - uk.religiononfire.mar.in.ua - LetsEncrypt on sslhost is OK: OK - Certificate 'uk.religiononfire.mar.in.ua' will expire on Fri 27 Oct 2023 17:44:36 GMT +0000. [16:30:44] PROBLEM - swiftobject121 Current Load on swiftobject121 is CRITICAL: CRITICAL - load average: 11.25, 6.95, 4.22 [16:30:49] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 22.35, 11.44, 6.28 [16:31:55] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.88, 5.46, 3.81 [16:32:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 17.78, 11.37, 6.53 [16:32:59] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is CRITICAL: CRITICAL - load average: 9.19, 6.77, 3.54 [16:33:00] PROBLEM - swiftobject113 Current Load on swiftobject113 is CRITICAL: CRITICAL - load average: 9.93, 6.99, 4.38 [16:33:55] PROBLEM - swiftobject112 Current Load on swiftobject112 is CRITICAL: CRITICAL - load average: 8.56, 6.82, 4.54 [16:35:33] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: CRITICAL - load average: 4.91, 3.81, 2.72 [16:35:55] PROBLEM - swiftobject112 Current Load on swiftobject112 is WARNING: WARNING - load average: 6.82, 6.62, 4.73 [16:36:44] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 4.67, 7.18, 5.35 [16:37:00] PROBLEM - swiftproxy131 Current Load on swiftproxy131 is WARNING: WARNING - load average: 5.73, 7.26, 4.57 [16:37:00] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 4.92, 7.07, 5.14 [16:37:32] PROBLEM - graylog131 Current Load on graylog131 is WARNING: WARNING - load average: 3.88, 3.66, 2.79 [16:39:55] RECOVERY - swiftobject112 Current Load on swiftobject112 is OK: OK - load average: 5.62, 6.46, 5.15 [16:40:44] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 5.76, 6.79, 5.63 [16:40:46] PROBLEM - ping6 on cp32 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 123.02 ms [16:40:59] RECOVERY - swiftproxy131 Current Load on swiftproxy131 is OK: OK - load average: 5.76, 6.71, 4.97 [16:41:00] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 4.99, 6.28, 5.27 [16:42:48] PROBLEM - ping6 on cp32 is WARNING: PING WARNING - Packet loss = 0%, RTA = 123.26 ms [16:43:30] RECOVERY - graylog131 Current Load on graylog131 is OK: OK - load average: 3.35, 3.37, 2.93 [17:02:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.60, 7.22, 7.87 [17:06:49] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 4.68, 6.19, 7.56 [17:08:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.62, 6.94, 7.43 [17:08:49] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 10.38, 7.19, 7.72 [17:09:59] PROBLEM - wiki.ff6worldscollide.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ff6worldscollide.com' expires in 15 day(s) (Sun 27 Aug 2023 16:54:23 GMT +0000). [17:10:49] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.70, 6.64, 7.47 [17:13:58] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1f301a18480d...cb534a348b7f [17:14:01] [02miraheze/ssl] 07MirahezeSSLBot 03cb534a3 - Bot: Update SSL cert for wiki.ff6worldscollide.com [17:14:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.12, 7.12, 7.31 [17:18:49] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.11, 5.67, 6.65 [17:24:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.22, 5.79, 6.62 [17:25:56] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [17:29:56] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [18:08:58] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.99, 3.69, 2.90 [18:09:45] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:00:58] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 2.68, 3.37, 3.91 [19:08:00] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1001998782 secs [19:08:44] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [19:12:58] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.40, 2.72, 3.27 [19:39:56] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [19:43:56] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [20:03:45] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.23, 3.63, 3.03 [20:05:43] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.18, 3.47, 3.04 [20:11:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/86d073bfed72 [20:11:04] [02miraheze/puppet] 07paladox 0386d073b - varnish: Change KillMode to Process [20:11:06] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [20:11:07] [02puppet] 07paladox opened pull request 03#3380: varnish: Change KillMode to Process - 13https://github.com/miraheze/puppet/pull/3380 [20:11:36] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.04, 3.71, 3.28 [20:12:49] [02puppet] 07paladox synchronize pull request 03#3380: varnish: Change KillMode to Process - 13https://github.com/miraheze/puppet/pull/3380 [20:12:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/86d073bfed72...3a73ec33258d [20:12:53] [02miraheze/puppet] 07paladox 033a73ec3 - Update varnish.systemd.erb [20:15:32] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.99, 3.78, 3.40 [20:16:10] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [20:17:12] [02puppet] 07paladox closed pull request 03#3380: varnish: Change KillMode to Process - 13https://github.com/miraheze/puppet/pull/3380 [20:17:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7509b36e7210...db513bd2003a [20:17:17] [02miraheze/puppet] 07paladox 03db513bd - varnish: Change KillMode to Process (#3380) [20:17:20] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11 [20:17:22] [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [20:25:21] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.12, 3.81, 3.56 [20:29:16] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.20, 3.68, 3.59 [20:43:01] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.34, 2.93, 3.29 [21:13:48] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [21:18:00] PROBLEM - cp32 NTP time on cp32 is CRITICAL: NTP CRITICAL: Offset 0.7452906072 secs [21:21:51] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.12, 3.55, 3.06 [21:23:49] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.26, 3.43, 3.08 [21:25:47] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.38, 3.34, 3.09 [21:29:40] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.49, 3.52, 3.22 [21:33:36] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 3.11, 3.37, 3.23 [21:37:29] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.10, 3.42, 3.29 [21:39:27] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 4.12, 3.69, 3.41 [21:41:25] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 3.39, 3.63, 3.43 [21:43:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/346327471231 [21:43:16] [02miraheze/puppet] 07paladox 033463274 - mem131/mem141: set role::memcached::threads to 6 [21:43:17] [02puppet] 07paladox created branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [21:43:20] [02puppet] 07paladox opened pull request 03#3381: mem131/mem141: set role::memcached::threads to 6 - 13https://github.com/miraheze/puppet/pull/3381 [21:43:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-11 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/346327471231...62236862ad6f [21:43:42] [02miraheze/puppet] 07paladox 036223686 - Update mem141.yaml [21:43:44] [02puppet] 07paladox synchronize pull request 03#3381: mem131/mem141: set role::memcached::threads to 6 - 13https://github.com/miraheze/puppet/pull/3381 [21:43:49] [02puppet] 07paladox closed pull request 03#3381: mem131/mem141: set role::memcached::threads to 6 - 13https://github.com/miraheze/puppet/pull/3381 [21:43:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/db513bd2003a...281d5eeef446 [21:43:55] [02miraheze/puppet] 07paladox 03281d5ee - mem131/mem141: set role::memcached::threads to 6 (#3381) [21:43:57] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-11 [21:43:59] [02puppet] 07paladox deleted branch 03paladox-patch-11 - 13https://github.com/miraheze/puppet [21:49:03] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/007f98428d83...9f8d93b2973f [21:49:04] [02miraheze/mw-config] 07paladox 039f8d93b - Store message cache in memcached-mem-2 [21:49:27] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [21:49:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:40] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 12s [21:49:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:46] !log [paladox@mwtask141] starting deploy of {'config': True} to all [21:49:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:49:54] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [21:49:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:50:02] miraheze/mw-config - paladox the build passed. [21:50:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.79, 6.28, 4.95 [21:52:08] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.01, 5.47, 4.80 [21:55:48] !log [@test131] starting deploy of {'config': True} to all [21:55:49] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [21:55:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:55:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:08:58] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 2.50, 3.03, 3.38 [23:04:21] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 15.05, 9.32, 4.42 [23:10:21] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 0.61, 5.79, 4.80 [23:18:15] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [23:45:16] RECOVERY - wiki.altilunium.my.id - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.altilunium.my.id' will expire on Wed 11 Oct 2023 05:49:31 GMT +0000. [23:48:15] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'}