[00:06:17] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.108867228 secs [00:10:17] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.0973854661 secs [00:35:10] [02WikiDiscover] 07Universal-Omega synchronize pull request 03#93: Support CreateWiki configs - 13https://github.com/miraheze/WikiDiscover/pull/93 [00:35:31] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:38:01] miraheze/WikiDiscover - Universal-Omega the build has errored. [00:42:22] [02WikiDiscover] 07Universal-Omega synchronize pull request 03#93: Support CreateWiki configs - 13https://github.com/miraheze/WikiDiscover/pull/93 [00:45:07] miraheze/WikiDiscover - Universal-Omega the build has errored. [00:48:58] [02WikiDiscover] 07Universal-Omega synchronize pull request 03#93: Support CreateWiki configs - 13https://github.com/miraheze/WikiDiscover/pull/93 [00:51:53] miraheze/WikiDiscover - Universal-Omega the build has errored. [00:52:17] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset -0.1094231308 secs [00:53:51] [02WikiDiscover] 07Universal-Omega synchronize pull request 03#93: Support CreateWiki configs - 13https://github.com/miraheze/WikiDiscover/pull/93 [00:54:17] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset -0.08973693848 secs [00:55:03] [02WikiDiscover] 07Universal-Omega synchronize pull request 03#93: Support CreateWiki configs - 13https://github.com/miraheze/WikiDiscover/pull/93 [00:56:38] [02WikiDiscover] 07Universal-Omega synchronize pull request 03#93: Support CreateWiki configs - 13https://github.com/miraheze/WikiDiscover/pull/93 [00:56:43] miraheze/WikiDiscover - Universal-Omega the build has errored. [00:59:53] miraheze/WikiDiscover - Universal-Omega the build passed. [01:08:14] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.89, 1.53, 0.91 [01:10:14] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.10, 1.78, 1.08 [01:14:14] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.80, 1.85, 1.27 [01:21:50] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [01:22:14] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.82, 2.24, 1.66 [01:34:14] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.86, 2.00, 1.88 [01:36:14] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 1.84, 2.01, 1.90 [01:46:16] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.2513938546 secs [01:56:14] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.76, 1.97, 2.00 [01:58:14] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.08, 2.09, 2.04 [02:00:14] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.64, 1.91, 1.98 [02:04:15] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [02:08:14] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.15, 1.91, 1.94 [02:10:15] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset -0.02246385813 secs [02:12:14] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.85, 1.90, 1.93 [02:16:14] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 1.09, 1.34, 1.68 [02:49:50] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:03:56] PROBLEM - db121 Backups SQL on db121 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864232 seconds old and 275 bytes [03:13:28] [02mw-config] 07brandon-wm opened pull request 03#5196: T10707: Add CleanSignatures to ManageWiki - 13https://github.com/miraheze/mw-config/pull/5196 [03:14:26] miraheze/mw-config - brandon-wm the build passed. [03:19:50] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [03:27:54] [02mw-config] 07brandon-wm synchronize pull request 03#5196: T10707: Add CleanSignatures to ManageWiki - 13https://github.com/miraheze/mw-config/pull/5196 [03:28:56] miraheze/mw-config - brandon-wm the build has errored. [03:29:57] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1764494479 secs [03:32:22] [02mw-config] 07brandon-wm synchronize pull request 03#5196: T10707: Add CleanSignatures to ManageWiki - 13https://github.com/miraheze/mw-config/pull/5196 [03:33:20] miraheze/mw-config - brandon-wm the build has errored. [03:46:03] [02mw-config] 07brandon-wm synchronize pull request 03#5196: T10707: Add CleanSignatures to ManageWiki - 13https://github.com/miraheze/mw-config/pull/5196 [03:46:57] miraheze/mw-config - brandon-wm the build passed. [03:59:57] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.04965335131 secs [04:07:12] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [04:33:57] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset -0.1209785044 secs [04:36:25] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.andreijiroh.uk.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [04:47:57] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.01307678223 secs [05:35:12] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp22.miraheze.org - CNAME OK [06:05:32] PROBLEM - raras.polandball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'raras.polandball.wiki' expires in 15 day(s) (Mon 01 May 2023 05:45:36 GMT +0000). [06:11:33] PROBLEM - mercy.theking1543.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mercy.theking1543.eu.org' expires in 15 day(s) (Mon 01 May 2023 05:47:41 GMT +0000). [06:12:27] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/efa1c2a311f1...2b3514793872 [06:12:29] [02miraheze/ssl] 07MirahezeSSLBot 032b35147 - Bot: Update SSL cert for raras.polandball.wiki [06:13:09] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2b3514793872...2e292f0366f0 [06:13:12] [02miraheze/ssl] 07MirahezeSSLBot 032e292f0 - Bot: Update SSL cert for mercy.theking1543.eu.org [06:14:29] PROBLEM - teamballs.polandball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'teamballs.polandball.wiki' expires in 15 day(s) (Mon 01 May 2023 05:44:33 GMT +0000). [06:18:43] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2e292f0366f0...84cadd1eb7b6 [06:18:45] [02miraheze/ssl] 07MirahezeSSLBot 0384cadd1 - Bot: Update SSL cert for teamballs.polandball.wiki [06:31:02] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [06:40:37] RECOVERY - mercy.theking1543.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'mercy.theking1543.eu.org' will expire on Fri 14 Jul 2023 05:13:02 GMT +0000. [06:43:30] RECOVERY - teamballs.polandball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'teamballs.polandball.wiki' will expire on Fri 14 Jul 2023 05:18:35 GMT +0000. [06:59:01] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:03:31] RECOVERY - raras.polandball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'raras.polandball.wiki' will expire on Fri 14 Jul 2023 05:12:19 GMT +0000. [07:07:23] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [07:36:34] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp23.miraheze.org - CNAME OK [07:48:16] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1058256924 secs [08:12:15] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.08106264472 secs [08:20:18] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1806368232 secs [08:48:58] PROBLEM - natiospherae.polandball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'natiospherae.polandball.wiki' expires in 15 day(s) (Mon 01 May 2023 08:43:01 GMT +0000). [08:50:17] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.0754391551 secs [08:51:53] PROBLEM - ru.countryhumans.polandball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ru.countryhumans.polandball.wiki' expires in 15 day(s) (Mon 01 May 2023 08:43:45 GMT +0000). [08:52:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/84cadd1eb7b6...9ddd882b57ac [08:52:40] [02miraheze/ssl] 07MirahezeSSLBot 039ddd882 - Bot: Update SSL cert for natiospherae.polandball.wiki [08:53:30] PROBLEM - es.countryhumans.polandball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'es.countryhumans.polandball.wiki' expires in 15 day(s) (Mon 01 May 2023 08:43:18 GMT +0000). [08:57:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9ddd882b57ac...20eadd5c28a9 [08:57:35] [02miraheze/ssl] 07MirahezeSSLBot 0320eadd5 - Bot: Update SSL cert for ru.countryhumans.polandball.wiki [09:01:25] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/20eadd5c28a9...70b841a3fb3b [09:01:28] [02miraheze/ssl] 07MirahezeSSLBot 0370b841a - Bot: Update SSL cert for es.countryhumans.polandball.wiki [09:17:56] RECOVERY - natiospherae.polandball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'natiospherae.polandball.wiki' will expire on Fri 14 Jul 2023 07:52:30 GMT +0000. [09:20:45] RECOVERY - ru.countryhumans.polandball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ru.countryhumans.polandball.wiki' will expire on Fri 14 Jul 2023 07:57:27 GMT +0000. [09:31:03] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [09:40:42] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:51:27] RECOVERY - es.countryhumans.polandball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'es.countryhumans.polandball.wiki' will expire on Fri 14 Jul 2023 08:01:18 GMT +0000. [09:57:01] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 0 seconds ago with 0 failures [10:07:11] [02WikiDiscover] 07Reception123 closed pull request 03#93: Support CreateWiki configs - 13https://github.com/miraheze/WikiDiscover/pull/93 [10:07:14] [02miraheze/WikiDiscover] 07Reception123 pushed 0310 commits to 03master [+0/-0/±13] 13https://github.com/miraheze/WikiDiscover/compare/8c58bcda5781...11c09e933295 [10:07:16] [02miraheze/WikiDiscover] 07Universal-Omega 030a47e48 - Support CreateWiki configs [10:07:17] [02miraheze/WikiDiscover] 07Universal-Omega 03ce1b448 - Support CreateWiki configs [10:07:18] [02miraheze/WikiDiscover] 07Universal-Omega 03029cebc - Support CreateWiki configs [10:07:20] [02miraheze/WikiDiscover] ... and 7 more commits. [10:08:09] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/7e26f1ff0680...af4b21b15791 [10:08:12] [02miraheze/mediawiki] 07Reception123 03af4b21b - Update WikiDiscover [10:08:34] !log [reception@mwtask141] starting deploy of {'folders': 'w/extensions/WikiDiscover'} to all [10:08:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:08:40] !log [reception@mwtask141] finished deploy of {'folders': 'w/extensions/WikiDiscover'} to all - SUCCESS in 6s [10:08:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:09:18] miraheze/WikiDiscover - Reception123 the build has errored. [10:10:13] !log [reception@mwtask141] starting deploy of {'world': True} to all [10:10:17] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [10:10:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:14:52] !log [reception@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 278s [10:14:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:28:42] [02miraheze/mw-config] 07redbluegreenhat pushed 035 commits to 03master [+0/-0/±6] 13https://github.com/miraheze/mw-config/compare/bb71ba8fb9fb...d224f858fae6 [10:28:43] [02miraheze/mw-config] 07brandon-wm 03a97b6a4 - T10707: Add CleanSignatures to ManageWiki [10:28:44] [02miraheze/mw-config] 07brandon-wm 03776c6ba - T10707 [10:28:47] [02miraheze/mw-config] 07brandon-wm 03f4baa4e - Fix overridefault [10:28:49] [02miraheze/mw-config] ... and 2 more commits. [10:28:51] [02mw-config] 07redbluegreenhat closed pull request 03#5196: T10707: Add CleanSignatures to ManageWiki - 13https://github.com/miraheze/mw-config/pull/5196 [10:29:39] miraheze/mw-config - redbluegreenhat the build passed. [10:29:50] !log [alex@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [10:29:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:30:04] !log [alex@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 14s [10:30:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:57:08] Orange_Star: use squash and merge rather than merge [10:57:11] !log [@test131] starting deploy of {'config': True} to all [10:57:12] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [10:57:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:57:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:42:38] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [11:55:57] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1007040739 secs [11:57:57] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09795415401 secs [12:18:17] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.101025492 secs [12:36:17] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09429770708 secs [12:41:47] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [14:39:57] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.100405097 secs [14:49:57] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.08204030991 secs [15:00:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/bfab0a642398 [15:00:48] [02miraheze/puppet] 07paladox 03bfab0a6 - logbot: allow setting logging level [15:00:49] [02puppet] 07paladox created branch 03paladox-patch-3 - 13https://github.com/miraheze/puppet [15:00:51] [02puppet] 07paladox opened pull request 03#3187: logbot: allow setting logging level - 13https://github.com/miraheze/puppet/pull/3187 [15:01:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-3 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/bfab0a642398...006cc6d78f7f [15:01:37] [02miraheze/puppet] 07paladox 03006cc6d - Update config.py [15:01:40] [02puppet] 07paladox synchronize pull request 03#3187: logbot: allow setting logging level - 13https://github.com/miraheze/puppet/pull/3187 [15:09:57] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1011527479 secs [15:11:57] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09677881002 secs [15:21:54] PROBLEM - mon141 IRC-Discord Relay Bot on mon141 is CRITICAL: PROCS CRITICAL: 3 processes with args 'relaybot' [15:31:54] RECOVERY - mon141 IRC-Discord Relay Bot on mon141 is OK: PROCS OK: 2 processes with args 'relaybot' [15:33:57] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1038264632 secs [15:35:58] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09430414438 secs [16:05:02] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.4064414501190186 seconds [16:36:51] [02CreateWiki] 07Universal-Omega closed pull request 03#393: Add a hook to change the cw_wikis table from being used - 13https://github.com/miraheze/CreateWiki/pull/393 [16:39:42] [02WikiDiscover] 07Universal-Omega opened pull request 03#94: Fix tabbing - 13https://github.com/miraheze/WikiDiscover/pull/94 [16:43:30] miraheze/WikiDiscover - Universal-Omega the build passed. [16:48:02] [02CreateWiki] 07Reception123 reviewed pull request 03#394 commit - 13https://github.com/miraheze/CreateWiki/pull/394#discussion_r1167580564 [16:48:11] !log [salt-user@mw131] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=tsbwiki (END - exit=0) [16:48:12] !log [salt-user@mw132] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=tsbwiki (END - exit=0) [16:48:13] !log [salt-user@mw142] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=tsbwiki (END - exit=0) [16:48:14] !log [salt-user@mw122] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=tsbwiki (END - exit=0) [16:48:15] !log [salt-user@mw121] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=tsbwiki (END - exit=0) [16:48:16] !log [salt-user@mw141] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=tsbwiki (END - exit=0) [16:48:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:48:17] !log [salt-user@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=tsbwiki (END - exit=0) [16:48:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:48:21] [02WikiDiscover] 07Reception123 closed pull request 03#94: Fix tabbing - 13https://github.com/miraheze/WikiDiscover/pull/94 [16:48:24] [02miraheze/WikiDiscover] 07Reception123 pushed 032 commits to 03master [+0/-0/±2] 13https://github.com/miraheze/WikiDiscover/compare/11c09e933295...bf2d6368fdbb [16:48:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:48:27] [02miraheze/WikiDiscover] 07Universal-Omega 03e91855a - Fix tabbing [16:48:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:48:30] [02miraheze/WikiDiscover] 07Reception123 03bf2d636 - Merge pull request #94 from Universal-Omega/patch-5 [16:48:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:48:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:48:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:51:50] miraheze/WikiDiscover - Reception123 the build passed. [16:54:54] [02CreateWiki] 07Universal-Omega opened pull request 03#395: Support checking configs for states - 13https://github.com/miraheze/CreateWiki/pull/395 [16:57:54] [02CreateWiki] 07Universal-Omega reviewed pull request 03#394 commit - 13https://github.com/miraheze/CreateWiki/pull/394#discussion_r1167581577 [16:58:08] miraheze/CreateWiki - Universal-Omega the build has errored. [16:59:57] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset -0.1109341085 secs [17:01:43] [02CreateWiki] 07Universal-Omega synchronize pull request 03#395: Support checking configs for states - 13https://github.com/miraheze/CreateWiki/pull/395 [17:04:19] miraheze/CreateWiki - Universal-Omega the build has errored. [17:06:00] [02CreateWiki] 07Universal-Omega synchronize pull request 03#395: Support checking configs for states - 13https://github.com/miraheze/CreateWiki/pull/395 [17:07:57] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.09802678227 secs [17:08:37] miraheze/CreateWiki - Universal-Omega the build has errored. [17:12:28] [02CreateWiki] 07Universal-Omega synchronize pull request 03#395: Support checking configs for states - 13https://github.com/miraheze/CreateWiki/pull/395 [17:16:27] miraheze/CreateWiki - Universal-Omega the build has errored. [17:19:41] [02CreateWiki] 07Universal-Omega synchronize pull request 03#395: Support checking configs for states - 13https://github.com/miraheze/CreateWiki/pull/395 [17:24:17] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.29, 5.03, 2.80 [17:25:56] [02CreateWiki] 07Universal-Omega edited pull request 03#395: Support checking configs for states - 13https://github.com/miraheze/CreateWiki/pull/395 [17:26:17] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.46, 4.32, 2.84 [17:27:47] miraheze/CreateWiki - Universal-Omega the build passed. [17:31:03] [02CreateWiki] 07Universal-Omega synchronize pull request 03#394: Add config to disable the ManageInactiveWikis script - 13https://github.com/miraheze/CreateWiki/pull/394 [17:33:17] [02CreateWiki] 07Universal-Omega synchronize pull request 03#394: Add config to disable the ManageInactiveWikis script - 13https://github.com/miraheze/CreateWiki/pull/394 [17:34:21] [02CreateWiki] 07Universal-Omega synchronize pull request 03#394: Add config to disable the ManageInactiveWikis script - 13https://github.com/miraheze/CreateWiki/pull/394 [17:38:52] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/62a0b33857d7 [17:38:54] [02miraheze/mw-config] 07Reception123 0362a0b33 - set wgCreateWikiEnableManageInactiveWikis to true [17:38:56] [02mw-config] 07Reception123 created branch 03Reception123-patch-3 - 13https://github.com/miraheze/mw-config [17:39:09] [02mw-config] 07Reception123 opened pull request 03#5197: set wgCreateWikiEnableManageInactiveWikis to true - 13https://github.com/miraheze/mw-config/pull/5197 [17:39:13] miraheze/CreateWiki - Universal-Omega the build passed. [17:39:42] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/62a0b33857d7...e242989ef94c [17:39:43] [02miraheze/mw-config] 07github-actions 03e242989 - CI: lint code to MediaWiki standards [17:39:46] [02mw-config] 07github-actions[bot] synchronize pull request 03#5197: set wgCreateWikiEnableManageInactiveWikis to true - 13https://github.com/miraheze/mw-config/pull/5197 [17:40:13] miraheze/mw-config - Reception123 the build passed. [17:42:18] miraheze/CreateWiki - Universal-Omega the build passed. [17:51:15] [02CreateWiki] 07Reception123 closed pull request 03#394: Add config to disable the ManageInactiveWikis script - 13https://github.com/miraheze/CreateWiki/pull/394 [17:51:16] [02miraheze/CreateWiki] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/CreateWiki/compare/a01e9d3b89e7...98dd2c0eccc2 [17:51:17] [02miraheze/CreateWiki] 07Universal-Omega 0398dd2c0 - Add config to disable the ManageInactiveWikis script (#394) [17:51:20] [02mw-config] 07Reception123 closed pull request 03#5197: set wgCreateWikiEnableManageInactiveWikis to true - 13https://github.com/miraheze/mw-config/pull/5197 [17:51:22] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/d224f858fae6...d48392eefa44 [17:51:23] [02miraheze/mw-config] 07Reception123 03d48392e - set wgCreateWikiEnableManageInactiveWikis to true (#5197) [17:51:25] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-3 [17:51:27] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-3 - 13https://github.com/miraheze/mw-config [17:52:13] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/af4b21b15791...37e4cff5d3ff [17:52:16] [02miraheze/mediawiki] 07Reception123 0337e4cff - Update CreateWiki [17:52:18] miraheze/mw-config - Reception123 the build passed. [17:57:17] !log [@test131] starting deploy of {'config': True} to all [17:57:18] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [17:57:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:57:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:01:19] miraheze/CreateWiki - Reception123 the build passed. [18:10:40] !log [@mwtask141] starting deploy of {'config': True} to all [18:10:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:10:46] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 5s [18:10:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:15] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1102432311 secs [18:40:17] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1820017099 secs [18:54:17] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.06984007359 secs [18:58:15] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.07983851433 secs [19:01:57] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1031372249 secs [19:15:58] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09914138913 secs [19:22:38] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [19:35:40] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [19:40:18] [02CreateWiki] 07Universal-Omega opened pull request 03#396: Make WikiManager::create() more extendable - 13https://github.com/miraheze/CreateWiki/pull/396 [19:43:30] miraheze/CreateWiki - Universal-Omega the build has errored. [19:45:10] [02CreateWiki] 07Universal-Omega synchronize pull request 03#396: Make WikiManager::create() more extendable - 13https://github.com/miraheze/CreateWiki/pull/396 [19:48:15] miraheze/CreateWiki - Universal-Omega the build has errored. [19:49:19] [02CreateWiki] 07Universal-Omega synchronize pull request 03#396: Make WikiManager::create() more extendable - 13https://github.com/miraheze/CreateWiki/pull/396 [19:51:28] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [19:52:59] [02CreateWiki] 07Universal-Omega synchronize pull request 03#396: Make WikiManager::create() more extendable - 13https://github.com/miraheze/CreateWiki/pull/396 [19:53:05] miraheze/CreateWiki - Universal-Omega the build has errored. [20:00:53] miraheze/CreateWiki - Universal-Omega the build passed. [20:04:44] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [20:16:15] PROBLEM - cp32 NTP time on cp32 is WARNING: NTP WARNING: Offset 0.1108896434 secs [20:27:03] [02CreateWiki] 07Universal-Omega opened pull request 03#397: WikiManager: support not setting $wgCreateWikiStateDays - 13https://github.com/miraheze/CreateWiki/pull/397 [20:32:11] [02CreateWiki] 07Universal-Omega closed pull request 03#397: WikiManager: support not setting $wgCreateWikiStateDays - 13https://github.com/miraheze/CreateWiki/pull/397 [20:34:15] RECOVERY - cp32 NTP time on cp32 is OK: NTP OK: Offset 0.09990265965 secs [20:34:48] miraheze/CreateWiki - Universal-Omega the build passed. [21:51:50] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query en.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:10:07] RECOVERY - Host cloud11 is UP: PING OK - Packet loss = 0%, RTA = 0.53 ms [22:10:56] RECOVERY - cloud11 SSH on cloud11 is OK: SSH OK - OpenSSH_7.9p1 Debian-10+deb10u2 (protocol 2.0) [22:11:26] RECOVERY - ping6 on cloud11 is OK: PING OK - Packet loss = 0%, RTA = 0.48 ms [22:14:36] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:15:27] PROBLEM - ping6 on cloud11 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:200) [22:16:15] PROBLEM - Host cloud11 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:200) [22:20:41] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [22:58:46] RECOVERY - Host cloud11 is UP: PING OK - Packet loss = 0%, RTA = 1.84 ms [22:59:22] RECOVERY - ping6 on cloud11 is OK: PING OK - Packet loss = 0%, RTA = 0.62 ms [23:08:55] PROBLEM - cloud11 SSH on cloud11 is CRITICAL: Server answer [23:10:55] RECOVERY - cloud11 SSH on cloud11 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [23:13:45] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [23:18:57] PROBLEM - cloud11 SSH on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 and port 22: No route to host [23:19:11] PROBLEM - Host cloud11 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:200) [23:20:18] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.63, 5.86, 2.87 [23:22:17] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.29, 3.99, 2.55 [23:31:21] RECOVERY - Host cloud11 is UP: PING OK - Packet loss = 0%, RTA = 1.22 ms [23:32:58] RECOVERY - cloud11 SSH on cloud11 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [23:38:33] PROBLEM - cloud11 ferm_active on cloud11 is UNKNOWN: NRPE: Unable to read output [23:38:34] RECOVERY - cloud11 PowerDNS Recursor on cloud11 is OK: DNS OK: 0.144 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [23:38:46] RECOVERY - cloud11 Current Load on cloud11 is OK: OK - load average: 0.77, 0.48, 0.22 [23:39:02] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,2] - Reallocated_Sector_Ct is non-zero (21) --- [cciss,3] - Reallocated_Sector_Ct is non-zero (7) --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean| [23:40:17] RECOVERY - cloud11 Disk Space on cloud11 is OK: DISK OK - free space: / 24483 MB (85% inode=94%); [23:40:26] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [23:40:50] RECOVERY - cloud11 APT on cloud11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:40:50] PROBLEM - cloud11 conntrack_table_size on cloud11 is UNKNOWN: NRPE: Unable to read output [23:40:51] PROBLEM - cloud11 Puppet on cloud11 is UNKNOWN: NRPE: Unable to read output [23:40:52] RECOVERY - cloud11 NTP time on cloud11 is OK: NTP OK: Offset -0.003885954618 secs [23:42:32] RECOVERY - cloud11 ferm_active on cloud11 is OK: OK ferm input default policy is set [23:42:48] RECOVERY - cloud11 conntrack_table_size on cloud11 is OK: OK: nf_conntrack is 0 % full [23:42:50] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:43:20] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [23:53:32] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [23:54:21] PROBLEM - cloud11 APT on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [23:54:21] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [23:54:34] PROBLEM - cloud11 Disk Space on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [23:54:34] PROBLEM - cloud11 ferm_active on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [23:54:38] PROBLEM - cloud11 PowerDNS Recursor on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [23:54:38] PROBLEM - cloud11 Current Load on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [23:54:44] PROBLEM - cloud11 conntrack_table_size on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [23:54:51] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [23:54:59] PROBLEM - cloud11 SSH on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 and port 22: No route to host [23:55:16] PROBLEM - cloud11 NTP time on cloud11 is CRITICAL: connect to address 2a10:6740::6:200 port 5666: No route to hostconnect to host 2a10:6740::6:200 port 5666: No route to host [23:55:21] PROBLEM - ping6 on cloud11 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:200) [23:55:52] PROBLEM - Host cloud11 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:200) [23:57:55] RECOVERY - Host cloud11 is UP: PING OK - Packet loss = 0%, RTA = 0.43 ms [23:58:13] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [23:58:13] RECOVERY - cloud11 APT on cloud11 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:58:33] RECOVERY - cloud11 Disk Space on cloud11 is OK: DISK OK - free space: / 24618 MB (86% inode=94%); [23:58:33] RECOVERY - cloud11 ferm_active on cloud11 is OK: OK ferm input default policy is set [23:58:33] RECOVERY - cloud11 PowerDNS Recursor on cloud11 is OK: DNS OK: 0.024 seconds response time. miraheze.org returns 109.228.51.216,217.174.247.33,2a00:da00:1800:326::1,2a00:da00:1800:328::1 [23:58:38] RECOVERY - cloud11 Current Load on cloud11 is OK: OK - load average: 0.48, 0.73, 0.33 [23:58:43] RECOVERY - cloud11 conntrack_table_size on cloud11 is OK: OK: nf_conntrack is 0 % full [23:58:53] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:58:58] RECOVERY - cloud11 SSH on cloud11 is OK: SSH OK - OpenSSH_8.4p1 Debian-5+deb11u1 (protocol 2.0) [23:59:23] RECOVERY - ping6 on cloud11 is OK: PING OK - Packet loss = 0%, RTA = 0.50 ms [23:59:28] RECOVERY - cloud11 NTP time on cloud11 is OK: NTP OK: Offset -0.001580119133 secs [23:59:48] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,2] - Reallocated_Sector_Ct is non-zero (21) --- [cciss,3] - Reallocated_Sector_Ct is non-zero (7) --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean --- [cciss,6] - Device is clean|