[00:00:33] RECOVERY - en.religiononfire.mar.in.ua - LetsEncrypt on sslhost is OK: OK - Certificate 'en.religiononfire.mar.in.ua' will expire on Thu 01 Jun 2023 18:42:11 GMT +0000. [00:01:53] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 12.90, 7.17, 3.31 [00:02:14] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:02:44] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.41, 3.98, 2.96 [00:08:31] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.71, 3.82, 3.33 [00:10:20] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [00:10:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.07, 4.15, 3.52 [00:13:26] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.60, 6.48, 4.09 [00:13:46] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.45, 3.26, 3.77 [00:13:48] PROBLEM - atnarsia.com - LetsEncrypt on sslhost is CRITICAL: connect to address atnarsia.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:14:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.49, 3.77, 3.51 [00:15:46] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.73, 2.43, 3.40 [00:16:26] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.02, 4.01, 3.64 [00:17:25] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 5.82, 7.11, 4.98 [00:18:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.21, 3.59, 3.55 [00:19:24] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 5.24, 6.58, 5.05 [00:20:26] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.33, 4.40, 3.82 [00:22:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.74, 3.89, 3.72 [00:26:26] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.99, 4.80, 4.02 [00:30:18] PROBLEM - es141 Current Load on es141 is CRITICAL: CRITICAL - load average: 4.97, 3.71, 2.69 [00:31:19] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.67, 8.30, 6.06 [00:32:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.76, 3.96, 3.96 [00:34:26] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.75, 5.31, 4.43 [00:42:19] PROBLEM - es141 Current Load on es141 is WARNING: WARNING - load average: 2.95, 3.73, 3.51 [00:42:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.07, 3.42, 3.93 [00:44:19] RECOVERY - es141 Current Load on es141 is OK: OK - load average: 2.37, 3.31, 3.38 [00:46:26] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.63, 2.27, 3.33 [00:46:57] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query uk.religiononfire.mar.in.ua. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:47:13] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.95, 7.04, 7.73 [00:53:28] !log [void@puppet141] updated grants for cargouser on all db servers [00:53:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:59:08] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.45, 4.98, 6.45 [01:03:06] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.49, 7.42, 7.10 [01:05:05] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.12, 5.64, 6.48 [01:07:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 2.68, 4.25, 3.62 [01:11:02] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 11.04, 8.27, 7.23 [01:14:14] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.85, 3.08, 2.53 [01:15:01] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 4.43, 7.30, 7.16 [01:16:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.95, 2.61, 2.42 [01:17:00] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 0.82, 4.97, 6.32 [01:17:36] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.62, 3.67, 3.87 [01:21:27] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.63, 4.14, 3.99 [01:23:22] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.86, 3.75, 3.87 [01:29:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.99, 2.83, 3.36 [01:30:42] [02miraheze/ManageWiki] 07The-Voidwalker pushed 031 commit to 03patch-jobqueue [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/commit/919b8f0dd6f2 [01:30:44] [02miraheze/ManageWiki] 07The-Voidwalker 03919b8f0 - use MediaWikiServices for jobqueue [01:30:46] [02ManageWiki] 07The-Voidwalker created branch 03patch-jobqueue - 13https://github.com/miraheze/ManageWiki [01:30:51] [02ManageWiki] 07The-Voidwalker opened pull request 03#391: use MediaWikiServices for jobqueue - 13https://github.com/miraheze/ManageWiki/pull/391 [01:32:07] [02miraheze/ManageWiki] 07The-Voidwalker pushed 031 commit to 03patch-jobqueue [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/919b8f0dd6f2...beab5cfa2662 [01:32:10] [02miraheze/ManageWiki] 07The-Voidwalker 03beab5cf - bump mediawiki version requirement [01:32:12] [02ManageWiki] 07The-Voidwalker synchronize pull request 03#391: use MediaWikiServices for jobqueue - 13https://github.com/miraheze/ManageWiki/pull/391 [01:36:12] miraheze/ManageWiki - The-Voidwalker the build passed. [01:36:25] [02ManageWiki] 07The-Voidwalker closed pull request 03#391: use MediaWikiServices for jobqueue - 13https://github.com/miraheze/ManageWiki/pull/391 [01:36:28] [02miraheze/ManageWiki] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/ManageWiki/compare/7e2f14bfb7ed...724602d892a5 [01:36:29] [02miraheze/ManageWiki] 07The-Voidwalker 03724602d - use MediaWikiServices for jobqueue (#391) [01:39:47] miraheze/ManageWiki - The-Voidwalker the build passed. [01:44:53] [02miraheze/mediawiki] 07The-Voidwalker pushed 031 commit to 03REL1_40 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/81ffc3983f27...c60e98a38845 [01:44:56] [02miraheze/mediawiki] 07The-Voidwalker 03c60e98a - update ManageWiki [01:46:06] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [01:50:00] !log [void@test131] starting deploy of {'world': True} to all [01:50:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:51:50] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.religiononfire.mar.in.ua All nameservers failed to answer the query. [01:53:59] !log [void@test131] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [01:54:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:56:33] [02mediawiki] 07The-Voidwalker closed pull request 03#14460: Remove GitInfo hack - 13https://github.com/miraheze/mediawiki/pull/14460 [01:56:34] [02miraheze/mediawiki] 07The-Voidwalker pushed 031 commit to 03REL1_40 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/c60e98a38845...ab6c8f5ae91d [01:56:35] [02miraheze/mediawiki] 07Universal-Omega 03ab6c8f5 - Remove GitInfo hack (#14460) [01:57:02] !log [void@test131] starting deploy of {'world': True} to all [01:57:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:00:27] !log [void@test131] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [02:00:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:01:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.27, 4.43, 3.52 [02:02:59] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.77, 3.77, 3.38 [02:04:52] [02miraheze/mediawiki] 07The-Voidwalker pushed 032 commits to 03REL1_40 [+0/-0/±2] 13https://github.com/miraheze/mediawiki/compare/ab6c8f5ae91d...0d54af068258 [02:04:54] [02miraheze/mediawiki] 07The-Voidwalker 03f8e9f54 - update MirahezeMagic [02:04:55] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.00, 3.19, 3.21 [02:04:57] [02miraheze/mediawiki] 07The-Voidwalker 030d54af0 - Merge branch 'REL1_40' of https://github.com/miraheze/mediawiki into REL1_40 [02:05:19] !log [void@test131] starting deploy of {'world': True} to all [02:05:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:07:08] [02mediawiki] 07MacFan4000 commented on pull request 03#13790: Bump extensions/ManageWiki from `e16d36e` to `9d7f0fa` - 13https://github.com/miraheze/mediawiki/pull/13790#issuecomment-1502586093 [02:07:11] [02mediawiki] 07dependabot[bot] edited pull request 03#13790: Bump extensions/ManageWiki from `e16d36e` to `9d7f0fa` - 13https://github.com/miraheze/mediawiki/pull/13790 [02:07:30] [02mediawiki] 07MacFan4000 commented on pull request 03#14274: bump extensions/ManageWiki from `c7a0f84` to `7e2f14b` - 13https://github.com/miraheze/mediawiki/pull/14274#issuecomment-1502586300 [02:07:33] [02mediawiki] 07dependabot[bot] edited pull request 03#14274: bump extensions/ManageWiki from `c7a0f84` to `7e2f14b` - 13https://github.com/miraheze/mediawiki/pull/14274 [02:08:01] [02mediawiki] 07dependabot[bot] edited pull request 03#13790: Bump extensions/ManageWiki from `e16d36e` to `9d7f0fa` - 13https://github.com/miraheze/mediawiki/pull/13790 [02:08:16] [02mediawiki] 07dependabot[bot] opened pull request 03#14461: Bump extensions/ManageWiki from `c7a0f84` to `724602d` - 13https://github.com/miraheze/mediawiki/pull/14461 [02:08:18] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_39/extensions/ManageWiki-724602d [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/7ee1968ad2a8 [02:08:20] [02miraheze/mediawiki] 07dependabot[bot] 037ee1968 - Bump extensions/ManageWiki from `c7a0f84` to `724602d` [02:08:21] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_39/extensions/ManageWiki-724602d - 13https://github.com/miraheze/mediawiki [02:08:23] [02mediawiki] 07dependabot[bot] commented on pull request 03#14274: bump extensions/ManageWiki from `c7a0f84` to `7e2f14b` - 13https://github.com/miraheze/mediawiki/pull/14274#issuecomment-1502586776 [02:08:26] [02mediawiki] 07dependabot[bot] edited pull request 03#14274: bump extensions/ManageWiki from `c7a0f84` to `7e2f14b` - 13https://github.com/miraheze/mediawiki/pull/14274 [02:08:29] [02mediawiki] 07dependabot[bot] closed pull request 03#14274: bump extensions/ManageWiki from `c7a0f84` to `7e2f14b` - 13https://github.com/miraheze/mediawiki/pull/14274 [02:08:29] !log [void@test131] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [02:08:31] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_40/extensions/ManageWiki-7e2f14b - 13https://github.com/miraheze/mediawiki [02:08:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:08:34] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_40/extensions/ManageWiki-7e2f14b [02:08:37] [02mediawiki] 07github-actions[bot] labeled pull request 03#14461: Bump extensions/ManageWiki from `c7a0f84` to `724602d` - 13https://github.com/miraheze/mediawiki/pull/14461 [02:08:40] [02mediawiki] 07github-actions[bot] labeled pull request 03#14461: Bump extensions/ManageWiki from `c7a0f84` to `724602d` - 13https://github.com/miraheze/mediawiki/pull/14461 [02:08:42] [02mediawiki] 07github-actions[bot] labeled pull request 03#14461: Bump extensions/ManageWiki from `c7a0f84` to `724602d` - 13https://github.com/miraheze/mediawiki/pull/14461 [02:09:15] [02mediawiki] 07MacFan4000 closed pull request 03#13790: Bump extensions/ManageWiki from `e16d36e` to `9d7f0fa` - 13https://github.com/miraheze/mediawiki/pull/13790 [02:09:17] [02mediawiki] 07dependabot[bot] commented on pull request 03#13790: Bump extensions/ManageWiki from `e16d36e` to `9d7f0fa` - 13https://github.com/miraheze/mediawiki/pull/13790#issuecomment-1502587372 [02:09:23] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_39/extensions/ManageWiki-9d7f0fa - 13https://github.com/miraheze/mediawiki [02:09:24] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_39/extensions/ManageWiki-9d7f0fa [02:09:47] [02mediawiki] 07MacFan4000 commented on pull request 03#14067: Bump extensions/ManageWiki from `c7a0f84` to `7e2f14b` - 13https://github.com/miraheze/mediawiki/pull/14067#issuecomment-1502587720 [02:09:50] [02mediawiki] 07dependabot[bot] edited pull request 03#14067: Bump extensions/ManageWiki from `c7a0f84` to `7e2f14b` - 13https://github.com/miraheze/mediawiki/pull/14067 [02:10:35] [02mediawiki] 07dependabot[bot] edited pull request 03#14067: Bump extensions/ManageWiki from `c7a0f84` to `7e2f14b` - 13https://github.com/miraheze/mediawiki/pull/14067 [02:10:38] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_39/extensions/ManageWiki-7e2f14b [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/2bed8ca1cece...0a2fdc22683c [02:10:40] [02miraheze/mediawiki] 07dependabot[bot] 030a2fdc2 - Bump extensions/ManageWiki from `c7a0f84` to `7e2f14b` [02:10:43] [02mediawiki] 07dependabot[bot] synchronize pull request 03#14067: Bump extensions/ManageWiki from `c7a0f84` to `7e2f14b` - 13https://github.com/miraheze/mediawiki/pull/14067 [02:11:00] [02mediawiki] 07github-actions[bot] labeled pull request 03#14067: Bump extensions/ManageWiki from `c7a0f84` to `7e2f14b` - 13https://github.com/miraheze/mediawiki/pull/14067 [02:15:41] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:20:40] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [02:27:57] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.81, 3.29, 1.66 [02:29:55] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.63, 3.29, 1.89 [02:42:27] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.69, 6.98, 4.16 [02:44:26] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.89, 5.07, 3.80 [02:51:01] PROBLEM - test131 Puppet on test131 is WARNING: WARNING: Puppet is currently disabled, message: Testing Cargo --Void, last run 23 minutes ago with 0 failures [02:54:25] !log [void@test131] starting deploy of {'folders': 'config,w/extensions/MirahezeMagic'} to all [02:54:26] !log [void@test131] finished deploy of {'folders': 'config,w/extensions/MirahezeMagic'} to all - SUCCESS in 0s [02:54:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:54:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:03:28] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.28, 4.31, 3.29 [03:05:24] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.90, 3.47, 3.11 [03:07:20] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.32, 3.40, 3.12 [03:33:58] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1140026152 secs [03:35:02] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.93, 4.65, 3.26 [03:36:57] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.12, 3.68, 3.07 [03:40:48] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.31, 3.34, 3.08 [03:41:58] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07820379734 secs [04:08:18] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1308036447 secs [04:18:21] [02miraheze/puppet] 07The-Voidwalker pushed 032 commits to 03patch-LimitCargoDB [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/bfa1838f53bc^...9bc52e799cb9 [04:18:23] [02miraheze/puppet] 07The-Voidwalker 03bfa1838 - update grants for cargouser [04:18:26] [02miraheze/puppet] 07The-Voidwalker 039bc52e7 - update grants [04:18:29] [02puppet] 07The-Voidwalker created branch 03patch-LimitCargoDB - 13https://github.com/miraheze/puppet [04:18:30] [02puppet] 07The-Voidwalker opened pull request 03#3184: Patch-LimitCargoDB - 13https://github.com/miraheze/puppet/pull/3184 [04:19:07] [02puppet] 07The-Voidwalker closed pull request 03#3184: Patch-LimitCargoDB - 13https://github.com/miraheze/puppet/pull/3184 [04:19:17] [02miraheze/puppet] 07The-Voidwalker deleted branch 03patch-LimitCargoDB [04:19:19] [02puppet] 07The-Voidwalker deleted branch 03patch-LimitCargoDB - 13https://github.com/miraheze/puppet [04:20:17] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.08603909612 secs [04:20:30] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0be907ef7c84...241009f1d77c [04:20:31] [02miraheze/puppet] 07The-Voidwalker 03241009f - update cargo grants [04:23:52] [02miraheze/mw-config] 07The-Voidwalker pushed 031 commit to 03patch-addCargoScript [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ceedfd60e1af...58ccff1caff4 [04:23:53] [02miraheze/mw-config] 07The-Voidwalker 0358ccff1 - set $wgCargoDBname [04:23:54] [02mw-config] 07The-Voidwalker synchronize pull request 03#5182: add new createCargoDB.php to cargo install - 13https://github.com/miraheze/mw-config/pull/5182 [04:24:56] miraheze/mw-config - The-Voidwalker the build passed. [04:27:01] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:50:34] PROBLEM - db112 Disk Space on db112 is WARNING: DISK WARNING - free space: / 14595 MB (10% inode=98%); [04:57:20] !log [@test131] starting deploy of {'config': True} to all [04:57:21] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [04:57:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:57:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:05:37] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [05:35:14] PROBLEM - gs.sidem.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query gs.sidem.wiki. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:45:41] PROBLEM - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - uk.religiononfire.mar.in.ua All nameservers failed to answer the query. [06:03:54] PROBLEM - gs.sidem.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - gs.sidem.wiki All nameservers failed to answer the query. [06:15:16] RECOVERY - uk.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - uk.religiononfire.mar.in.ua reverse DNS resolves to cp22.miraheze.org - CNAME OK [06:33:09] RECOVERY - gs.sidem.wiki - reverse DNS on sslhost is OK: SSL OK - gs.sidem.wiki reverse DNS resolves to cp23.miraheze.org - CNAME OK [06:58:36] PROBLEM - mem141 NTP time on mem141 is WARNING: NTP WARNING: Offset 0.1079033315 secs [07:08:17] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 10.08, 6.32, 3.90 [07:10:17] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 3.24, 5.26, 3.83 [07:17:21] RECOVERY - mem141 NTP time on mem141 is OK: NTP OK: Offset 0.08994072676 secs [07:30:17] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1012299657 secs [07:36:17] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.09999075532 secs [08:07:58] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1217982173 secs [08:13:11] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.61, 10.04, 8.48 [08:13:57] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09811675549 secs [08:15:10] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.93, 9.25, 8.38 [08:29:46] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.66, 2.97, 1.85 [08:31:46] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.01, 2.20, 1.70 [08:56:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.37, 12.15, 9.73 [08:57:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.98, 10.78, 9.10 [08:58:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.44, 10.93, 9.53 [08:59:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.66, 9.78, 8.94 [09:00:37] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.57, 9.99, 9.35 [09:10:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [09:28:42] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 20.05, 8.07, 3.92 [09:34:36] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.78, 3.39, 3.17 [09:47:57] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1300979555 secs [09:57:58] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07949835062 secs [09:59:11] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.45, 2.45, 2.21 [10:01:10] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.24, 2.38, 2.22 [10:09:30] PROBLEM - cloud11 IPMI Sensors on cloud11 is UNKNOWN: Cannot access cache directory: /tmp/.freeipmi-root-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [10:11:57] PROBLEM - swiftac111 SSH on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 and port 22: No route to host [10:12:02] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 port 5666: No route to hostconnect to host 2a10:6740::6:202 port 5666: No route to host [10:12:10] PROBLEM - swiftac111 APT on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 port 5666: No route to hostconnect to host 2a10:6740::6:202 port 5666: No route to host [10:12:15] PROBLEM - swiftac111 Swift Container Service on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 and port 6001: No route to host [10:12:24] PROBLEM - swiftac111 conntrack_table_size on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 port 5666: No route to hostconnect to host 2a10:6740::6:202 port 5666: No route to host [10:12:25] PROBLEM - ping6 on swiftac111 is CRITICAL: CRITICAL - Destination Unreachable (2a10:6740::6:202) [10:12:29] PROBLEM - swiftac111 Puppet on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 port 5666: No route to hostconnect to host 2a10:6740::6:202 port 5666: No route to host [10:12:40] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 401 Unauthorized [10:12:49] PROBLEM - swiftac111 Swift Account Service on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 and port 6002: No route to host [10:12:57] PROBLEM - Host swiftac111 is DOWN: CRITICAL - Destination Unreachable (2a10:6740::6:202) [10:12:57] PROBLEM - swiftac111 ferm_active on swiftac111 is CRITICAL: connect to address 2a10:6740::6:202 port 5666: No route to hostconnect to host 2a10:6740::6:202 port 5666: No route to host [10:13:14] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 401 Unauthorized [10:13:21] PROBLEM - swiftproxy131 HTTP on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 401 Unauthorized [10:13:58] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 401 Unauthorized [10:28:46] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.78, 3.82, 2.45 [10:30:45] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.79, 3.61, 2.54 [10:32:10] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:32:43] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.82, 5.24, 3.26 [10:37:09] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [10:38:38] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.01, 2.97, 2.92 [10:46:00] PROBLEM - swiftproxy111 Puppet on swiftproxy111 is WARNING: WARNING: Puppet last ran 1 hour ago [10:52:23] PROBLEM - cloud11 Puppet on cloud11 is WARNING: WARNING: Puppet last ran 1 hour ago [10:58:17] PROBLEM - cp22 NTP time on cp22 is WARNING: NTP WARNING: Offset 0.1341492534 secs [11:09:57] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1727074981 secs [11:10:17] RECOVERY - cp22 NTP time on cp22 is OK: NTP OK: Offset 0.0862852037 secs [11:26:18] I received a Grafana alert to the email, looks like there are a lot of refreshLinks jobs for bluepageswiki [11:28:24] I can't help but think this has to do with SMW, judging by the smw.changePropagationUpdate jobs when the refreshLinks jobs cascaded in: https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki?orgId=1&from=now-12h&to=now&var-node=jobchron121&var-job=AssembleUploadChunks&var-job=CentralAuthCreateLocalAccountJob&var-job=CentralAuthUnattachUserJob&var-job=ChangeDeletionNotification&var-job=ChangeNotification&var-job=ChangeVisibilityNotific [11:28:24] ation&var-job=CleanTermsIfUnused&var-job=CreateWikiJob&var-job=DataDumpGenerateJob&var-job=DeleteJob&var-job=DeleteTranslatableBundleJob&var-job=DispatchChangeDeletionNotification&var-job=DispatchChangeVisibilityNotification&var-job=DispatchChanges&var-job=EchoNotificationDeleteJob&var-job=EchoNotificationJob&var-job=EchoPushNotificationRequest&var-job=EntityChangeNotification&var-job=GlobalNewFilesDeleteJob&var-job=GlobalNewFiles [11:28:25] InsertJob&var-job=GlobalNewFilesMoveJob&var-job=GlobalUserPageLocalJobSubmitJob&var-job=InitImageDataJob&var-job=LocalGlobalUserPageCacheUpdateJob&var-job=LocalPageMoveJob&var-job=LocalRenameUserJob&var-job=LoginNotifyChecks&var-job=MDCreatePage&var-job=MDDeletePage&var-job=MWScriptJob&var-job=MassMessageJob&var-job=MassMessageServerSideJob&var-job=MassMessageSubmitJob&var-job=MessageGroupStatesUpdaterJob&var-job=MessageGroupStats [11:28:30] RebuildJob&var-job=MessageIndexRebuildJob&var-job=MessageUpdateJob&var-job=MoveTranslatableBundleJob&var-job=NamespaceMigrationJob&var-job=PageProperties&var-job=PublishStashedFile&var-job=PurgeEntityData&var-job=RecordLintJob&var-job=RemovePIIJob&var-job=RenderTranslationPageJob&var-job=RequestWikiAIJob&var-job=SMWRefreshJob&var-job=SMWUpdateJob&var-job=SMW%5CChangePropagationClassUpdateJob&var-job=SMW%5CChangePropagationDispatch [11:28:35] Job&var-job=SMW%5CChangePropagationUpdateJob&var-job=SMW%5CEntityIdDisposerJob&var-job=SMW%5CFulltextSearchTableRebuildJob&var-job=SMW%5CFulltextSearchTableUpdateJob&var-job=SMW%5CPropertyStatisticsRebuildJob&var-job=SMW%5CRefreshJob&var-job=SMW%5CUpdateDispatcherJob&var-job=SMW%5CUpdateJob&var-job=SetContainersAccessJob&var-job=TTMServerMessageUpdateJob&var-job=ThumbnailRender&var-job=TranslatableBundleDeleteJob&var-job=Translata [11:28:42] bleBundleMoveJob&var-job=TranslateRenderJob&var-job=TranslateSandboxEmailJob&var-job=TranslationNotificationsEmailJob&var-job=TranslationNotificationsSubmitJob&var-job=TranslationsUpdateJob&var-job=UpdateMessageBundle&var-job=UpdateRepoOnDelete&var-job=UpdateRepoOnMove&var-job=UpdateTranslatablePageJob&var-job=UpdateTranslatorActivity&var-job=activityUpdateJob&var-job=cargoPopulateTable&var-job=categoryMembershipChange&var-job=cdn [11:28:47] Purge&var-job=clearUserWatchlist&var-job=clearWatchlistNotifications&var-job=compileArticleMetadata&var-job=constraintsRunCheck&var-job=constraintsTableUpdate&var-job=crosswikiSuppressUser&var-job=deleteLinks&var-job=deletePage&var-job=dtImport&var-job=edReparse&var-job=enotifNotify&var-job=enqueue&var-job=fixDoubleRedirect&var-job=flaggedrevs_CacheUpdate&var-job=globalUsageCachePurge&var-job=htmlCacheUpdate&var-job=menteeOverview [11:28:52] UpdateDataForMentor&var-job=newUserMessageJob&var-job=newcomerTasksCacheRefreshJob&var-job=null&var-job=pageFormsCreatePage&var-job=pageSchemasCreatePage&var-job=reassignMenteesJob&var-job=recentChangesUpdate&var-job=refreshLinksDynamic&var-job=refreshLinksPrioritized&var-job=renameUser&var-job=revertedTagUpdate&var-job=sendMail&var-job=setUserMentorDatabaseJob&var-job=smw.changePropagationClassUpdate&var-job=smw.changePropagation [11:28:57] Dispatch&var-job=smw.changePropagationUpdate&var-job=smw.deferredConstraintCheckUpdateJob&var-job=smw.elasticFileIngest&var-job=smw.elasticIndexerRecovery&var-job=smw.entityIdDisposer&var-job=smw.fulltextSearchTableRebuild&var-job=smw.fulltextSearchTableUpdate&var-job=smw.parserCachePurgeJob&var-job=smw.propertyStatisticsRebuild&var-job=smw.refresh&var-job=smw.update&var-job=smw.updateDispatcher&var-job=updateBetaFeaturesUserCount [11:29:02] s&var-job=userEditCountInit&var-job=userGroupExpiry&var-job=userOptionsUpdate&var-job=watchlistExpiry&var-job=webVideoTranscode&var-job=webVideoTranscodePrioritized&var-job=wikibase-InjectRCRecords&var-job=wikibase-addUsagesForPage [11:29:06] oops, IRC doesn't like that URL much huh [11:29:39] https://grafana.miraheze.org/d/GtxbP1Xnk/mediawiki?orgId=1&from=now-12h&to=now&var-node=jobchron121&var-job=SMW%5CChangePropagationUpdateJob&var-job=smw.changePropagationUpdate [11:29:42] much better [11:30:07] hmm [11:39:57] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07882192731 secs [11:44:52] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.62, 2.41, 1.62 [11:48:07] PROBLEM - cp22 APT on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [11:50:04] RECOVERY - cp22 APT on cp22 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [11:50:52] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.52, 2.82, 2.19 [11:53:57] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset -0.1168517768 secs [11:55:58] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset -0.0863019526 secs [11:57:05] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.46, 2.93, 2.57 [11:59:05] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.79, 2.55, 2.48 [11:59:21] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.42, 2.99, 2.06 [12:01:19] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.41, 2.40, 1.96 [12:24:17] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.43, 6.17, 3.44 [12:25:58] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1216821373 secs [12:26:09] PROBLEM - wiki.geoparkcorumbatai.com.br - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query geoparkcorumbatai.com.br. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:26:17] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.57, 4.83, 3.30 [12:29:01] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 27.66, 11.04, 4.93 [12:30:04] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.07005962729 secs [12:33:57] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.76, 3.14, 2.67 [12:35:56] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.73, 2.90, 2.63 [12:42:49] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.32, 2.62, 3.84 [12:42:50] Orange_Star: refreshLinks gets stuck sometimes due to memory issues [12:43:06] Just runJobs on that wiki with max memory until it’s clear [12:46:46] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.34, 2.23, 3.39 [12:51:50] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [12:54:48] RECOVERY - wiki.geoparkcorumbatai.com.br - reverse DNS on sslhost is OK: SSL OK - wiki.geoparkcorumbatai.com.br reverse DNS resolves to cp22.miraheze.org - CNAME OK [13:00:32] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.80, 3.77, 3.33 [13:02:30] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.77, 2.90, 3.05 [13:05:25] [02miraheze/MirahezeMagic] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/66b9db8d1970...5a59909a890e [13:05:28] [02miraheze/MirahezeMagic] 07Reception123 035a59909 - temporary message regarding cloud11 for uploads [13:08:36] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/997a709c2aa0 [13:08:37] [02miraheze/mw-config] 07Reception123 03997a709 - cloud11 failiures/upload issues sitenotice for File: pages [13:08:39] [02mw-config] 07Reception123 created branch 03Reception123-patch-3 - 13https://github.com/miraheze/mw-config [13:08:43] [02mw-config] 07Reception123 opened pull request 03#5190: cloud11 failiures/upload issues sitenotice for File: pages - 13https://github.com/miraheze/mw-config/pull/5190 [13:09:51] miraheze/mw-config - Reception123 the build passed. [13:10:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [13:12:23] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.64, 11.19, 9.38 [13:13:02] miraheze/MirahezeMagic - Reception123 the build passed. [13:14:22] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.19, 10.71, 9.40 [13:14:32] looks like the jobs are getting done very slowly, it has slowed down since 13:48 UTC+2 [13:14:48] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.65, 3.44, 3.00 [13:15:45] yeah refreshLinks can be like that... [13:16:11] [02mw-config] 07Reception123 synchronize pull request 03#5190: cloud11 failiures/upload issues sitenotice for File: pages - 13https://github.com/miraheze/mw-config/pull/5190 [13:16:13] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-3 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/997a709c2aa0...0f08cf1a2d9c [13:16:15] [02miraheze/mw-config] 07Reception123 030f08cf1 - Update Sitenotice.php [13:16:20] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.28, 9.84, 9.24 [13:16:47] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.34, 3.67, 3.13 [13:16:49] [02miraheze/mediawiki] 07Reception123 pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/9d6baedf2911...7e26f1ff0680 [13:16:51] [02miraheze/mediawiki] 07Reception123 037e26f1f - Update MirahezeMagic [13:17:14] miraheze/mw-config - Reception123 the build passed. [13:17:14] !log [reception@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'world': True, 'l10n': True} to all [13:17:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:17:20] !log [reception@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [13:17:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:17:29] [02mw-config] 07Reception123 closed pull request 03#5190: cloud11 failiures/upload issues sitenotice for File: pages - 13https://github.com/miraheze/mw-config/pull/5190 [13:17:30] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/421e3e47cc14...e846fcec3e89 [13:17:30] !log [reception@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'world': True, 'l10n': True} to all [13:17:31] [02miraheze/mw-config] 07Reception123 03e846fce - cloud11 failiures/upload issues sitenotice for File: pages (#5190) [13:17:34] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-3 [13:17:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:17:36] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-3 - 13https://github.com/miraheze/mw-config [13:18:28] miraheze/mw-config - Reception123 the build passed. [13:18:47] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.30, 3.52, 3.14 [13:20:40] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [13:22:14] !log [reception@mwtask141] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [13:22:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:22:46] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.73, 3.30, 3.15 [13:24:40] !log [reception@mwtask141] starting deploy of {'pull': 'config', 'config': True, 'l10n': True, 'folders': 'w/extensions/MirahezeMagic'} to all [13:24:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:27:12] !log [@test131] starting deploy of {'config': True} to all [13:27:13] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [13:27:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:27:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:28:05] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.16, 3.11, 2.30 [13:31:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.12, 10.29, 9.73 [13:32:25] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:34:25] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [13:35:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.40, 11.05, 10.16 [13:35:59] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.02, 3.99, 3.44 [13:36:43] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.73, 3.85, 3.48 [13:37:03] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.96, 10.96, 10.26 [13:37:57] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.47, 3.39, 3.28 [13:38:43] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.36, 3.54, 3.40 [13:39:02] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.43, 9.76, 9.88 [13:42:42] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.03, 3.30, 3.35 [13:48:07] PROBLEM - cp22 APT on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [13:49:41] !log [reception@mwtask141] starting deploy of {'l10n': True, 'folders': 'w/extensions/MirahezeMagic', 'ignoretime': True} to all [13:49:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:50:03] RECOVERY - cp22 APT on cp22 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [13:55:24] !log [reception@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [13:55:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:55:40] !log [reception@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 15s [13:55:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:58:46] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.95, 11.62, 10.60 [13:59:47] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.22, 3.63, 2.72 [14:02:13] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:02:47] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.96, 6.54, 4.90 [14:03:37] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.57, 3.71, 3.44 [14:03:46] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.92, 3.54, 2.92 [14:04:42] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.42, 11.54, 10.83 [14:04:46] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.88, 7.28, 5.36 [14:04:52] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [14:05:36] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.89, 3.33, 3.32 [14:05:46] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.06, 3.13, 2.85 [14:06:19] PROBLEM - es141 Current Load on es141 is CRITICAL: CRITICAL - load average: 4.16, 3.12, 2.14 [14:06:46] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 6.79, 7.18, 5.57 [14:07:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.02, 11.02, 9.33 [14:08:18] RECOVERY - es141 Current Load on es141 is OK: OK - load average: 2.35, 2.85, 2.16 [14:08:45] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.71, 5.67, 5.23 [14:09:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.94, 11.34, 9.66 [14:11:35] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.84, 3.82, 3.53 [14:15:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.05, 10.17, 9.70 [14:16:07] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 18.50, 6.83, 3.40 [14:16:34] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.91, 11.23, 11.53 [14:18:15] PROBLEM - cp22 APT on cp22 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:19:34] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.33, 3.87, 3.63 [14:20:11] RECOVERY - cp22 APT on cp22 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [14:21:33] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.82, 3.81, 3.64 [14:21:50] PROBLEM - en.religiononfire.mar.in.ua - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query mar.in.ua. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [14:24:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.98, 9.00, 10.17 [14:24:38] PROBLEM - db142 Current Load on db142 is WARNING: WARNING - load average: 7.75, 7.08, 5.98 [14:25:33] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.30, 3.05, 3.37 [14:26:37] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 9.14, 7.25, 6.14 [14:27:47] !log [reception@mwtask141] finished deploy of {'l10n': True, 'folders': 'w/extensions/MirahezeMagic', 'ignoretime': True} to all - SUCCESS in 2280s [14:27:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:29:47] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.58, 3.80, 2.64 [14:29:53] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 0.64, 2.08, 3.65 [14:30:36] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 2.51, 6.20, 6.10 [14:31:51] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.24, 1.79, 3.36 [14:32:13] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [14:32:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.68, 10.33, 10.18 [14:34:21] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.52, 9.25, 9.80 [14:38:30] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.93, 3.52, 3.36 [14:40:10] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [14:41:46] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.90, 3.54, 3.58 [14:43:47] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.54, 2.95, 3.36 [14:44:13] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.10, 10.35, 9.93 [14:46:12] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.09, 9.92, 9.84 [14:48:28] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.47, 3.92, 3.58 [14:50:27] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.52, 3.83, 3.59 [14:50:40] RECOVERY - en.religiononfire.mar.in.ua - reverse DNS on sslhost is OK: SSL OK - en.religiononfire.mar.in.ua reverse DNS resolves to cp23.miraheze.org - CNAME OK [14:57:47] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 10.38, 5.07, 3.51 [15:00:25] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.23, 3.83, 3.67 [15:01:18] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [15:02:25] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.25, 3.60, 3.60 [15:03:19] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [15:03:46] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.95, 3.98, 3.69 [15:05:47] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.32, 3.02, 3.37 [15:12:23] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.10, 3.69, 3.61 [15:13:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.02, 10.98, 9.87 [15:17:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.31, 11.62, 10.41 [15:18:22] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.72, 3.86, 3.73 [15:20:22] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.20, 3.95, 3.77 [15:21:47] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.47, 10.09, 10.07 [15:22:21] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.39, 3.74, 3.72 [15:24:21] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.70, 4.17, 3.88 [15:25:43] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.17, 11.16, 10.51 [15:27:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.31, 10.37, 10.30 [15:28:20] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.51, 3.89, 3.84 [15:29:52] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 15.20, 12.91, 6.77 [15:37:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.32, 9.76, 10.14 [15:40:18] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.61, 3.74, 3.67 [15:41:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.34, 11.69, 10.79 [15:42:18] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.19, 3.44, 3.56 [15:43:35] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.48, 11.18, 10.72 [15:43:47] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 0.58, 1.89, 3.68 [15:45:46] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.84, 1.53, 3.32 [15:51:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.65, 9.41, 10.09 [15:52:18] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.13, 2.94, 3.32 [15:57:35] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.11, 10.79, 10.38 [15:59:35] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.10, 9.31, 9.89 [15:59:47] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 9.56, 5.60, 3.75 [16:01:14] PROBLEM - cp23 APT on cp23 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [16:02:18] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 5.19, 3.86, 3.53 [16:03:15] RECOVERY - cp23 APT on cp23 is OK: APT OK: 2 packages available for upgrade (0 critical updates). [16:03:46] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 1.25, 3.51, 3.36 [16:05:46] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.29, 2.75, 3.09 [16:07:31] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.68, 9.34, 8.18 [16:08:35] !log puppet141: upgrade puppet-agent puppetdb puppetdb-termini puppetserver tzdata [16:08:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:09:31]