[00:05:40] <icinga-miraheze>	 PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 320 failures. Last run 5 minutes ago with 320 failures. Failed resources (up to 3 shown): File[olwest.org],File[olwest.org_private],File[wiki.umbranet.mandelbrotgames.com],File[wiki.umbranet.mandelbrotgames.com_private]
[00:08:28] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.14, 9.79, 8.62
[00:08:42] <Not-9244>	 [02puppet] 07Universal-Omega opened pull request 03#2364: Add webVideoTranscode to high intensity jobs - 13https://github.com/miraheze/puppet/pull/2364
[00:09:22] <Not-9244>	 [02puppet] 07Universal-Omega synchronize pull request 03#2364: Add webVideoTranscode to high intensity jobs - 13https://github.com/miraheze/puppet/pull/2364
[00:10:20] <Not-9244>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/a5627ae5149b
[00:10:21] <Not-9244>	 [02miraheze/mw-config] 07Universal-Omega 03a5627ae - Modify TimedMediaHandler configs
[00:10:23] <Not-9244>	 [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config
[00:10:24] <Not-9244>	 [02mw-config] 07Universal-Omega opened pull request 03#4408: Modify TimedMediaHandler configs - 13https://github.com/miraheze/mw-config/pull/4408
[00:11:29] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[00:13:16] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.44, 8.85, 7.98
[00:19:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[00:19:49] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.27, 9.74, 9.31
[00:20:06] <Not-9244>	 [02mw-config] 07Universal-Omega closed pull request 03#4408: Modify TimedMediaHandler configs - 13https://github.com/miraheze/mw-config/pull/4408
[00:20:08] <Not-9244>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/92decb334527...6f0d17f744ff
[00:20:09] <Not-9244>	 [02miraheze/mw-config] 07Universal-Omega 036f0d17f - Modify TimedMediaHandler configs (#4408)
[00:20:11] <Not-9244>	 [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config
[00:20:12] <Not-9244>	 [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8
[00:21:12] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[00:21:33] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.29, 8.73, 9.99
[00:21:36] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.63, 9.01, 8.18
[00:22:01] <MirahezeLSBot>	 !log [@mwtask111] starting deploy of {'config': True} to scsvg
[00:22:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:22:11] <MirahezeLSBot>	 !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 9s
[00:22:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:22:43] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.71, 10.71, 9.77
[00:24:00] <Not-9244>	 [02ManageWiki] 07Universal-Omega closed pull request 03#324: Use SidebarBeforeOutput hook - 13https://github.com/miraheze/ManageWiki/pull/324
[00:24:01] <Not-9244>	 [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/b9bb66b04e63...be249c927463
[00:24:03] <Not-9244>	 [02miraheze/ManageWiki] 07Universal-Omega 03be249c9 - Use SidebarBeforeOutput hook (#324)
[00:24:04] <Not-9244>	 [02miraheze/ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1
[00:24:06] <Not-9244>	 [02ManageWiki] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/ManageWiki
[00:24:07] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.53, 4.03, 3.49
[00:24:29] <Not-9244>	 [02mediawiki] 07Universal-Omega commented on pull request 03#4963: Bump extensions/ManageWiki from `dc6542e` to `a5a55c2` - 13https://github.com/miraheze/mediawiki/pull/4963#issuecomment-1015956037
[00:24:30] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.44, 9.59, 8.80
[00:24:32] <Not-9244>	 [02mediawiki] 07dependabot[bot] edited pull request 03#4963: Bump extensions/ManageWiki from `dc6542e` to `a5a55c2` - 13https://github.com/miraheze/mediawiki/pull/4963
[00:24:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[00:25:33] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.39, 9.95, 9.66
[00:25:36] <Not-9244>	 [02mediawiki] 07dependabot[bot] edited pull request 03#4963: Bump extensions/ManageWiki from `dc6542e` to `a5a55c2` - 13https://github.com/miraheze/mediawiki/pull/4963
[00:25:41] <Not-9244>	 [02mediawiki] 07dependabot[bot] opened pull request 03#5021: Bump extensions/ManageWiki from `dc6542e` to `be249c9` - 13https://github.com/miraheze/mediawiki/pull/5021
[00:25:42] <Not-9244>	 [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/ManageWiki-be249c9 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/ae4fa00164a2
[00:25:44] <Not-9244>	 [02miraheze/mediawiki] 07dependabot[bot] 03ae4fa00 - Bump extensions/ManageWiki from `dc6542e` to `be249c9`
[00:25:45] <Not-9244>	 [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/ManageWiki-be249c9 - 13https://github.com/miraheze/mediawiki
[00:25:47] <Not-9244>	 [02mediawiki] 07dependabot[bot] commented on pull request 03#4963: Bump extensions/ManageWiki from `dc6542e` to `a5a55c2` - 13https://github.com/miraheze/mediawiki/pull/4963#issuecomment-1015956664
[00:25:48] <Not-9244>	 [02mediawiki] 07dependabot[bot] closed pull request 03#4963: Bump extensions/ManageWiki from `dc6542e` to `a5a55c2` - 13https://github.com/miraheze/mediawiki/pull/4963
[00:25:50] <Not-9244>	 [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/ManageWiki-a5a55c2 - 13https://github.com/miraheze/mediawiki
[00:25:51] <Not-9244>	 [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/ManageWiki-a5a55c2
[00:26:06] <Not-9244>	 [02mediawiki] 07github-actions[bot] labeled pull request 03#5021: Bump extensions/ManageWiki from `dc6542e` to `be249c9` - 13https://github.com/miraheze/mediawiki/pull/5021
[00:26:08] <Not-9244>	 [02mediawiki] 07github-actions[bot] labeled pull request 03#5021: Bump extensions/ManageWiki from `dc6542e` to `be249c9` - 13https://github.com/miraheze/mediawiki/pull/5021
[00:26:59] <icinga-miraheze>	 RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.79, 3.11, 3.22
[00:27:02] <Not-9244>	 [02mediawiki] 07Universal-Omega closed pull request 03#5021: Bump extensions/ManageWiki from `dc6542e` to `be249c9` - 13https://github.com/miraheze/mediawiki/pull/5021
[00:27:03] <Not-9244>	 [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/b57ada3bd9e8...1355b5ffea8d
[00:27:05] <Not-9244>	 [02miraheze/mediawiki] 07dependabot[bot] 031355b5f - Bump extensions/ManageWiki from `dc6542e` to `be249c9` (#5021)
[00:27:06] <Not-9244>	 [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ManageWiki-be249c9 - 13https://github.com/miraheze/mediawiki
[00:27:08] <Not-9244>	 [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/ManageWiki-be249c9
[00:27:17] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.06, 8.45, 8.24
[00:27:18] <Not-9244>	 [02mediawiki] 07Universal-Omega closed pull request 03#5000: Bump extensions/Gadgets from `4979fe1` to `f958b46` - 13https://github.com/miraheze/mediawiki/pull/5000
[00:27:19] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.94, 8.83, 8.66
[00:27:19] <Not-9244>	 [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Gadgets-f958b46
[00:27:21] <Not-9244>	 [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/1355b5ffea8d...862f0ce57687
[00:27:22] <Not-9244>	 [02miraheze/mediawiki] 07dependabot[bot] 03862f0ce - Bump extensions/Gadgets from `4979fe1` to `f958b46` (#5000)
[00:27:24] <Not-9244>	 [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/Gadgets-f958b46 - 13https://github.com/miraheze/mediawiki
[00:30:04] <icinga-miraheze>	 RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.96, 7.86, 8.34
[00:31:03] <icinga-miraheze>	 RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures
[00:31:11] <miraheze-github>	 miraheze/ManageWiki - Universal-Omega the build passed.
[00:32:49] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.45, 9.08, 9.38
[00:35:39] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.22, 8.10, 8.99
[00:37:34] <Not-9244>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/eecb54d02364
[00:37:36] <Not-9244>	 [02miraheze/mw-config] 07Universal-Omega 03eecb54d - Set `$wgPreprocessorCacheThreshold` to 1000 (default)
[00:37:37] <Not-9244>	 [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config
[00:37:39] <Not-9244>	 [02mw-config] 07Universal-Omega opened pull request 03#4409: Set `$wgPreprocessorCacheThreshold` to 1000 (default) - 13https://github.com/miraheze/mw-config/pull/4409
[00:38:40] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[00:38:50] <Not-9244>	 [02mw-config] 07Universal-Omega closed pull request 03#4409: Set `$wgPreprocessorCacheThreshold` to 1000 (default) - 13https://github.com/miraheze/mw-config/pull/4409
[00:38:51] <Not-9244>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6f0d17f744ff...f356980b417a
[00:38:53] <Not-9244>	 [02miraheze/mw-config] 07Universal-Omega 03f356980 - Set `$wgPreprocessorCacheThreshold` to 1000 (default) (#4409)
[00:38:54] <Not-9244>	 [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config
[00:38:56] <Not-9244>	 [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8
[00:39:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[00:39:56] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[00:43:54] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.11, 8.45, 8.23
[00:43:59] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.27, 9.39, 9.07
[00:46:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[00:46:41] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.36, 7.75, 7.99
[00:46:44] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.47, 8.62, 8.84
[00:46:46] <Not-9244>	 [02puppet] 07Universal-Omega opened pull request 03#2365: deploy-mediawiki: always use http proxy - 13https://github.com/miraheze/puppet/pull/2365
[00:47:30] <Not-9244>	 [02puppet] 07Universal-Omega synchronize pull request 03#2365: deploy-mediawiki: always use http proxy - 13https://github.com/miraheze/puppet/pull/2365
[00:48:18] <MirahezeLSBot>	 !log [@test101] starting deploy of {'config': True} to skip
[00:48:19] <MirahezeLSBot>	 !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s
[00:48:29] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:48:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:50:46] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.94, 9.45, 9.18
[00:52:38] <MirahezeLSBot>	 !log [@mwtask111] starting deploy of {'config': True} to scsvg
[00:52:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:52:46] <MirahezeLSBot>	 !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 7s
[00:52:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[00:55:04] <icinga-miraheze>	 RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.50, 7.87, 8.43
[00:56:09] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.32, 9.59, 9.33
[00:56:36] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.13, 8.92, 8.39
[00:57:24] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.33, 8.69, 7.83
[00:59:17] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.76, 10.13, 9.52
[01:00:51] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 19.15, 12.58, 9.45
[01:03:00] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 22.47, 16.63, 11.98
[01:04:09] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 9.83, 12.77, 10.51
[01:04:18] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb
[01:04:38] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.62, 14.48, 10.76
[01:06:58] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 5.99, 9.59, 9.65
[01:07:06] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[01:11:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[01:11:56] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.85, 8.68, 9.81
[01:13:03] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.22, 9.58, 9.82
[01:15:17] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.38, 6.53, 8.13
[01:15:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[01:17:06] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.19, 8.69, 9.92
[01:17:07] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 5.83, 7.61, 9.71
[01:17:18] <Not-9244>	 [02puppet] 07paladox closed pull request 03#2363: Add webVideoTranscodePrioritized to jobQueueCollector - 13https://github.com/miraheze/puppet/pull/2363
[01:17:20] <Not-9244>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b2bb749b5d80...d7c2effc0af6
[01:17:21] <Not-9244>	 [02miraheze/puppet] 07Universal-Omega 03d7c2eff - Add webVideoTranscodePrioritized to jobQueueCollector (#2363)
[01:17:46] <Not-9244>	 [02puppet] 07paladox closed pull request 03#2364: Add webVideoTranscode to high intensity jobs - 13https://github.com/miraheze/puppet/pull/2364
[01:17:47] <Not-9244>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/d7c2effc0af6...df772d0a74ac
[01:17:49] <Not-9244>	 [02miraheze/puppet] 07Universal-Omega 03df772d0 - Add webVideoTranscode to high intensity jobs (#2364)
[01:19:52] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.71, 9.74, 10.12
[01:21:44] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.48, 8.25, 8.40
[01:22:38] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.98, 9.39, 9.89
[01:22:55] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.06, 8.99, 9.27
[01:24:34] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.06, 6.97, 7.89
[01:25:42] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.06, 8.47, 9.04
[01:26:57] <icinga-miraheze>	 RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.41, 7.35, 8.39
[01:28:18] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.11, 9.43, 9.72
[01:28:33] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.89, 9.01, 9.11
[01:31:13] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.57, 9.24, 9.20
[01:33:44] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.19, 9.17, 8.95
[01:33:57] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 15.18, 11.68, 10.13
[01:36:37] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.34, 8.99, 9.74
[01:36:39] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.28, 8.45, 8.73
[01:39:38] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.16, 8.60, 8.15
[01:40:03] <icinga-miraheze>	 PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/u
[01:40:03] <icinga-miraheze>	 ython3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wikimicrofinanza.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[01:42:28] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.25, 9.13, 8.92
[01:42:37] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.22, 8.10, 8.01
[01:45:22] <icinga-miraheze>	 RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.00, 7.21, 8.48
[01:45:24] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.95, 9.00, 8.87
[01:48:02] <icinga-miraheze>	 PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.16, 1.80, 1.14
[01:48:29] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.46, 10.26, 9.37
[01:52:04] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb
[01:52:49] <Not-9244>	 [02puppet] 07Universal-Omega opened pull request 03#2366: Revert "Revert "absent manageInactiveWikis cron"" - 13https://github.com/miraheze/puppet/pull/2366
[01:53:10] <CosmicAlpha>	 paladox: if still around can you please merge ^
[01:53:32] <Not-9244>	 [02puppet] 07paladox closed pull request 03#2366: Revert "Revert "absent manageInactiveWikis cron"" - 13https://github.com/miraheze/puppet/pull/2366
[01:53:33] <Not-9244>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/df772d0a74ac...a33ba8600605
[01:53:35] <Not-9244>	 [02miraheze/puppet] 07Universal-Omega 03a33ba86 - Revert "Revert "absent manageInactiveWikis cron (#2348)" (#2352)" (#2366)
[01:53:50] <CosmicAlpha>	 Thank you! It seems we still have an issue somewhere.
[01:55:00] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[02:03:03] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 6.95, 7.42, 9.58
[02:03:59] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 5.32, 7.26, 9.90
[02:05:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[02:07:15] <icinga-miraheze>	 PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is CRITICAL: connect to address www.wikimicrofinanza.it and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket
[02:10:24] <icinga-miraheze>	 RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp20.miraheze.org - CNAME OK
[02:11:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[02:11:53] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.89, 9.06, 8.61
[02:18:02] <icinga-miraheze>	 RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 5.74, 6.69, 8.24
[02:23:13] <icinga-miraheze>	 RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.56, 8.04, 8.40
[02:25:35] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.05, 9.15, 8.92
[02:26:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[02:28:26] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.30, 8.95, 8.87
[02:29:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[02:29:47] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.70, 9.44, 8.94
[02:33:55] <icinga-miraheze>	 RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.73, 7.97, 8.47
[02:34:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[02:35:13] <icinga-miraheze>	 RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.62, 8.04, 8.43
[02:37:24] <icinga-miraheze>	 PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.wikimicrofinanza.it' expires in 12 day(s) (Mon 31 Jan 2022 22:54:41 GMT +0000).
[02:38:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[02:47:56] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.76, 9.51, 8.96
[02:50:43] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.78, 9.64, 9.12
[02:50:44] <icinga-miraheze>	 PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.67, 1.90, 2.00
[02:50:57] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.21, 9.12, 9.86
[02:56:13] <icinga-miraheze>	 RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.04, 0.83, 1.52
[03:00:15] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.52, 9.73, 8.58
[03:00:15] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.39, 8.58, 8.11
[03:03:07] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.95, 9.14, 8.48
[03:04:20] <icinga-miraheze>	 RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.98, 7.66, 8.33
[03:04:42] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.40, 8.81, 8.98
[03:05:41] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.53, 9.38, 8.79
[03:05:44] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.53, 8.05, 7.99
[03:05:50] <icinga-miraheze>	 RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.71, 8.45, 8.32
[03:06:17] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.21, 2.96
[03:07:25] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.85, 8.11, 8.67
[03:08:28] <icinga-miraheze>	 RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.67, 8.29, 8.47
[03:08:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[03:09:02] <icinga-miraheze>	 RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.26, 2.94, 2.91
[03:13:00] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.46, 9.68, 9.11
[03:15:47] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.92, 9.58, 9.16
[03:16:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[03:16:39] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.57, 9.31, 8.66
[03:17:55] <Not-9244>	 [02puppet] 07Universal-Omega opened pull request 03#2367: Remove support for ufw - 13https://github.com/miraheze/puppet/pull/2367
[03:18:35] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 9.83, 10.21, 9.50
[03:19:30] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.16, 8.65, 8.51
[03:19:41] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.78, 9.01, 8.60
[03:20:59] <Not-9244>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f356980b417a...30a97635860b
[03:21:00] <Not-9244>	 [02miraheze/mw-config] 07Universal-Omega 0330a9763 - Remove `$wgEnableNiceBackgroundTranscodeJobs`
[03:21:35] <MirahezeLSBot>	 !log [@mwtask111] starting deploy of {'config': True} to scsvg
[03:21:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:21:45] <MirahezeLSBot>	 !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 10s
[03:21:52] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:22:36] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.89, 7.71, 8.18
[03:22:46] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[03:23:10] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.31, 9.16, 8.36
[03:26:00] <icinga-miraheze>	 RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.75, 8.38, 8.19
[03:26:33] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.79, 8.54, 8.16
[03:28:01] <icinga-miraheze>	 RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.99, 8.10, 8.41
[03:32:06] <icinga-miraheze>	 RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.43, 8.35, 8.21
[03:32:29] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.45, 8.83, 8.58
[03:32:33] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.21, 9.44, 9.68
[03:35:10] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.08, 10.20, 9.16
[03:36:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[03:37:56] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.75, 8.95, 8.84
[03:41:18] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.10, 8.78, 8.27
[03:44:01] <icinga-miraheze>	 RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.02, 8.21, 8.11
[03:47:16] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.76, 8.50, 8.25
[03:47:31] <MirahezeLSBot>	 !log [@test101] starting deploy of {'config': True} to skip
[03:47:32] <MirahezeLSBot>	 !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s
[03:47:35] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:47:40] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:48:58] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.11, 8.72, 8.67
[03:51:46] <icinga-miraheze>	 RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.50, 8.05, 8.41
[03:52:53] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.72, 7.94, 8.08
[03:55:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[03:56:32] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.29, 9.50, 8.67
[03:58:06] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.92, 9.72, 8.97
[03:59:18] <icinga-miraheze>	 RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.40, 8.42, 8.41
[04:00:54] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.93, 9.23, 8.89
[04:02:06] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.51, 9.40, 8.67
[04:03:37] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.36, 9.64, 9.09
[04:04:52] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.84, 8.29, 8.36
[04:05:41] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.75, 9.53, 9.17
[04:06:17] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.12, 8.77, 8.83
[04:08:21] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.61, 8.98, 9.01
[04:08:59] <icinga-miraheze>	 RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.24, 7.77, 8.42
[04:10:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[04:13:58] <icinga-miraheze>	 RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 5.89, 7.66, 8.50
[04:14:57] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.35, 10.35, 8.86
[04:15:32] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 16.85, 12.40, 10.16
[04:15:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[04:17:59] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.81, 9.06, 8.63
[04:19:52] <icinga-miraheze>	 PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[04:20:31] <icinga-miraheze>	 PROBLEM - cp30 Stunnel Http for mw122 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[04:20:52] <icinga-miraheze>	 PROBLEM - cp20 Stunnel Http for mw122 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[04:21:02] <icinga-miraheze>	 RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.53, 7.92, 8.26
[04:21:31] <icinga-miraheze>	 PROBLEM - cp21 Stunnel Http for mw122 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[04:22:54] <icinga-miraheze>	 RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 4.056 second response time
[04:23:29] <icinga-miraheze>	 RECOVERY - cp30 Stunnel Http for mw122 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.939 second response time
[04:23:47] <icinga-miraheze>	 RECOVERY - cp20 Stunnel Http for mw122 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 4.243 second response time
[04:24:18] <icinga-miraheze>	 RECOVERY - cp21 Stunnel Http for mw122 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.015 second response time
[04:27:25] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.43, 9.50, 8.77
[04:30:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[04:35:33] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 6.74, 8.75, 8.82
[04:38:18] <icinga-miraheze>	 RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 5.40, 7.25, 8.23
[04:43:32] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 5.62, 7.20, 9.90
[04:44:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[04:49:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[04:54:25] <icinga-miraheze>	 RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.95, 7.33, 8.50
[05:02:00] <icinga-miraheze>	 PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.58, 1.75, 1.53
[05:04:39] <icinga-miraheze>	 RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.57, 1.19, 1.34
[05:07:00] <Not-9244>	 [02puppet] 07Universal-Omega commented on pull request 03#2367: Remove support for ufw - 13https://github.com/miraheze/puppet/pull/2367#issuecomment-1016086095
[05:08:10] <Not-9244>	 [02puppet] 07Universal-Omega deleted a comment on pull request 03#2367: Remove support for ufw - 13https://github.com/miraheze/puppet/pull/2367#issuecomment-1016086095
[05:13:20] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 19.23, 13.70, 9.81
[05:16:31] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.52, 9.32, 8.15
[05:16:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[05:24:56] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.14, 9.50, 9.04
[05:28:28] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.42, 11.08, 8.94
[05:31:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[05:37:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[05:38:33] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 6.65, 7.93, 9.97
[05:39:14] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.81, 9.01, 9.37
[05:42:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[05:44:01] <icinga-miraheze>	 RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.40, 7.65, 8.26
[05:49:29] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.08, 6.96, 8.40
[05:50:13] <icinga-miraheze>	 RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.76, 6.88, 8.17
[05:58:34] <Not-9244>	 [02mediawiki] 07lens0021 opened pull request 03#5022: Install PropertySuggester - 13https://github.com/miraheze/mediawiki/pull/5022
[06:01:58] <Not-9244>	 [02mediawiki] 07Universal-Omega closed pull request 03#5022: Install PropertySuggester - 13https://github.com/miraheze/mediawiki/pull/5022
[06:01:59] <Not-9244>	 [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+1/-0/±1] 13https://github.com/miraheze/mediawiki/compare/862f0ce57687...5d8432cad7b9
[06:02:01] <Not-9244>	 [02miraheze/mediawiki] 07lens0021 035d8432c - Install PropertySuggester (#5022)
[06:04:52] <Not-9244>	 [02mediawiki] 07lens0021 commented on pull request 03#5022: Install PropertySuggester - 13https://github.com/miraheze/mediawiki/pull/5022#issuecomment-1016114055
[06:07:22] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.70, 8.69, 8.42
[06:10:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[06:10:49] <Not-9244>	 [02mediawiki] 07Universal-Omega commented on pull request 03#5022: Install PropertySuggester - 13https://github.com/miraheze/mediawiki/pull/5022#issuecomment-1016116945
[06:10:53] <icinga-miraheze>	 PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.99, 1.73, 1.41
[06:11:11] <Not-9244>	 [02mediawiki] 07Universal-Omega edited a comment on pull request 03#5022: Install PropertySuggester - 13https://github.com/miraheze/mediawiki/pull/5022#issuecomment-1016116945
[06:11:25] <Not-9244>	 [02mediawiki] 07Universal-Omega edited a comment on pull request 03#5022: Install PropertySuggester - 13https://github.com/miraheze/mediawiki/pull/5022#issuecomment-1016116945
[06:12:48] <icinga-miraheze>	 RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.42, 7.99, 8.29
[06:15:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[06:16:54] <icinga-miraheze>	 PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.10, 2.00, 1.64
[06:19:16] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.98, 8.98, 8.18
[06:19:27] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.65, 9.83, 8.51
[06:19:55] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 9.26, 10.41, 9.44
[06:21:58] <icinga-miraheze>	 RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.77, 7.43, 7.71
[06:22:07] <icinga-miraheze>	 RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.18, 8.11, 8.05
[06:22:36] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 6.91, 8.70, 8.93
[06:27:14] <Not-9244>	 [02mediawiki] 07lens0021 opened pull request 03#5023: Labeler: run only when the actor is dependabot - 13https://github.com/miraheze/mediawiki/pull/5023
[06:28:37] <Not-9244>	 [02mediawiki] 07lens0021 edited pull request 03#5023: Labeler: run only when the actor is dependabot - 13https://github.com/miraheze/mediawiki/pull/5023
[06:30:52] <icinga-miraheze>	 RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.56, 7.57, 8.32
[06:35:35] <Not-9244>	 [02mediawiki] 07Universal-Omega commented on pull request 03#5023: Labeler: run only when the actor is dependabot - 13https://github.com/miraheze/mediawiki/pull/5023#issuecomment-1016129270
[06:35:45] <Not-9244>	 [02mediawiki] 07Universal-Omega edited a comment on pull request 03#5023: Labeler: run only when the actor is dependabot - 13https://github.com/miraheze/mediawiki/pull/5023#issuecomment-1016129270
[07:08:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[07:13:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[07:20:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[07:22:28] <icinga-miraheze>	 PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.71, 1.89, 1.99
[07:31:01] <icinga-miraheze>	 RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.01, 1.26, 1.64
[07:35:06] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.71, 3.72, 3.27
[07:36:34] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.39, 8.56, 8.09
[07:37:49] <icinga-miraheze>	 RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.70, 2.98, 3.07
[07:39:19] <icinga-miraheze>	 RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.46, 8.31, 8.08
[07:40:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[07:43:37] <icinga-miraheze>	 PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2]
[07:43:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[07:48:08] <Not-9244>	 [02mw-config] 07lens0021 opened pull request 03#4410: Set PropertySuggesterInitialSuggestions for horimiyawiki - 13https://github.com/miraheze/mw-config/pull/4410
[07:48:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[07:49:12] <miraheze-github>	 miraheze/mw-config - lens0021 the build passed.
[07:49:54] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.35, 3.80, 3.59
[07:52:06] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.34, 10.27, 9.11
[07:54:51] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.40, 9.50, 9.02
[07:55:29] <icinga-miraheze>	 RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.99, 2.80, 3.23
[08:08:25] <icinga-miraheze>	 RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.46, 7.73, 8.39
[08:10:38] <icinga-miraheze>	 RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[08:15:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[08:20:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[08:32:40] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.70, 9.77, 8.05
[08:35:21] <icinga-miraheze>	 RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.05, 8.33, 7.76
[08:37:09] <icinga-miraheze>	 PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is CRITICAL: connect to address www.wikimicrofinanza.it and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket
[08:58:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[09:03:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[09:07:07] <icinga-miraheze>	 PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.wikimicrofinanza.it' expires in 12 day(s) (Mon 31 Jan 2022 22:54:41 GMT +0000).
[10:03:14] <icinga-miraheze>	 PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn
[10:03:14] <icinga-miraheze>	 AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.4051618576049805 seconds
[10:44:08] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.71, 8.09, 7.54
[10:45:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[10:46:47] <icinga-miraheze>	 RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.73, 7.54, 7.42
[10:50:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[11:02:24] <icinga-miraheze>	 RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp21.miraheze.org - CNAME OK
[11:21:44] <icinga-miraheze>	 PROBLEM - grc.repository.archiopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'grc.repository.archiopedia.org' expires in 15 day(s) (Fri 04 Feb 2022 10:56:12 GMT +0000).
[11:29:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[11:34:02] <Not-9244>	 [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a33ba8600605...9a5ce7ef7fea
[11:34:04] <Not-9244>	 [02miraheze/puppet] 07Reception123 039a5ce7e - remove wikiplus-app.com from CSP whitelist (per T7911)
[11:35:40] <icinga-miraheze>	 PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn
[11:35:41] <icinga-miraheze>	 AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.4098522663116455 seconds
[11:41:50] <Not-9244>	 [02mw-config] 07lens0021 opened pull request 03#4411: Add PropertySuggester to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4411
[11:42:52] <miraheze-github>	 miraheze/mw-config - lens0021 the build passed.
[11:44:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[11:54:54] <Not-9244>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/9d124e3c4b8b
[11:54:55] <Not-9244>	 [02miraheze/mw-config] 07Reception123 039d124e3 - set expat license on spnatiwiki per T8520
[11:54:57] <Not-9244>	 [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config
[11:54:58] <Not-9244>	 [02mw-config] 07Reception123 opened pull request 03#4412: set expat license on spnatiwiki per T8520 - 13https://github.com/miraheze/mw-config/pull/4412
[11:56:03] <miraheze-github>	 miraheze/mw-config - Reception123 the build passed.
[11:56:48] <Not-9244>	 [02mw-config] 07Reception123 closed pull request 03#4412: set expat license on spnatiwiki per T8520 - 13https://github.com/miraheze/mw-config/pull/4412
[11:56:50] <Not-9244>	 [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config
[11:56:51] <Not-9244>	 [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1
[11:56:52] <Not-9244>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/30a97635860b...a9498a84e881
[11:56:54] <Not-9244>	 [02miraheze/mw-config] 07Reception123 03a9498a8 - set expat license on spnatiwiki per T8520 (#4412)
[11:57:55] <miraheze-github>	 miraheze/mw-config - Reception123 the build passed.
[12:09:14] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.71, 9.27, 8.32
[12:09:47] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.01, 8.91, 8.04
[12:10:19] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.69, 8.73, 7.92
[12:10:40] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.21, 9.59, 8.45
[12:11:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[12:11:59] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.30, 8.93, 8.35
[12:12:32] <icinga-miraheze>	 RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.69, 8.43, 8.00
[12:13:02] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.12, 8.58, 7.97
[12:13:23] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.04, 8.75, 8.34
[12:15:34] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.99, 8.59, 8.41
[12:16:07] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.53, 8.32, 8.24
[12:17:34] <MirahezeLSBot>	 !log [@test101] starting deploy of {'config': True} to skip
[12:17:35] <MirahezeLSBot>	 !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s
[12:17:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:17:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:18:16] <icinga-miraheze>	 RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.18, 8.17, 8.26
[12:20:21] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.42, 10.77, 9.32
[12:21:25] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.65, 10.70, 9.16
[12:21:43] <MirahezeLSBot>	 !log [@mwtask111] starting deploy of {'config': True} to scsvg
[12:21:45] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.36, 10.87, 9.25
[12:21:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:21:52] <MirahezeLSBot>	 !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 8s
[12:21:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[12:22:47] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.34, 8.87, 8.57
[12:23:11] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.90, 9.56, 9.12
[12:24:09] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.93, 9.39, 8.90
[12:24:32] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.15, 9.52, 9.00
[12:24:51] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.55, 9.34, 8.92
[12:25:34] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.96, 8.46, 8.46
[12:28:47] <icinga-miraheze>	 RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.35, 7.87, 8.50
[12:29:46] <icinga-miraheze>	 RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.45, 7.87, 8.39
[12:30:03] <icinga-miraheze>	 RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.12, 7.89, 8.47
[12:31:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[12:37:06] <icinga-miraheze>	 PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is CRITICAL: connect to address www.wikimicrofinanza.it and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket
[12:38:26] <icinga-miraheze>	 RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.12, 7.66, 8.27
[12:41:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[12:42:12] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.48, 9.85, 8.99
[12:44:39] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.81, 9.86, 8.91
[12:45:02] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.55, 9.57, 8.95
[12:46:10] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.24, 9.12, 8.65
[12:46:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[12:46:47] <icinga-miraheze>	 PROBLEM - pj-masks-info.cf - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(h
[12:46:47] <icinga-miraheze>	  'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407600164413452 seconds
[12:47:06] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.63, 8.65, 8.41
[12:47:38] <icinga-miraheze>	 PROBLEM - lakehub.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/python
[12:47:38] <icinga-miraheze>	 ackages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.409571647644043 seconds
[12:47:39] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.40, 9.08, 8.78
[12:48:03] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.43, 9.43, 9.07
[12:48:04] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.29, 8.72, 8.71
[12:50:05] <icinga-miraheze>	 PROBLEM - wikidiffs.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn
[12:50:05] <icinga-miraheze>	 AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.4082934856414795 seconds
[12:50:05] <icinga-miraheze>	 RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.38, 8.26, 8.29
[12:50:14] <icinga-miraheze>	 PROBLEM - memipedia.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn
[12:50:14] <icinga-miraheze>	 AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406820774078369 seconds
[12:50:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[12:51:01] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.75, 9.58, 9.06
[12:51:51] <icinga-miraheze>	 PROBLEM - wiki.ripto.gq - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(host
[12:51:52] <icinga-miraheze>	 AAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407483100891113 seconds
[12:52:47] <icinga-miraheze>	 PROBLEM - miraheze.gq - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - miraheze.gq All nameservers failed to answer the query.
[12:53:59] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.83, 9.44, 9.11
[12:55:12] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.93, 8.32, 8.49
[12:56:21] <icinga-miraheze>	 PROBLEM - wiki.thedev.gq - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hos
[12:56:22] <icinga-miraheze>	 AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.414672374725342 seconds
[12:56:36] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.19, 9.95, 9.34
[12:58:30] <icinga-miraheze>	 PROBLEM - linkwiki.cf - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - linkwiki.cf All nameservers failed to answer the query.
[12:59:45] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.65, 9.99, 9.52
[13:00:10] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.23, 8.65, 8.93
[13:00:56] <icinga-miraheze>	 RECOVERY - linkwiki.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'linkwiki.cf' will expire on Mon 21 Feb 2022 17:45:52 GMT +0000.
[13:01:18] <icinga-miraheze>	 PROBLEM - otcg.ml - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostname, 
[13:01:18] <icinga-miraheze>	 0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.406448602676392 seconds
[13:02:27] <icinga-miraheze>	 PROBLEM - wikileague.cf - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(host
[13:02:28] <icinga-miraheze>	 AAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.4045445919036865 seconds
[13:05:18] <icinga-miraheze>	 PROBLEM - wiki.ct777.cf - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(host
[13:05:18] <icinga-miraheze>	 AAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404446840286255 seconds
[13:05:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[13:06:52] <icinga-miraheze>	 PROBLEM - gp.ct777.cf - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - gp.ct777.cf All nameservers failed to answer the query.
[13:07:27] <icinga-miraheze>	 PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.wikimicrofinanza.it' expires in 12 day(s) (Mon 31 Jan 2022 22:54:41 GMT +0000).
[13:07:56] <icinga-miraheze>	 PROBLEM - miraheze.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostna
[13:07:56] <icinga-miraheze>	 A')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404641151428223 seconds
[13:08:06] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 6.91, 8.60, 8.50
[13:08:09] <icinga-miraheze>	 PROBLEM - wiki-asterix.cf - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(ho
[13:08:09] <icinga-miraheze>	 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407358884811401 seconds
[13:08:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[13:08:44] <Reception123>	 !log DELETED and DROPPED vilemaximwiki
[13:08:48] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:09:06] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.59, 9.59, 9.79
[13:11:02] <icinga-miraheze>	 RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.16, 7.70, 8.15
[13:11:30] <icinga-miraheze>	 RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.09, 7.95, 8.48
[13:12:00] <icinga-miraheze>	 PROBLEM - data.memipedia.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(
[13:12:00] <icinga-miraheze>	 , 'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.410609245300293 seconds
[13:12:05] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.24, 10.26, 10.05
[13:13:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[13:13:47] <icinga-miraheze>	 PROBLEM - ircwiki.cf - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - ircwiki.cf All nameservers failed to answer the query.
[13:14:41] <icinga-miraheze>	 RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.90, 7.61, 8.43
[13:14:45] <icinga-miraheze>	 PROBLEM - miraheze.ml - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostna
[13:14:45] <icinga-miraheze>	 A')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404779434204102 seconds
[13:16:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[13:17:24] <icinga-miraheze>	 PROBLEM - pj-masks-info.cf - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - pj-masks-info.cf All nameservers failed to answer the query.
[13:21:20] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.37, 8.56, 8.60
[13:22:30] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.74, 8.79, 8.75
[13:23:27] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.39, 9.29, 9.78
[13:23:29] <icinga-miraheze>	 PROBLEM - miraheze.gq - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostna
[13:23:29] <icinga-miraheze>	 A')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407447099685669 seconds
[13:24:12] <icinga-miraheze>	 RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.22, 8.04, 8.39
[13:25:26] <icinga-miraheze>	 RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.37, 7.45, 8.22
[13:26:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[13:26:39] <icinga-miraheze>	 PROBLEM - wiki.thedev.gq - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.thedev.gq All nameservers failed to answer the query.
[13:27:11] <icinga-miraheze>	 PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/u
[13:27:11] <icinga-miraheze>	 ython3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wikimicrofinanza.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[13:27:37] <MirahezeLSBot>	 !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=estypemoonwiki --username-prefix wikia:typemoon --report 1 --no-updates /home/reception/estypemoon_pages_full.xml (START)
[13:27:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:28:16] <icinga-miraheze>	 PROBLEM - linkwiki.cf - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostna
[13:28:16] <icinga-miraheze>	 A')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.403559446334839 seconds
[13:34:46] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.92, 8.98, 9.02
[13:37:14] <icinga-miraheze>	 PROBLEM - gp.ct777.cf - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostna
[13:37:14] <icinga-miraheze>	 A')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.403799772262573 seconds
[13:37:34] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.12, 8.93, 9.01
[13:39:42] <RhinosF1>	 $log added reCaptcha to enterprise secret key
[13:40:06] <RhinosF1>	 !log added reCaptcha enterprise to enterprise secret key
[13:40:12] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:40:27] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.57, 10.16, 9.48
[13:42:37] <MirahezeLSBot>	 !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildImages.php --wiki=rctwiki --missing (END - exit=0)
[13:42:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[13:43:59] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.68, 9.92, 8.63
[13:44:32] <icinga-miraheze>	 PROBLEM - ircwiki.cf - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostnam
[13:44:32] <icinga-miraheze>	 ')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.409162521362305 seconds
[13:44:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[13:45:07] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.75, 8.80, 8.46
[13:45:13] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.83, 8.97, 8.17
[13:46:21] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.88, 9.17, 8.25
[13:47:56] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.54, 9.58, 8.81
[13:48:02] <icinga-miraheze>	 PROBLEM - pj-masks-info.cf - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(h
[13:48:02] <icinga-miraheze>	  'AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407277584075928 seconds
[13:49:31] <icinga-miraheze>	 PROBLEM - memipedia.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - memipedia.ga All nameservers failed to answer the query.
[13:49:49] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.57, 8.21, 8.25
[13:50:55] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.27, 8.91, 8.68
[13:57:05] <icinga-miraheze>	 PROBLEM - wiki.thedev.gq - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hos
[13:57:05] <icinga-miraheze>	 AAAA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.410568952560425 seconds
[13:57:39] <icinga-miraheze>	 RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp20.miraheze.org - CNAME OK
[13:58:06] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.94, 9.61, 9.32
[13:59:45] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.11, 9.47, 8.99
[14:02:38] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.44, 9.69, 9.18
[14:03:03] <icinga-miraheze>	 PROBLEM - wikileague.cf - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wikileague.cf All nameservers failed to answer the query.
[14:03:52] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.30, 10.01, 9.52
[14:05:27] <icinga-miraheze>	 PROBLEM - wiki.ct777.cf - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.ct777.cf All nameservers failed to answer the query.
[14:05:27] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.46, 10.11, 9.44
[14:05:36] <icinga-miraheze>	 RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.90, 8.19, 8.44
[14:06:35] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.94, 9.46, 9.38
[14:08:13] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.90, 9.72, 9.37
[14:09:15] <icinga-miraheze>	 PROBLEM - wiki-asterix.cf - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki-asterix.cf All nameservers failed to answer the query.
[14:09:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[14:11:02] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.97, 9.82, 9.43
[14:12:33] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 7.27, 8.41, 9.63
[14:13:51] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.86, 9.35, 9.31
[14:14:37] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.25, 13.47, 10.59
[14:14:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[14:16:25] <icinga-miraheze>	 RECOVERY - miraheze.ml - reverse DNS on sslhost is OK: SSL OK - miraheze.ml reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK
[14:16:38] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.59, 10.53, 9.79
[14:18:13] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.63, 10.45, 10.11
[14:19:31] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.83, 8.85, 8.20
[14:20:41] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.05, 10.03, 9.53
[14:22:24] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.18, 9.87, 9.84
[14:22:42] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.86, 8.46, 8.40
[14:24:07] <icinga-miraheze>	 RECOVERY - wiki.ripto.gq - reverse DNS on sslhost is OK: SSL OK - wiki.ripto.gq reverse DNS resolves to cp21.miraheze.org - CNAME OK
[14:24:53] <icinga-miraheze>	 RECOVERY - miraheze.gq - reverse DNS on sslhost is OK: SSL OK - miraheze.gq reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK
[14:25:15] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.82, 10.53, 10.12
[14:25:34] <icinga-miraheze>	 RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.98, 8.41, 8.41
[14:26:09] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.52, 9.40, 9.84
[14:26:29] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.60, 8.82, 9.20
[14:27:26] <icinga-miraheze>	 RECOVERY - wiki.thedev.gq - reverse DNS on sslhost is OK: SSL OK - wiki.thedev.gq reverse DNS resolves to cp21.miraheze.org - CNAME OK
[14:28:14] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.94, 9.92, 9.95
[14:30:42] <icinga-miraheze>	 RECOVERY - otcg.ml - reverse DNS on sslhost is OK: SSL OK - otcg.ml reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK
[14:32:40] <MirahezeLSBot>	 !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=estypemoonwiki --username-prefix wikia:typemoon --report 1 --no-updates /home/reception/estypemoon_pages_full.xml (START)
[14:32:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:33:30] <icinga-miraheze>	 PROBLEM - wikileague.cf - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/pyt
[14:33:30] <icinga-miraheze>	 t-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.404007911682129 seconds
[14:34:17] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.97, 10.52, 10.15
[14:34:25] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.59, 9.72, 9.10
[14:34:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[14:35:25] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 11.02, 9.78, 9.43
[14:35:28] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.33, 8.49, 8.55
[14:35:40] <icinga-miraheze>	 RECOVERY - wiki.ct777.cf - reverse DNS on sslhost is OK: SSL OK - wiki.ct777.cf reverse DNS resolves to cp21.miraheze.org - CNAME OK
[14:36:37] <Not-9244>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/edf9f74b953c
[14:36:38] <Not-9244>	 [02miraheze/mw-config] 07Reception123 03edf9f74 - add edit protection level for pokemonarowiki (T8597)
[14:36:40] <Not-9244>	 [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config
[14:36:41] <Not-9244>	 [02mw-config] 07Reception123 opened pull request 03#4413: add edit protection level for pokemonarowiki (T8597) - 13https://github.com/miraheze/mw-config/pull/4413
[14:37:01] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.95, 9.92, 9.99
[14:37:10] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.07, 8.39, 8.68
[14:37:48] <miraheze-github>	 miraheze/mw-config - Reception123 the build passed.
[14:37:48] <Not-9244>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/edf9f74b953c...385486a10c26
[14:37:49] <icinga-miraheze>	 RECOVERY - gp.ct777.cf - reverse DNS on sslhost is OK: SSL OK - gp.ct777.cf reverse DNS resolves to cp21.miraheze.org - CNAME OK
[14:37:50] <Not-9244>	 [02miraheze/mw-config] 07Reception123 03385486a - Update LocalSettings.php
[14:37:51] <Not-9244>	 [02mw-config] 07Reception123 synchronize pull request 03#4413: add edit protection level for pokemonarowiki (T8597) - 13https://github.com/miraheze/mw-config/pull/4413
[14:38:11] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.61, 8.74, 9.08
[14:38:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[14:38:57] <miraheze-github>	 miraheze/mw-config - Reception123 the build passed.
[14:39:45] <icinga-miraheze>	 RECOVERY - wiki-asterix.cf - reverse DNS on sslhost is OK: SSL OK - wiki-asterix.cf reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK
[14:39:49] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.93, 10.25, 10.10
[14:40:08] <Not-9244>	 [02mw-config] 07Reception123 closed pull request 03#4413: add edit protection level for pokemonarowiki (T8597) - 13https://github.com/miraheze/mw-config/pull/4413
[14:40:09] <Not-9244>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a9498a84e881...9bfe54b4eea1
[14:40:10] <Not-9244>	 [02miraheze/mw-config] 07Reception123 039bfe54b - add edit protection level for pokemonarowiki (T8597) (#4413)
[14:40:12] <Not-9244>	 [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1
[14:40:13] <Not-9244>	 [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config
[14:40:17] <MirahezeLSBot>	 !log [reception@mwtask111] starting deploy of {'config': True} to scsvg
[14:40:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:40:28] <MirahezeLSBot>	 !log [reception@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 10s
[14:40:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:40:45] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.39, 9.93, 9.68
[14:40:58] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.79, 9.38, 9.24
[14:41:08] <icinga-miraheze>	 RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.19, 8.08, 8.36
[14:41:19] <miraheze-github>	 miraheze/mw-config - Reception123 the build passed.
[14:42:43] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.52, 9.92, 9.20
[14:43:38] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.39, 9.18, 9.43
[14:44:05] <icinga-miraheze>	 RECOVERY - data.memipedia.ga - reverse DNS on sslhost is OK: SSL OK - data.memipedia.ga reverse DNS resolves to cp20.miraheze.org - CNAME OK
[14:45:32] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.09, 9.60, 9.87
[14:45:39] <icinga-miraheze>	 RECOVERY - ircwiki.cf - reverse DNS on sslhost is OK: SSL OK - ircwiki.cf reverse DNS resolves to cp20.miraheze.org - CNAME FLAT
[14:45:39] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.02, 9.01, 8.97
[14:47:44] <MirahezeLSBot>	 !log [@test101] starting deploy of {'config': True} to skip
[14:47:45] <MirahezeLSBot>	 !log [@test101] finished deploy of {'config': True} to skip - SUCCESS in 0s
[14:47:45] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.67, 9.38, 8.73
[14:47:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:47:53] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:48:42] <icinga-miraheze>	 RECOVERY - memipedia.ga - reverse DNS on sslhost is OK: SSL OK - memipedia.ga reverse DNS resolves to cp20.miraheze.org - CNAME FLAT
[14:49:19] <icinga-miraheze>	 RECOVERY - pj-masks-info.cf - reverse DNS on sslhost is OK: SSL OK - pj-masks-info.cf reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK
[14:50:39] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.73, 9.30, 8.84
[14:51:14] <icinga-miraheze>	 RECOVERY - lakehub.ga - reverse DNS on sslhost is OK: SSL OK - lakehub.ga reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK
[14:51:21] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.27, 10.33, 10.10
[14:51:25] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.55, 3.17
[14:51:45] <MirahezeLSBot>	 !log [@mwtask111] starting deploy of {'config': True} to scsvg
[14:51:57] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:51:58] <MirahezeLSBot>	 !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 12s
[14:52:10] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:52:36] <icinga-miraheze>	 RECOVERY - wikidiffs.ga - reverse DNS on sslhost is OK: SSL OK - wikidiffs.ga reverse DNS resolves to cp20.miraheze.org - CNAME FLAT
[14:53:34] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.38, 9.88, 9.12
[14:54:19] <icinga-miraheze>	 RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.50, 3.36, 3.18
[14:56:23] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.50, 9.10, 8.97
[15:00:03] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.48, 9.12, 9.75
[15:01:06] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.63, 9.65, 9.94
[15:02:58] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.61, 9.47, 9.76
[15:03:47] <icinga-miraheze>	 RECOVERY - wikileague.cf - reverse DNS on sslhost is OK: SSL OK - wikileague.cf reverse DNS resolves to cp21.miraheze.org - NS RECORDS OK
[15:04:26] <icinga-miraheze>	 PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 66, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/u
[15:04:27] <icinga-miraheze>	 ython3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wikimicrofinanza.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[15:05:22] <icinga-miraheze>	 PROBLEM - linkwiki.cf - LetsEncrypt on sslhost is CRITICAL: connect to address linkwiki.cf and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket
[15:08:35] <icinga-miraheze>	 RECOVERY - miraheze.ga - reverse DNS on sslhost is OK: SSL OK - miraheze.ga reverse DNS resolves to cp20.miraheze.org - NS RECORDS OK
[15:08:41] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.81, 8.97, 9.59
[15:12:18] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.03, 9.22, 9.50
[15:13:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[15:13:39] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.08, 9.22, 8.95
[15:14:13] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.31, 9.33, 9.51
[15:15:00] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.07, 8.94, 9.34
[15:16:29] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.76, 9.49, 9.11
[15:16:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[15:17:02] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.93, 9.53, 9.55
[15:17:10] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.16, 9.28, 8.92
[15:17:57] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.73, 10.81, 9.97
[15:18:08] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.71, 8.19, 8.48
[15:19:25] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.76, 9.68, 9.22
[15:19:59] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.06, 9.41, 9.48
[15:20:07] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.33, 9.15, 8.93
[15:22:23] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.73, 9.78, 9.34
[15:22:55] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.31, 9.55, 9.55
[15:25:05] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.24, 8.98, 8.83
[15:25:58] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.76, 10.19, 9.80
[15:28:51] <MirahezeLSBot>	 !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=estypemoonwiki --username-prefix wikia:typemoon --report 1 --no-updates /home/reception/estypemoon_pages_full.xml (END - exit=0)
[15:28:52] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.99, 9.72, 9.70
[15:28:59] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:31:01] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.40, 9.65, 9.37
[15:31:43] <icinga-miraheze>	 RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.82, 7.93, 8.42
[15:33:58] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.12, 9.61, 9.42
[15:34:34] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 9.32, 10.24, 9.97
[15:34:55] <icinga-miraheze>	 PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.wikimicrofinanza.it All nameservers failed to answer the query.
[15:37:31] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.31, 8.99, 9.54
[15:38:10] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.22, 9.08, 9.95
[15:38:38] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.61, 8.59, 8.62
[15:39:05] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.92, 3.29
[15:39:32] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.60, 9.21, 8.94
[15:41:53] <icinga-miraheze>	 RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.78, 3.15, 3.12
[15:45:20] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.15, 9.74, 9.60
[15:45:58] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.04, 9.89, 9.61
[15:46:49] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.92, 10.35, 10.06
[15:47:22] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.13, 9.65, 9.00
[15:49:09] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.45, 9.72, 9.61
[15:50:18] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.64, 9.02, 8.86
[15:51:28] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.64, 9.25, 9.51
[15:52:06] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.35, 9.80, 9.64
[15:54:23] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.31, 9.90, 9.70
[15:58:20] <MirahezeLSBot>	 !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=estypemoonwiki (START)
[15:58:25] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[15:58:47] <MirahezeLSBot>	 !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=estypemoonwiki (END - exit=256)
[15:58:51] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:00:04] <Reception123>	 !log sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php
[16:00:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:00:15] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.69, 9.21, 9.60
[16:00:17] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.41, 8.76, 9.33
[16:00:45] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.19, 9.41, 9.65
[16:01:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[16:03:34] <icinga-miraheze>	 PROBLEM - diamowiki.ga - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - diamowiki.ga All nameservers failed to answer the query.
[16:03:38] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.07, 10.81, 10.14
[16:05:15] <icinga-miraheze>	 RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp21.miraheze.org - CNAME OK
[16:06:28] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.42, 9.65, 9.84
[16:07:39] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[16:09:16] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.17, 10.23, 10.05
[16:10:28] <icinga-miraheze>	 RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 6.19, 7.55, 8.34
[16:12:11] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 5.26, 8.12, 9.27
[16:12:39] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[16:12:53] <icinga-miraheze>	 PROBLEM - wiki.worldsofweary.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:13:05] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 2.55, 6.98, 9.03
[16:13:20] <icinga-miraheze>	 PROBLEM - wiki-asterix.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
[16:14:18] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 3.49, 6.37, 8.92
[16:14:49] <icinga-miraheze>	 PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.85, 2.42, 1.42
[16:14:53] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb
[16:15:08] <icinga-miraheze>	 RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 3.02, 5.56, 7.72
[16:15:09] <MirahezeLSBot>	 !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=turtlewikiwiki --username-prefix turtlewikiwiki --report 1 --no-updates /home/reception/turtlewikiwiki.xml (START)
[16:15:20] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 2.78, 5.31, 7.43
[16:15:45] <icinga-miraheze>	 RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 2.54, 5.43, 7.95
[16:16:21] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb
[16:16:25] <icinga-miraheze>	 RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 3.63, 5.22, 7.93
[16:17:22] <icinga-miraheze>	 PROBLEM - duepedia.uk.to - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:17:40] <icinga-miraheze>	 RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 3.03, 4.73, 7.76
[16:17:46] <icinga-miraheze>	 PROBLEM - wikimoma.art - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:17:53] <icinga-miraheze>	 PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is CRITICAL: CRITICAL - NGINX Error Rate is 70%
[16:18:21] <icinga-miraheze>	 PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:18:45] <icinga-miraheze>	 PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 5.05, 3.28, 2.09
[16:19:56] <icinga-miraheze>	 PROBLEM - wiki.meregos.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
[16:20:28] <icinga-miraheze>	 PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.43, 3.75, 2.80
[16:21:18] <icinga-miraheze>	 RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 1%
[16:21:42] <icinga-miraheze>	 RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3239 bytes in 2.065 second response time
[16:22:02] <icinga-miraheze>	 PROBLEM - cp30 HTTPS on cp30 is CRITICAL: CRITICAL - Cannot make SSL connection.
[16:22:17] <MirahezeLSBot>	 !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=turtlewikiwiki --search-recursively /home/reception/turtlewikiwiki --comment=Imported from turtlewiki.miraheze.org (START)
[16:22:18] <MirahezeLSBot>	 !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=turtlewikiwiki --search-recursively /home/reception/turtlewikiwiki --comment=Imported from turtlewiki.miraheze.org (END - exit=0)
[16:22:28] <MirahezeLSBot>	 !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=turtlewikiwiki --search-recursively /home/reception/turtlewiki --comment=Imported from turtlewiki.miraheze.org (START)
[16:22:52] <icinga-miraheze>	 PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 1.93, 2.93, 2.03
[16:25:22] <MirahezeLSBot>	 !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=turtlewikiwiki --search-recursively /home/reception/turtlewiki --comment=Imported from turtlewiki.miraheze.org (END - exit=0)
[16:25:39] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[16:28:22] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.29, 4.25, 3.57
[16:29:06] <icinga-miraheze>	 RECOVERY - cp30 HTTPS on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3239 bytes in 3.333 second response time
[16:29:24] <Agent>	 Per #general, users are experiencing 500s, 503s, 504s and connection timeouts
[16:29:36] <Agent>	 Has been happening on and off for about 10 minutes now
[16:30:00] <icinga-miraheze>	 PROBLEM - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is WARNING: WARNING - NGINX Error Rate is 56%
[16:31:16] <Agent>	 !sre ^
[16:31:24] <MirahezeLSBot>	 !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=turtlewikiwiki --username-prefix turtlewikiwiki --report 1 --no-updates /home/reception/turtlewikiwiki.xml (END - exit=0)
[16:31:36] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.76, 3.68, 3.45
[16:31:37] <Agent>	 Reception123 perhaps ^
[16:32:15] <icinga-miraheze>	 PROBLEM - www.fraterniterebellion.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:32:28] <Reception123>	 Hmm that's odd, I've done some imports before and I don't remember anything happening
[16:32:42] <Agent>	 Lots of users are reporting issues
[16:33:00] <Agent>	 And -feed shows we're having lapses of times where we have no edits
[16:33:13] <icinga-miraheze>	 RECOVERY - cp21 HTTP 4xx/5xx ERROR Rate on cp21 is OK: OK - NGINX Error Rate is 27%
[16:33:18] <Agent>	 Seems to have recovered though for now
[16:33:44] <icinga-miraheze>	 PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn
[16:33:44] <icinga-miraheze>	 AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.407209396362305 seconds
[16:34:37] <icinga-miraheze>	 PROBLEM - puzzles.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
[16:34:42] <icinga-miraheze>	 PROBLEM - johanloopmans.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
[16:34:45] <Agent>	 We've gone down again per -feed
[16:35:05] <icinga-miraheze>	 RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.45, 3.00, 3.24
[16:35:35] <icinga-miraheze>	 PROBLEM - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is CRITICAL: CRITICAL - NGINX Error Rate is 73%
[16:35:54] <Agent>	 that's not good
[16:36:39] <icinga-miraheze>	 PROBLEM - wiki.projectdiablo2.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:36:44] <icinga-miraheze>	 PROBLEM - usaha.click - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:36:58] <icinga-miraheze>	 PROBLEM - wiki.astralprojections.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:38:47] <icinga-miraheze>	 RECOVERY - cp30 HTTP 4xx/5xx ERROR Rate on cp30 is OK: OK - NGINX Error Rate is 2%
[16:39:25] <icinga-miraheze>	 PROBLEM - aarthal.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
[16:39:27] <icinga-miraheze>	 PROBLEM - www.mcpk.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
[16:39:43] <icinga-miraheze>	 PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:41:20] <icinga-miraheze>	 PROBLEM - baharna.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:41:21] <icinga-miraheze>	 PROBLEM - wiki.fbpml.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:41:53] <icinga-miraheze>	 PROBLEM - dc.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
[16:43:11] <RhinosF1>	 Agent: here
[16:43:24] <icinga-miraheze>	 PROBLEM - wiki.worldsofweary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.worldsofweary.com' expires in 12 day(s) (Mon 31 Jan 2022 22:25:48 GMT +0000).
[16:44:19] <RhinosF1>	 Agent: looks like an edge issue
[16:44:28] <RhinosF1>	 icinga & grafana are down
[16:44:32] <RhinosF1>	 paladox, JohnLewis: ^
[16:44:36] <icinga-miraheze>	 PROBLEM - wiki-asterix.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki-asterix.cf' expires in 12 day(s) (Mon 31 Jan 2022 23:08:28 GMT +0000).
[16:45:50] <icinga-miraheze>	 PROBLEM - cp30 HTTPS on cp30 is CRITICAL: CRITICAL - Cannot make SSL connection.
[16:45:58] <icinga-miraheze>	 RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3228 bytes in 0.058 second response time
[16:46:28] <icinga-miraheze>	 PROBLEM - www.christipedia.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:46:59] <RhinosF1>	 Reception123: can you ssh into any cp*
[16:47:35] <icinga-miraheze>	 RECOVERY - duepedia.uk.to - LetsEncrypt on sslhost is OK: OK - Certificate 'duepedia.uk.to' will expire on Wed 09 Feb 2022 05:51:34 GMT +0000.
[16:50:08] <icinga-miraheze>	 RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.83, 0.21, 0.07
[16:50:12] <icinga-miraheze>	 PROBLEM - wiki.meregos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.meregos.com' expires in 12 day(s) (Mon 31 Jan 2022 22:24:31 GMT +0000).
[16:50:59] <icinga-miraheze>	 PROBLEM - cp31 HTTPS on cp31 is CRITICAL: CRITICAL - Cannot make SSL connection.
[16:51:12] <paladox>	 !log rebooted cp*
[16:51:15] <icinga-miraheze>	 RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.75, 0.31, 0.11
[16:51:23] <paladox>	 as i have no metrics available i don't have any idea on the cause
[16:51:45] <paladox>	 i'm out of ideas. Rebooted cp and it's still happening
[16:52:41] <RhinosF1>	 paladox: it's network
[16:52:47] <RhinosF1>	 mtr in -security
[16:53:02] <paladox>	 network is out of my experence. Only JohnLewis really has experence with that.
[16:53:25] <icinga-miraheze>	 PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:54:10] <icinga-miraheze>	 RECOVERY - cp31 HTTPS on cp31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3229 bytes in 0.619 second response time
[16:55:17] <icinga-miraheze>	 RECOVERY - cp30 HTTPS on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3239 bytes in 0.335 second response time
[16:56:35] <icinga-miraheze>	 PROBLEM - electrowiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
[16:57:23] <icinga-miraheze>	 PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.06, 2.79, 1.27
[16:58:50] <icinga-miraheze>	 PROBLEM - nonbinary.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
[16:58:51] <icinga-miraheze>	 PROBLEM - hololive.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:59:02] <icinga-miraheze>	 PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:59:20] <icinga-miraheze>	 PROBLEM - www.electrowiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[16:59:45] <icinga-miraheze>	 PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 5.19, 4.02, 2.04
[17:00:34] <Not-9244>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/ffb8f8565df4
[17:00:35] <Not-9244>	 [02miraheze/puppet] 07paladox 03ffb8f85 - cp20/21/30/31: set nginx::worker_processes to 4
[17:00:37] <Not-9244>	 [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet
[17:00:38] <Not-9244>	 [02puppet] 07paladox opened pull request 03#2368: cp20/21/30/31: set nginx::worker_processes to 4 - 13https://github.com/miraheze/puppet/pull/2368
[17:00:50] <icinga-miraheze>	 PROBLEM - dcmultiverse.miraheze.org - Sectigo on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:00:52] <Not-9244>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ffb8f8565df4...e810f3d1aa7a
[17:00:54] <Not-9244>	 [02miraheze/puppet] 07paladox 03e810f3d - Update cp21.yaml
[17:00:55] <Not-9244>	 [02puppet] 07paladox synchronize pull request 03#2368: cp20/21/30/31: set nginx::worker_processes to 4 - 13https://github.com/miraheze/puppet/pull/2368
[17:01:02] <Not-9244>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e810f3d1aa7a...2a410e30a2ff
[17:01:04] <Not-9244>	 [02miraheze/puppet] 07paladox 032a410e3 - Update cp30.yaml
[17:01:05] <Not-9244>	 [02puppet] 07paladox synchronize pull request 03#2368: cp20/21/30/31: set nginx::worker_processes to 4 - 13https://github.com/miraheze/puppet/pull/2368
[17:01:13] <Not-9244>	 [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/2a410e30a2ff...348d9e963645
[17:01:14] <Not-9244>	 [02miraheze/puppet] 07paladox 03348d9e9 - Update cp31.yaml
[17:01:16] <Not-9244>	 [02puppet] 07paladox synchronize pull request 03#2368: cp20/21/30/31: set nginx::worker_processes to 4 - 13https://github.com/miraheze/puppet/pull/2368
[17:02:56] <icinga-miraheze>	 RECOVERY - www.fraterniterebellion.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fraterniterebellion.com' will expire on Sun 27 Mar 2022 23:23:46 GMT +0000.
[17:03:24] <icinga-miraheze>	 PROBLEM - ping6 on cp31 is WARNING: PING WARNING - Packet loss = 0%, RTA = 120.06 ms
[17:03:58] <icinga-miraheze>	 RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3229 bytes in 6.808 second response time
[17:04:13] <icinga-miraheze>	 PROBLEM - cp30 HTTPS on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:05:08] <icinga-miraheze>	 PROBLEM - puzzles.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'puzzles.wiki' expires in 12 day(s) (Tue 01 Feb 2022 05:45:17 GMT +0000).
[17:05:15] <icinga-miraheze>	 PROBLEM - johanloopmans.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.johanloopmans.nl' expires in 12 day(s) (Mon 31 Jan 2022 22:42:56 GMT +0000).
[17:06:36] <icinga-miraheze>	 RECOVERY - ping6 on cp31 is OK: PING OK - Packet loss = 0%, RTA = 90.22 ms
[17:07:02] <icinga-miraheze>	 RECOVERY - wiki.projectdiablo2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.projectdiablo2.com' will expire on Thu 03 Mar 2022 06:03:47 GMT +0000.
[17:07:09] <icinga-miraheze>	 RECOVERY - wiki.astralprojections.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.astralprojections.org' will expire on Thu 07 Apr 2022 03:01:23 GMT +0000.
[17:07:24] <icinga-miraheze>	 RECOVERY - usaha.click - LetsEncrypt on sslhost is OK: OK - Certificate 'usaha.click' will expire on Wed 02 Mar 2022 08:00:07 GMT +0000.
[17:07:41] <icinga-miraheze>	 PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is CRITICAL: connect to address www.wikimicrofinanza.it and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket
[17:08:11] <icinga-miraheze>	 PROBLEM - cp20 APT on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[17:08:53] <icinga-miraheze>	 RECOVERY - aarthal.com - LetsEncrypt on sslhost is OK: OK - Certificate 'aarthal.com' will expire on Mon 28 Mar 2022 14:04:44 GMT +0000.
[17:09:29] <icinga-miraheze>	 PROBLEM - www.mcpk.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.mcpk.wiki' expires in 12 day(s) (Mon 31 Jan 2022 22:52:23 GMT +0000).
[17:10:17] <icinga-miraheze>	 RECOVERY - cp30 HTTPS on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3240 bytes in 8.014 second response time
[17:11:14] <icinga-miraheze>	 RECOVERY - cp20 APT on cp20 is OK: APT OK: 1 packages available for upgrade (0 critical updates).
[17:11:47] <icinga-miraheze>	 PROBLEM - baharna.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.baharna.org' expires in 12 day(s) (Mon 31 Jan 2022 22:29:54 GMT +0000).
[17:11:59] <icinga-miraheze>	 RECOVERY - wiki.fbpml.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.fbpml.org' will expire on Mon 21 Feb 2022 17:45:02 GMT +0000.
[17:12:21] <icinga-miraheze>	 RECOVERY - dc.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000.
[17:16:17] <icinga-miraheze>	 PROBLEM - www.christipedia.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.christipedia.nl' expires in 12 day(s) (Mon 31 Jan 2022 22:39:36 GMT +0000).
[17:19:51] <icinga-miraheze>	 PROBLEM - wikimoma.art - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikimoma.art' expires in 12 day(s) (Tue 01 Feb 2022 05:24:41 GMT +0000).
[17:23:18] <icinga-miraheze>	 PROBLEM - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is CRITICAL: CRITICAL - NGINX Error Rate is 100%
[17:25:17] <RhinosF1>	 .op
[17:25:17] <MirahezeBot>	 Attempting to OP...
[17:26:12] <icinga-miraheze>	 RECOVERY - cp20 HTTP 4xx/5xx ERROR Rate on cp20 is OK: OK - NGINX Error Rate is 2%
[17:27:12] <icinga-miraheze>	 PROBLEM - electrowiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'electrowiki.org' expires in 12 day(s) (Tue 01 Feb 2022 05:17:32 GMT +0000).
[17:28:29] <icinga-miraheze>	 RECOVERY - nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'nonbinary.wiki' will expire on Sun 27 Mar 2022 23:33:20 GMT +0000.
[17:28:29] <icinga-miraheze>	 RECOVERY - hololive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hololive.wiki' will expire on Sun 27 Mar 2022 23:40:50 GMT +0000.
[17:28:35] <icinga-miraheze>	 RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.53, 0.16, 0.06
[17:28:46] <icinga-miraheze>	 RECOVERY - www.dariawiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'dariawiki.org' will expire on Sun 27 Mar 2022 23:38:58 GMT +0000.
[17:29:06] <icinga-miraheze>	 RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.52, 0.16, 0.06
[17:29:54] <icinga-miraheze>	 PROBLEM - www.electrowiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'electrowiki.org' expires in 12 day(s) (Tue 01 Feb 2022 05:17:32 GMT +0000).
[17:30:50] <icinga-miraheze>	 PROBLEM - cp30 HTTPS on cp30 is CRITICAL: CRITICAL - Cannot make SSL connection.
[17:31:03] <icinga-miraheze>	 RECOVERY - dcmultiverse.miraheze.org - Sectigo on sslhost is OK: OK - Certificate '*.miraheze.org' will expire on Wed 09 Nov 2022 23:59:59 GMT +0000.
[17:31:44] <icinga-miraheze>	 PROBLEM - cp31 HTTPS on cp31 is CRITICAL: HTTP CRITICAL - No data received from host
[17:32:06] <icinga-miraheze>	 PROBLEM - www.bluepageswiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:32:09] <icinga-miraheze>	 PROBLEM - patternarchive.online - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:33:27] <icinga-miraheze>	 PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[17:34:20] <icinga-miraheze>	 RECOVERY - cp30 HTTPS on cp30 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3229 bytes in 1.166 second response time
[17:36:32] <icinga-miraheze>	 RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3229 bytes in 0.022 second response time
[17:36:35] <paladox>	 !log cp*: sysctl net.core.somaxconn=16384
[17:36:37] <icinga-miraheze>	 PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.98, 2.40, 1.24
[17:36:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:38:25] <icinga-miraheze>	 RECOVERY - cp31 HTTPS on cp31 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3239 bytes in 0.324 second response time
[17:38:48] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.08, 3.98, 3.56
[17:41:10] <icinga-miraheze>	 PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/us
[17:41:11] <icinga-miraheze>	 thon3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query www.wikimicrofinanza.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[17:41:50] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.13, 4.19, 3.76
[17:45:05] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[17:45:07] <icinga-miraheze>	 PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.50, 2.31, 1.51
[17:50:42] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.56, 3.48, 3.62
[17:51:58] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb
[17:54:34] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[17:56:40] <icinga-miraheze>	 RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.89, 3.11, 3.38
[17:58:20] <RhinosF1>	 !log updated recaptcha key to enable enterprise
[17:58:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:01:21] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb
[18:02:17] <icinga-miraheze>	 PROBLEM - www.bluepageswiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.bluepageswiki.org' expires in 12 day(s) (Mon 31 Jan 2022 22:56:37 GMT +0000).
[18:02:19] <icinga-miraheze>	 PROBLEM - patternarchive.online - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'patternarchive.online' expires in 12 day(s) (Tue 01 Feb 2022 05:12:22 GMT +0000).
[18:03:38] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.88, 3.43, 3.42
[18:06:40] <icinga-miraheze>	 PROBLEM - johanloopmans.nl - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
[18:06:58] <icinga-miraheze>	 RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.59, 2.74, 3.15
[18:07:59] <icinga-miraheze>	 PROBLEM - www.wikimicrofinanza.it - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.wikimicrofinanza.it' expires in 12 day(s) (Mon 31 Jan 2022 22:54:41 GMT +0000).
[18:08:36] <icinga-miraheze>	 PROBLEM - www.erikapedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:08:51] <icinga-miraheze>	 PROBLEM - thestarsareright.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
[18:08:57] <icinga-miraheze>	 PROBLEM - erikapedia.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Cannot make SSL connection.
[18:09:37] <icinga-miraheze>	 RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.01, 0.23, 0.08
[18:09:45] <icinga-miraheze>	 PROBLEM - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is CRITICAL: CRITICAL - NGINX Error Rate is 99%
[18:10:14] <icinga-miraheze>	 RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.35, 0.48, 0.17
[18:11:54] <icinga-miraheze>	 RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.81, 0.32, 0.12
[18:15:06] <icinga-miraheze>	 RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.00, 1.17, 0.55
[18:15:35] <icinga-miraheze>	 RECOVERY - cp31 HTTP 4xx/5xx ERROR Rate on cp31 is OK: OK - NGINX Error Rate is 1%
[18:16:35] <icinga-miraheze>	 PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 5.72, 3.33, 1.54
[18:16:53] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.37, 3.40, 3.08
[18:17:04] <icinga-miraheze>	 PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 3.15, 2.27, 1.08
[18:18:42] <icinga-miraheze>	 PROBLEM - cp21 HTTPS on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:18:56] <icinga-miraheze>	 PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.14, 1.96, 0.94
[18:19:52] <icinga-miraheze>	 RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.22, 3.37, 3.13
[18:21:46] <icinga-miraheze>	 RECOVERY - cp21 HTTPS on cp21 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3237 bytes in 0.040 second response time
[18:22:13] <icinga-miraheze>	 PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.36, 1.90, 1.14
[18:25:14] <icinga-miraheze>	 PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 0.95, 1.89, 1.31
[18:25:39] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[18:25:52] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[18:27:45] <icinga-miraheze>	 PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.59, 3.54, 3.02
[18:28:17] <icinga-miraheze>	 RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.21, 1.14, 1.12
[18:28:39] <icinga-miraheze>	 RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.21, 1.41, 1.66
[18:28:57] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.03, 10.33, 7.38
[18:29:07] <icinga-miraheze>	 PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.11, 1.95, 1.81
[18:30:05] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.74, 3.80, 3.43
[18:30:32] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[18:30:47] <icinga-miraheze>	 PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.11, 1.77, 1.59
[18:30:55] <icinga-miraheze>	 PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.56, 10.50, 7.66
[18:31:33] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.78, 9.53, 6.79
[18:31:33] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.02, 8.18, 6.52
[18:31:48] <icinga-miraheze>	 PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.60, 4.01, 3.35
[18:32:02] <icinga-miraheze>	 RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.86, 1.38, 1.60
[18:32:15] <MirahezeLSBot>	 !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=turtlewikiwiki (START)
[18:32:16] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.89, 8.43, 6.62
[18:32:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:33:02] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.48, 4.79, 3.90
[18:33:42] <icinga-miraheze>	 RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.18, 1.04, 1.34
[18:34:31] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.94, 9.35, 7.23
[18:34:32] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.18, 9.56, 7.35
[18:36:44] <icinga-miraheze>	 RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.09, 3.33, 3.26
[18:37:09] <icinga-miraheze>	 PROBLEM - johanloopmans.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.johanloopmans.nl' expires in 12 day(s) (Mon 31 Jan 2022 22:42:56 GMT +0000).
[18:38:38] <icinga-miraheze>	 PROBLEM - www.erikapedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.erikapedia.com' expires in 12 day(s) (Mon 31 Jan 2022 22:50:10 GMT +0000).
[18:39:06] <icinga-miraheze>	 PROBLEM - thestarsareright.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'thestarsareright.org' expires in 12 day(s) (Mon 31 Jan 2022 22:47:28 GMT +0000).
[18:39:40] <icinga-miraheze>	 PROBLEM - erikapedia.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.erikapedia.com' expires in 12 day(s) (Mon 31 Jan 2022 22:50:10 GMT +0000).
[18:40:30] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.20, 9.32, 7.98
[18:40:46] <icinga-miraheze>	 PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.42, 3.64, 3.54
[18:40:54] <icinga-miraheze>	 PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.60, 2.15, 1.50
[18:42:13] <icinga-miraheze>	 RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp20.miraheze.org - CNAME OK
[18:43:26] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.57, 9.55, 8.27
[18:43:37] <icinga-miraheze>	 PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.23, 3.56, 3.44
[18:44:19] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.43, 9.64, 8.15
[18:46:28] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.03, 9.98, 8.57
[18:46:44] <icinga-miraheze>	 PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.15, 4.48, 3.88
[18:47:30] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 4.69, 8.12, 7.90
[18:49:39] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.87, 9.17, 8.51
[18:49:39] <icinga-miraheze>	 RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.18, 8.03, 8.01
[18:49:48] <icinga-miraheze>	 PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.62, 3.79, 3.57
[18:50:32] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[18:52:46] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.25, 10.28, 9.08
[18:53:01] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb
[18:54:32] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[18:55:29] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.08, 8.52, 7.90
[18:56:03] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[18:56:41] <MirahezeLSBot>	 !log [reception@mwtask111] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=turtlewikiwiki (END - exit=0)
[18:56:48] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 11.35, 9.95, 8.87
[18:56:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:57:45] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.26, 10.18, 9.03
[18:58:24] <icinga-miraheze>	 RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.21, 8.48, 8.00
[18:58:38] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.72, 9.55, 9.20
[18:58:50] <icinga-miraheze>	 PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.60, 3.92, 3.87
[19:04:21] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.45, 10.66, 9.75
[19:05:09] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.65, 10.00, 8.86
[19:07:48] <icinga-miraheze>	 PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.90, 4.12, 3.89
[19:10:49] <icinga-miraheze>	 PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.65, 3.82, 3.85
[19:14:07] <icinga-miraheze>	 PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 0.00, 0.29, 1.98
[19:17:13] <icinga-miraheze>	 RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 0.16, 0.19, 1.62
[19:19:51] <icinga-miraheze>	 PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.80, 3.84, 3.71
[19:20:24] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.77, 9.87, 9.69
[19:22:45] <icinga-miraheze>	 PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.68, 3.46, 3.92
[19:22:47] <icinga-miraheze>	 PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.39, 3.20, 3.49
[19:23:19] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.32, 9.59, 9.58
[19:25:37] <icinga-miraheze>	 PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.12, 4.07, 4.07
[19:26:11] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.30, 9.32, 9.49
[19:28:32] <icinga-miraheze>	 PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.16, 3.81, 3.97
[19:28:34] <icinga-miraheze>	 RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.57, 3.03, 3.33
[19:29:09] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.48, 10.15, 9.79
[19:32:01] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.64, 9.23, 9.49
[19:33:48] <icinga-miraheze>	 RECOVERY - diamowiki.ga - reverse DNS on sslhost is OK: SSL OK - diamowiki.ga reverse DNS resolves to cp20.miraheze.org - CNAME OK
[19:38:02] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.78, 9.94, 9.72
[19:43:55] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.13, 9.95, 9.82
[19:46:15] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.15, 9.05, 9.89
[19:49:23] <icinga-miraheze>	 PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.66, 4.97, 4.14
[19:49:29] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 14.04, 11.11, 10.49
[19:50:32] <Not-9244>	 [02puppet] 07supertassu opened pull request 03#2369: bastion: move firewall rule to role class - 13https://github.com/miraheze/puppet/pull/2369
[19:52:44] <icinga-miraheze>	 PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.15, 3.87, 3.48
[19:54:02] <icinga-miraheze>	 PROBLEM - test101 Puppet on test101 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega, last run 35 minutes ago with 0 failures
[19:54:12] <icinga-miraheze>	 PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:54:37] <icinga-miraheze>	 PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[19:54:53] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.87, 9.38, 10.00
[19:55:42] <icinga-miraheze>	 RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.62, 3.08, 3.26
[19:56:30] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.83, 9.50, 9.95
[19:57:10] <icinga-miraheze>	 RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.429 second response time
[19:57:34] <icinga-miraheze>	 RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 2.016 second response time
[19:59:27] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.38, 9.75, 9.96
[20:01:25] <icinga-miraheze>	 PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.86, 3.94
[20:03:39] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.04, 9.24, 9.62
[20:04:47] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.59, 9.93, 9.60
[20:05:06] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.47, 9.49, 9.82
[20:06:35] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.26, 9.52, 9.67
[20:07:41] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.29, 9.42, 9.48
[20:08:04] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.99, 9.57, 9.76
[20:09:41] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.25, 10.66, 10.11
[20:10:13] <icinga-miraheze>	 PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.56, 3.95, 3.86
[20:10:53] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.42, 10.63, 9.92
[20:11:07] <MirahezeLSBot>	 !log [universalomega@test101] starting deploy of {'folders': 'w/extensions/MirahezeMagic'} to skip
[20:11:08] <MirahezeLSBot>	 !log [universalomega@test101] finished deploy of {'folders': 'w/extensions/MirahezeMagic'} to skip - SUCCESS in 0s
[20:11:23] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:11:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:13:46] <icinga-miraheze>	 PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.79, 3.63, 3.48
[20:15:07] <Not-9244>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-8 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/643c877217f4
[20:15:09] <Not-9244>	 [02miraheze/mw-config] 07Universal-Omega 03643c877 - set `$wgReCaptchaEnterpriseProjectId`
[20:15:10] <Not-9244>	 [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config
[20:15:12] <Not-9244>	 [02mw-config] 07Universal-Omega opened pull request 03#4414: set `$wgReCaptchaEnterpriseProjectId` - 13https://github.com/miraheze/mw-config/pull/4414
[20:16:16] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[20:19:18] <icinga-miraheze>	 PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.85, 3.60, 3.74
[20:19:56] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 7.84, 9.43, 9.73
[20:22:08] <icinga-miraheze>	 PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.05, 4.09, 3.87
[20:22:45] <icinga-miraheze>	 RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.73, 3.11, 3.31
[20:24:58] <icinga-miraheze>	 PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.31, 3.80, 3.81
[20:25:41] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.56, 9.84, 9.81
[20:27:56] <icinga-miraheze>	 PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.25, 3.84, 3.80
[20:28:42] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.09, 9.23, 9.59
[20:30:50] <icinga-miraheze>	 PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.91, 3.21, 3.55
[20:31:01] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.39, 8.75, 9.80
[20:32:33] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.65, 3.25, 3.89
[20:32:33] <CosmicAlpha>	 !log mhglobal: UPDATE cw_wikis SET wiki_closed_timestamp = NULL WHERE wiki_closed_timestamp = cast(0 as binary(14));
[20:32:41] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:36:18] <icinga-miraheze>	 PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.69, 3.82, 3.45
[20:36:47] <icinga-miraheze>	 RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.91, 3.11, 3.40
[20:37:35] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.59, 9.64, 9.50
[20:39:26] <icinga-miraheze>	 PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.63, 3.53, 3.42
[20:40:10] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.19, 10.45, 10.08
[20:43:50] <MirahezeLSBot>	 !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (START)
[20:44:01] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:45:32] <icinga-miraheze>	 RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.25, 3.07, 3.30
[20:46:41] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.84, 9.88, 9.79
[20:48:40] <Not-9244>	 [02mw-config] 07Universal-Omega closed pull request 03#4414: set `$wgReCaptchaEnterpriseProjectId` - 13https://github.com/miraheze/mw-config/pull/4414
[20:48:42] <Not-9244>	 [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8
[20:48:43] <Not-9244>	 [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9bfe54b4eea1...197bd8091731
[20:48:45] <Not-9244>	 [02miraheze/mw-config] 07Universal-Omega 03197bd80 - set `$wgReCaptchaEnterpriseProjectId` (#4414)
[20:48:46] <Not-9244>	 [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-8 - 13https://github.com/miraheze/mw-config
[20:49:21] <MirahezeLSBot>	 !log [universalomega@test101] starting deploy of {'pull': 'config', 'config': True} to skip
[20:49:23] <MirahezeLSBot>	 !log [universalomega@test101] finished deploy of {'pull': 'config', 'config': True} to skip - SUCCESS in 1s
[20:49:41] <miraheze-github>	 miraheze/mw-config - Universal-Omega the build passed.
[20:49:42] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:49:54] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:53:04] <icinga-miraheze>	 PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 3.05, 2.21, 1.14
[20:53:34] <icinga-miraheze>	 RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.22, 2.58, 3.24
[20:54:13] <MirahezeLSBot>	 !log [@mwtask111] starting deploy of {'config': True} to scsvg
[20:54:21] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:54:41] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.12, 8.92, 9.86
[20:54:42] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 6.15, 8.33, 9.92
[20:54:45] <MirahezeLSBot>	 !log [@mwtask111] finished deploy of {'config': True} to scsvg - SUCCESS in 31s
[20:55:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[20:55:45] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 7.59, 9.12, 9.99
[20:58:22] <icinga-miraheze>	 PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[20:58:34] <icinga-miraheze>	 PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[21:01:21] <icinga-miraheze>	 RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.014 second response time
[21:01:30] <icinga-miraheze>	 RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.328 second response time
[21:01:38] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 15.16, 10.37, 9.54
[21:03:54] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.40, 9.71, 9.70
[21:09:50] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.15, 10.21, 9.47
[21:12:48] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.34, 8.89, 9.33
[21:12:57] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.36, 9.72, 9.49
[21:13:52] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.93, 11.04, 9.90
[21:16:07] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.32, 9.35, 9.40
[21:17:29] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb
[21:18:04] <Not-9244>	 [02puppet] 07Universal-Omega opened pull request 03#2370: Add webVideoTranscodePrioritized to high intensity jobs - 13https://github.com/miraheze/puppet/pull/2370
[21:18:43] <Not-9244>	 [02puppet] 07Universal-Omega synchronize pull request 03#2370: Add webVideoTranscodePrioritized to high intensity jobs - 13https://github.com/miraheze/puppet/pull/2370
[21:19:19] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.20, 8.54, 9.10
[21:19:30] <icinga-miraheze>	 PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[21:19:56] <icinga-miraheze>	 PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 1595 bytes in 0.008 second response time
[21:20:33] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[21:21:16] <icinga-miraheze>	 PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[21:21:31] <icinga-miraheze>	 PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[21:22:19] <icinga-miraheze>	 PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[21:23:14] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 5.98, 7.74, 9.81
[21:24:32] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[21:25:21] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.04, 9.15, 9.11
[21:25:37] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.85, 7.44, 8.35
[21:27:15] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::4c25/cpweb
[21:27:32] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[21:28:18] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 7.39, 8.55, 9.69
[21:28:25] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.10, 8.59, 8.87
[21:30:20] <RhinosF1>	 !log limit reCaptcha to 3,333 Assesments per day to prevent charges
[21:30:23] <RhinosF1>	 Reception123: ^
[21:30:51] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 198.244.148.90/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb
[21:31:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:33:26] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[21:34:32] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.19, 8.89, 8.81
[21:35:28] <icinga-miraheze>	 RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.46, 7.28, 8.42
[21:36:53] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[21:38:41] <RhinosF1>	 !log updated account request form to ask for error and added reCaptcha reviewed column to google sheet
[21:39:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:40:31] <icinga-miraheze>	 PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.51, 8.49, 8.75
[21:40:43] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.70, 8.21, 8.51
[21:41:12] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.79, 8.96, 8.64
[21:41:22] <icinga-miraheze>	 RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 9.455 second response time
[21:41:42] <icinga-miraheze>	 RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 3.257 second response time
[21:42:25] <icinga-miraheze>	 RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 0.641 second response time
[21:43:18] <icinga-miraheze>	 RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.336 second response time
[21:43:54] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.90, 10.28, 9.21
[21:43:55] <icinga-miraheze>	 RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.019 second response time
[21:44:12] <icinga-miraheze>	 PROBLEM - diamowiki.ga - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 116, in main rdns_hostname = get_reverse_dnshostname(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 101, in get_reverse_dnshostname resolved_ip_addr = str(dns_resolver.resolve(hostn
[21:44:12] <icinga-miraheze>	 AA')[0]) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.4065728187561035 seconds
[21:44:35] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.140.43/cpweb
[21:46:56] <MirahezeLSBot>	 !log [universalomega@mwtask111] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/extensions/MirahezeMagic/maintenance/resetWikiCaches.php (END - exit=0)
[21:46:59] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.07, 9.54, 9.16
[21:47:05] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[21:47:27] <icinga-miraheze>	 RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.71, 7.76, 8.30
[21:47:39] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[21:49:39] <icinga-miraheze>	 PROBLEM - cp21 Stunnel Http for mw101 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[21:50:23] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb
[21:52:02] <icinga-miraheze>	 PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.82, 1.73, 1.96
[21:52:48] <icinga-miraheze>	 RECOVERY - cp21 Stunnel Http for mw101 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.015 second response time
[21:53:32] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[21:54:58] <icinga-miraheze>	 PROBLEM - cp20 Stunnel Http for mw111 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[21:55:33] <icinga-miraheze>	 PROBLEM - cp21 Stunnel Http for mw111 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[21:56:01] <icinga-miraheze>	 PROBLEM - cp31 Stunnel Http for mw111 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds.
[21:56:30] <icinga-miraheze>	 PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[21:56:33] <icinga-miraheze>	 RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 6.52, 7.39, 8.31
[21:57:05] <icinga-miraheze>	 PROBLEM - cp30 Stunnel Http for mw111 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.240 second response time
[21:57:31] <icinga-miraheze>	 PROBLEM - www.wikimicrofinanza.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in <module> main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/us
[21:57:31] <icinga-miraheze>	 thon3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query www.wikimicrofinanza.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[21:58:20] <icinga-miraheze>	 RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.02, 1.29, 1.69
[21:58:40] <icinga-miraheze>	 RECOVERY - cp21 Stunnel Http for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.534 second response time
[21:59:08] <icinga-miraheze>	 RECOVERY - cp31 Stunnel Http for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 1.814 second response time
[21:59:41] <icinga-miraheze>	 RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22193 bytes in 7.703 second response time
[22:00:16] <icinga-miraheze>	 RECOVERY - cp30 Stunnel Http for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.053 second response time
[22:01:05] <icinga-miraheze>	 PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 2607:5300:201:3100::929a/cpweb
[22:01:32] <icinga-miraheze>	 RECOVERY - cp20 Stunnel Http for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14548 bytes in 8.397 second response time
[22:03:37] <icinga-miraheze>	 PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 51.195.220.68/cpweb
[22:04:15] <icinga-miraheze>	 RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online
[22:06:41] <icinga-miraheze>	 RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online
[22:11:56] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.12, 10.03, 9.19
[22:12:28] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.51, 9.24, 8.55
[22:13:07] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.51, 3.76, 3.12
[22:13:15] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.19, 9.28, 8.76
[22:18:57] <icinga-miraheze>	 RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.89, 3.21, 3.07
[22:21:19] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.41, 9.35, 8.86
[22:23:35] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.43, 9.33, 9.44
[22:24:09] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.29, 8.69, 8.69
[22:26:22] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.80, 9.77, 9.57
[22:26:58] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.20, 9.22, 8.89
[22:29:16] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.03, 9.28, 9.45
[22:29:54] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.43, 9.29, 8.97
[22:32:07] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.33, 9.82, 9.62
[22:35:40] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.02, 9.75, 9.25
[22:42:25] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.73, 10.24, 9.37
[22:46:12] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.49, 3.87, 3.34
[22:50:35] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 7.70, 9.29, 9.77
[22:51:26] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.27, 9.28, 9.39
[22:52:05] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.75, 4.04, 3.53
[22:54:51] <icinga-miraheze>	 PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.99, 3.58, 3.44
[22:57:37] <icinga-miraheze>	 RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.80, 2.81, 3.16
[22:59:04] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.42, 9.79, 9.77
[22:59:23] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.99, 9.14, 9.88
[23:00:01] <MirahezeLSBot>	 !log [@test101] starting deploy of {'l10nupdate': True} to skip
[23:00:02] <MirahezeLSBot>	 !log [@mwtask111] starting deploy of {'l10nupdate': True} to scsvg
[23:00:09] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:00:14] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:01:54] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.57, 9.80, 9.79
[23:07:29] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.58, 9.45, 9.50
[23:09:22] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.68, 8.76, 9.57
[23:10:25] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.24, 9.51, 9.52
[23:12:18] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.00, 10.33, 10.03
[23:13:18] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 11.64, 10.31, 9.83
[23:15:10] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.60, 9.81, 9.90
[23:15:23] <icinga-miraheze>	 PROBLEM - mwtask111 Current Load on mwtask111 is CRITICAL: CRITICAL - load average: 2.21, 1.97, 1.57
[23:16:14] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.94, 9.37, 9.54
[23:18:07] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.60, 9.76, 9.83
[23:19:48] <MirahezeLSBot>	 !log [@test101] finished deploy of {'l10nupdate': True} to skip - SUCCESS in 1187s
[23:19:56] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:21:04] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.56, 9.09, 9.55
[23:21:39] <MirahezeLSBot>	 !log [@mwtask111] finished deploy of {'l10nupdate': True} to scsvg - SUCCESS in 1296s
[23:21:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[23:22:19] <icinga-miraheze>	 RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.26, 8.08, 8.46
[23:22:55] <icinga-miraheze>	 RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.12, 7.93, 8.45
[23:26:34] <icinga-miraheze>	 PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.27, 10.12, 9.79
[23:27:49] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.50, 10.11, 9.57
[23:28:42] <icinga-miraheze>	 RECOVERY - www.wikimicrofinanza.it - reverse DNS on sslhost is OK: SSL OK - www.wikimicrofinanza.it reverse DNS resolves to cp20.miraheze.org - CNAME OK
[23:30:42] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.24, 9.39, 9.39
[23:32:32] <icinga-miraheze>	 ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[23:34:22] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.44, 9.31, 8.90
[23:35:32] <icinga-miraheze>	 alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki
[23:36:23] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.74, 10.21, 9.61
[23:39:22] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.73, 9.89, 9.59
[23:42:19] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.09, 10.55, 9.88
[23:43:12] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.55, 9.11, 8.55
[23:49:02] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 10.47, 9.77, 9.29
[23:51:06] <icinga-miraheze>	 PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.33, 9.78, 9.92
[23:51:52] <icinga-miraheze>	 PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.43, 9.70, 9.35
[23:54:42] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 11.67, 9.54, 8.96
[23:57:29] <icinga-miraheze>	 PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.84, 9.04, 8.85