[00:02:52] RECOVERY - matomo101 Disk Space on matomo101 is OK: DISK OK - free space: / 2024 MB (21% inode=80%); [00:05:11] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:05:18] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 303 failures. Last run 3 minutes ago with 303 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg],File[/usr/local/bin/puppet-enabled],File[authority certificates],File[/etc/apt/apt.conf.d/50unattended-upgrades] [00:07:10] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 3.871 second response time [00:07:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.97, 2.94, 3.37 [00:11:23] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:13:19] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.724 second response time [00:16:27] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.81, 6.74, 5.78 [00:18:25] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.66, 6.04, 5.64 [00:19:43] PROBLEM - mw112 MediaWiki Rendering on mw112 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:21:40] RECOVERY - mw112 MediaWiki Rendering on mw112 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.852 second response time [00:23:20] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [00:25:45] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:25:46] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:26:15] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:27:28] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:27:30] PROBLEM - cp21 Stunnel HTTP for mw111 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:14] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 4.916 second response time [00:29:26] RECOVERY - cp21 Stunnel HTTP for mw111 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.384 second response time [00:29:26] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.306 second response time [00:29:46] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 0.878 second response time [00:29:51] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.319 second response time [00:31:19] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [00:33:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.81, 3.28, 3.05 [00:37:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.30, 2.89, 2.94 [00:42:22] PROBLEM - wiki.wisrail.de - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/lib/p [00:42:22] ist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query wisrail.de. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:46:14] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.51, 3.23, 3.04 [00:47:12] [02mediawiki] 07dependabot[bot] opened pull request 03#6178: Bump extensions/RemovePII from `e068054` to `e9475c7` - 13https://github.com/miraheze/mediawiki/pull/6178 [00:47:13] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/RemovePII-e9475c7 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/215c7aea6064 [00:47:15] [02miraheze/mediawiki] 07dependabot[bot] 03215c7ae - Bump extensions/RemovePII from `e068054` to `e9475c7` [00:47:16] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/RemovePII-e9475c7 - 13https://github.com/miraheze/mediawiki [00:47:18] [02mediawiki] 07dependabot[bot] commented on pull request 03#6160: Bump extensions/RemovePII from `e068054` to `5d3c650` - 13https://github.com/miraheze/mediawiki/pull/6160#issuecomment-1086994564 [00:47:19] [02mediawiki] 07dependabot[bot] closed pull request 03#6160: Bump extensions/RemovePII from `e068054` to `5d3c650` - 13https://github.com/miraheze/mediawiki/pull/6160 [00:47:21] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/RemovePII-5d3c650 - 13https://github.com/miraheze/mediawiki [00:47:22] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/RemovePII-5d3c650 [00:47:27] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/extensions/Translate-400ce54 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/527f693db19d [00:47:28] [02miraheze/mediawiki] 07dependabot[bot] 03527f693 - Bump extensions/Translate from `c67646b` to `400ce54` [00:47:30] [02mediawiki] 07dependabot[bot] opened pull request 03#6179: Bump extensions/Translate from `c67646b` to `400ce54` - 13https://github.com/miraheze/mediawiki/pull/6179 [00:47:31] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/extensions/Translate-400ce54 - 13https://github.com/miraheze/mediawiki [00:47:33] [02mediawiki] 07dependabot[bot] commented on pull request 03#6175: Bump extensions/Translate from `c67646b` to `4f1a5a0` - 13https://github.com/miraheze/mediawiki/pull/6175#issuecomment-1086994682 [00:47:34] [02mediawiki] 07dependabot[bot] closed pull request 03#6175: Bump extensions/Translate from `c67646b` to `4f1a5a0` - 13https://github.com/miraheze/mediawiki/pull/6175 [00:47:36] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Translate-4f1a5a0 - 13https://github.com/miraheze/mediawiki [00:47:37] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/extensions/Translate-4f1a5a0 [00:47:39] [02miraheze/mediawiki] 07dependabot[bot] pushed 031 commit to 03dependabot/submodules/REL1_37/skins/Evelution-c8bf7f4 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/commit/b2a0823904e9 [00:47:40] [02miraheze/mediawiki] 07dependabot[bot] 03b2a0823 - Bump skins/Evelution from `7b9a744` to `c8bf7f4` [00:47:42] [02mediawiki] 07dependabot[bot] opened pull request 03#6180: Bump skins/Evelution from `7b9a744` to `c8bf7f4` - 13https://github.com/miraheze/mediawiki/pull/6180 [00:47:43] [02mediawiki] 07dependabot[bot] created branch 03dependabot/submodules/REL1_37/skins/Evelution-c8bf7f4 - 13https://github.com/miraheze/mediawiki [00:47:45] [02mediawiki] 07dependabot[bot] commented on pull request 03#6167: Bump skins/Evelution from `7b9a744` to `c612127` - 13https://github.com/miraheze/mediawiki/pull/6167#issuecomment-1086994730 [00:47:46] [02mediawiki] 07dependabot[bot] closed pull request 03#6167: Bump skins/Evelution from `7b9a744` to `c612127` - 13https://github.com/miraheze/mediawiki/pull/6167 [00:47:48] [02mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/skins/Evelution-c612127 - 13https://github.com/miraheze/mediawiki [00:47:49] [02miraheze/mediawiki] 07dependabot[bot] deleted branch 03dependabot/submodules/REL1_37/skins/Evelution-c612127 [00:47:51] [02mediawiki] 07github-actions[bot] labeled pull request 03#6178: Bump extensions/RemovePII from `e068054` to `e9475c7` - 13https://github.com/miraheze/mediawiki/pull/6178 [00:47:52] [02mediawiki] 07github-actions[bot] labeled pull request 03#6178: Bump extensions/RemovePII from `e068054` to `e9475c7` - 13https://github.com/miraheze/mediawiki/pull/6178 [00:48:03] [02mediawiki] 07github-actions[bot] labeled pull request 03#6179: Bump extensions/Translate from `c67646b` to `400ce54` - 13https://github.com/miraheze/mediawiki/pull/6179 [00:48:05] [02mediawiki] 07github-actions[bot] labeled pull request 03#6179: Bump extensions/Translate from `c67646b` to `400ce54` - 13https://github.com/miraheze/mediawiki/pull/6179 [00:48:07] [02mediawiki] 07github-actions[bot] labeled pull request 03#6180: Bump skins/Evelution from `7b9a744` to `c8bf7f4` - 13https://github.com/miraheze/mediawiki/pull/6180 [00:50:09] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.29, 3.26, 3.08 [00:51:19] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:54:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.94, 3.82, 3.32 [00:55:53] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [00:55:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.59, 3.28 [00:57:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.08, 3.94, 3.44 [00:58:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.59, 3.67, 3.16 [01:00:52] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.06, 3.89, 3.30 [01:02:32] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.13, 4.12, 3.45 [01:02:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.27, 3.73, 3.32 [01:04:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.67, 3.67, 3.38 [01:05:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.83, 3.88, 3.72 [01:06:31] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.56, 3.34, 3.29 [01:06:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.98, 3.28, 3.28 [01:10:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.38, 3.30 [01:12:13] RECOVERY - wiki.wisrail.de - reverse DNS on sslhost is OK: SSL OK - wiki.wisrail.de reverse DNS resolves to cp21.miraheze.org - CNAME OK [01:12:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.94, 3.18, 3.23 [01:15:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.34, 3.84, 3.67 [01:16:28] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.34, 4.78, 3.95 [01:17:59] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.15, 5.96, 5.66 [01:19:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.53, 3.60 [01:21:58] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.78, 6.01, 5.79 [01:22:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 1.96, 3.52, 3.69 [01:23:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.59, 2.99, 3.36 [01:26:25] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.35, 3.85, 3.77 [01:28:25] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.28, 3.53, 3.65 [01:28:50] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 12.21, 8.66, 6.91 [01:30:48] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.95, 7.75, 6.78 [01:32:24] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.60, 2.99, 3.39 [01:34:00] PROBLEM - cp30 Stunnel HTTP for mw111 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:34:02] PROBLEM - cp31 Stunnel HTTP for mw102 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:34:11] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:34:12] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:35:58] RECOVERY - cp30 Stunnel HTTP for mw111 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.329 second response time [01:36:02] RECOVERY - cp31 Stunnel HTTP for mw102 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.306 second response time [01:36:06] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 0.023 second response time [01:36:08] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22337 bytes in 1.718 second response time [01:36:40] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.86, 6.66, 6.63 [01:38:35] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:38:38] PROBLEM - cp21 Stunnel HTTP for mw121 on cp21 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:38:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.45, 3.23, 3.00 [01:39:54] PROBLEM - cp31 Stunnel HTTP for mw121 on cp31 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.248 second response time [01:39:58] PROBLEM - cp30 Stunnel HTTP for mw121 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:39:58] PROBLEM - cp20 Stunnel HTTP for mw121 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:40:54] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.26, 3.38, 3.09 [01:42:55] RECOVERY - cp21 Stunnel HTTP for mw121 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 9.829 second response time [01:44:12] RECOVERY - cp31 Stunnel HTTP for mw121 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 8.357 second response time [01:44:14] RECOVERY - cp30 Stunnel HTTP for mw121 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.357 second response time [01:44:14] RECOVERY - cp20 Stunnel HTTP for mw121 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14562 bytes in 7.129 second response time [01:44:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.59, 3.75, 3.32 [01:46:45] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.25, 3.71, 3.34 [01:47:10] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 22337 bytes in 7.634 second response time [01:48:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.33, 3.17, 3.19 [02:28:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.60, 3.20, 2.67 [02:30:52] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.25, 3.00, 2.68 [02:46:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.18, 3.68, 3.35 [02:50:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.50, 3.31, 3.29 [03:04:01] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:06:47] SRE, I'm getting alternating 502s and 503s on `metawiki`; persistent for the past 10 minutes [03:07:37] Seems to be better, for now [03:12:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:15:52] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.90, 3.72, 3.13 [03:19:50] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.26, 3.92, 3.34 [03:21:50] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.03, 3.50, 3.25 [03:26:35] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 7.37, 7.33, 6.69 [03:27:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.25, 3.58, 3.35 [03:28:33] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 9.00, 7.85, 6.95 [03:30:30] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.41, 7.34, 6.88 [03:32:27] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.35, 6.32, 6.56 [03:32:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.36, 3.45, 3.16 [03:34:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.92, 3.97, 3.38 [03:35:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.40, 3.57, 3.41 [03:36:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.96, 3.17, 3.16 [03:37:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.14, 3.37, 3.33 [03:39:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.87, 3.03, 3.20 [03:40:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.73, 4.20, 3.60 [03:42:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.15, 3.79, 3.53 [03:46:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.27, 4.11, 3.68 [03:48:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.47, 3.63, 3.54 [03:52:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.15, 4.05, 3.76 [03:54:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.74, 3.67 [03:58:04] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [03:58:04] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:58:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.01, 3.06, 3.37 [04:04:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.59, 3.47 [04:06:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.83, 3.59 [04:07:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:08:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.62, 4.15, 3.74 [04:10:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.92, 3.65, 3.61 [04:16:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.03, 3.00, 3.34 [04:20:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:27:38] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [04:49:41] dmehus: still having issues, or resolved? [04:49:42] [tell] CosmicAlpha: 2022-04-02 - 19:29:16UTC tell CosmicAlpha in future version we might not be able to use isRegistered [04:50:01] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:51:16] .tell RhinosF1 why can't we use isRegistred? It's the only one that does the stuff that isn't deprecated or planned to be deprecated, as I've heard somewhere isAnon is also going to be deprecated, leaving only isRegistered. [04:51:16] CosmicAlpha: I'll pass that on when RhinosF1 is around. [04:55:10] CosmicAlpha, it seems better. Was suffering from extreme load and time outs I think [04:55:51] also re: ^ that's interesting that isAnon might be deprecated [04:55:58] PROBLEM - db101 Current Load on db101 is CRITICAL: CRITICAL - load average: 8.24, 7.26, 6.19 [04:57:58] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.89, 7.16, 6.29 [04:58:45] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.69, 2.12, 1.34 [04:59:06] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [04:59:06] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:00:44] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.01, 1.67, 1.26 [05:02:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.63, 3.41, 2.90 [05:05:58] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.98, 6.63, 6.44 [05:08:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:08:27] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.26, 3.69, 3.24 [05:14:27] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.11, 3.26, 3.21 [05:18:01] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [05:23:23] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.96, 4.27, 3.67 [05:28:40] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [05:29:22] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.41, 3.60, 3.60 [05:35:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.28, 3.10, 3.40 [05:44:24] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.44, 3.09 [05:46:21] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.39, 3.37, 3.10 [05:52:13] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.32, 3.64, 3.46 [05:57:03] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.89, 3.54, 3.28 [05:57:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.57, 3.24, 2.72 [05:58:12] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.83, 3.64 [05:59:16] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.29, 3.36, 2.83 [06:00:11] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.58, 3.97, 3.70 [06:00:57] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.28, 3.14, 3.19 [06:02:11] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.31, 3.25, 3.47 [06:04:10] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.58, 3.04, 3.36 [06:16:16] dmehus: OK, glad it's better. Thanks! [06:20:05] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.57, 3.03, 2.99 [06:22:04] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.85, 2.62, 2.85 [06:24:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:29:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:31:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.34, 7.52, 6.12 [06:35:41] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.75, 7.90, 6.63 [06:37:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.40, 3.21, 2.96 [06:39:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.02, 8.85, 7.66 [06:39:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.71, 2.95, 2.89 [06:40:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.46, 9.15, 7.35 [06:41:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:41:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.07, 9.40, 8.01 [06:41:43] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.84, 8.78, 7.51 [06:43:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.97, 9.15, 8.09 [06:46:01] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:47:39] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.54, 8.49, 7.85 [06:50:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.06, 3.83, 3.29 [06:51:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [06:51:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.08, 9.22, 8.32 [06:52:09] PROBLEM - dragonquestwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dragonquestwiki.com' expires in 15 day(s) (Wed 20 Apr 2022 06:43:30 GMT +0000). [06:52:45] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.32, 3.38, 3.20 [06:53:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.63, 9.69, 8.60 [06:54:21] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d2c7414e7b4a...f37d78bc0f62 [06:54:22] [02miraheze/ssl] 07MirahezeSSLBot 03f37d78b - Bot: Update SSL cert for dragonquestwiki.com [06:55:20] PROBLEM - crocwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'crocwiki.com' expires in 15 day(s) (Wed 20 Apr 2022 06:46:10 GMT +0000). [06:55:22] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.42, 9.72, 8.92 [06:55:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.54, 9.47, 8.65 [06:55:58] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.99, 6.65, 6.29 [06:57:22] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.57, 9.65, 9.00 [06:57:58] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 4.49, 5.87, 6.05 [07:01:47] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f37d78bc0f62...93add5622f29 [07:01:49] [02miraheze/ssl] 07MirahezeSSLBot 0393add56 - Bot: Update SSL cert for crocwiki.com [07:02:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 5.93, 8.54, 9.25 [07:03:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.67, 7.89, 8.36 [07:09:22] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 5.53, 7.16, 8.17 [07:09:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.90, 8.04, 7.12 [07:11:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:13:41] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.38, 8.38, 7.49 [07:14:33] PROBLEM - experimyco.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'experimyco.com' expires in 15 day(s) (Wed 20 Apr 2022 06:47:50 GMT +0000). [07:16:55] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/93add5622f29...6a19642ac994 [07:16:56] [02miraheze/ssl] 07MirahezeSSLBot 036a19642 - Bot: Update SSL cert for experimyco.com [07:18:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.61, 7.78, 8.46 [07:26:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:28:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.82, 3.89, 3.17 [07:28:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.62, 8.99, 8.57 [07:30:36] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.73, 4.06, 3.05 [07:30:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.85, 9.22, 8.70 [07:30:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.09, 3.79, 3.02 [07:31:01] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:31:16] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 10.94, 8.88, 8.31 [07:32:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 10.41, 9.46, 8.84 [07:32:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.25, 3.76, 3.11 [07:33:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.34, 8.09, 8.11 [07:34:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.38, 3.96, 3.25 [07:34:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.34, 4.24, 3.38 [07:36:44] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.25, 3.53, 3.22 [07:38:28] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.04, 3.12, 3.07 [07:40:43] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.16, 3.77, 3.38 [07:42:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.15, 9.46, 9.36 [07:43:37] RECOVERY - experimyco.com - LetsEncrypt on sslhost is OK: OK - Certificate 'experimyco.com' will expire on Sun 03 Jul 2022 06:16:48 GMT +0000. [07:43:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.37, 3.95, 3.91 [07:44:42] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.79, 3.97, 3.56 [07:45:39] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.03, 4.18, 4.01 [07:46:44] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 11.18, 10.07, 9.62 [07:48:41] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.72, 3.87, 3.57 [07:50:25] RECOVERY - dragonquestwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'dragonquestwiki.com' will expire on Sun 03 Jul 2022 05:54:14 GMT +0000. [07:50:40] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.97, 3.76, 3.56 [07:52:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.83, 4.08, 3.69 [07:54:00] RECOVERY - crocwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'crocwiki.com' will expire on Sun 03 Jul 2022 06:01:41 GMT +0000. [07:57:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:58:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.28, 8.55, 7.34 [07:59:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.10, 3.78, 3.96 [07:59:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.13, 3.76, 3.44 [08:01:18] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 3.93, 4.00 [08:01:55] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.75, 3.39, 3.35 [08:03:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.99, 3.60, 3.87 [08:04:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.03, 9.45, 9.89 [08:06:12] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.45, 9.60, 8.34 [08:07:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 3.67, 3.83 [08:08:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.33, 9.50, 8.47 [08:08:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.05, 3.33, 3.78 [08:09:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.04, 3.29, 3.66 [08:10:12] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 10.26, 9.79, 8.70 [08:12:12] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.20, 9.61, 8.78 [08:13:02] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.64, 4.02, 3.83 [08:14:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.30, 6.82, 8.40 [08:14:59] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.93, 3.83 [08:16:12] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 4.60, 7.39, 8.11 [08:16:33] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.33, 2.75, 3.31 [08:18:53] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.14, 4.01, 3.88 [08:20:51] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.20, 3.69, 3.77 [08:26:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.02, 2.84, 3.33 [08:29:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.80, 3.55, 3.24 [08:30:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.44, 3.41, 3.50 [08:31:18] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.66, 3.39, 3.22 [08:32:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:32:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.41, 2.97, 3.32 [08:33:27] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.54, 6.86, 6.26 [08:35:25] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 5.27, 6.27, 6.11 [08:42:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:52:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [08:53:20] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [09:06:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:11:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:15:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.19, 3.79, 3.21 [09:17:43] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.97, 3.31, 3.10 [09:18:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:21:19] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:28:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:33:21] PROBLEM - vedopedia.witches-empire.com - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File [09:33:21] ib/python3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query vedopedia.witches-empire.com. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:45:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [09:50:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [10:03:12] RECOVERY - vedopedia.witches-empire.com - reverse DNS on sslhost is OK: SSL OK - vedopedia.witches-empire.com reverse DNS resolves to cp20.miraheze.org - CNAME OK [10:35:50] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [11:05:23] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [11:06:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:08:59] [02miraheze/IncidentReporting] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/6df0c6572162...47936f4de89e [11:09:01] [02miraheze/IncidentReporting] 07translatewiki 0347936f4 - Localisation updates from https://translatewiki.net. [11:09:02] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/landing/compare/108b18b9bf9c...f7a0ab7d88d2 [11:09:04] [02miraheze/landing] 07translatewiki 03f7a0ab7 - Localisation updates from https://translatewiki.net. [11:09:05] [02miraheze/RemovePII] 07translatewiki pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/RemovePII/compare/e9475c78719b...a0ed5d9a2ef1 [11:09:07] [02miraheze/RemovePII] 07translatewiki 03a0ed5d9 - Localisation updates from https://translatewiki.net. [11:09:08] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/740bb6eff4e3...366a5d49fc65 [11:09:10] [02miraheze/ManageWiki] 07translatewiki 03366a5d4 - Localisation updates from https://translatewiki.net. [11:09:11] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+2/-0/±5] 13https://github.com/miraheze/MirahezeMagic/compare/f43433cce820...5a6a6d955921 [11:09:13] [02miraheze/MirahezeMagic] 07translatewiki 035a6a6d9 - Localisation updates from https://translatewiki.net. [11:11:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:11:43] miraheze/landing - translatewiki the build passed. [11:18:12] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 3.09, 4.83, 2.69 [11:18:55] miraheze/IncidentReporting - translatewiki the build passed. [11:19:00] miraheze/RemovePII - translatewiki the build passed. [11:19:48] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 8.47, 4.66, 3.39 [11:20:00] miraheze/ManageWiki - translatewiki the build passed. [11:23:48] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.86, 3.70, 3.29 [11:25:37] miraheze/MirahezeMagic - translatewiki the build passed. [11:25:48] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.19, 3.15, 3.14 [11:26:12] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.00, 1.71, 1.98 [11:32:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:32:12] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.91, 1.12, 1.62 [11:37:01] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:37:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.64, 3.04, 2.80 [11:39:24] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.09, 3.00, 2.81 [11:54:00] [02puppet] 07RhinosF1 synchronize pull request 03#2456: mwdeploy: async command running - 13https://github.com/miraheze/puppet/pull/2456 [11:56:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.12, 8.49, 7.72 [11:58:52] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.34, 9.09, 8.03 [11:59:27] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.64, 3.37, 2.80 [12:00:00] [02puppet] 07RhinosF1 synchronize pull request 03#2456: mwdeploy: async command running - 13https://github.com/miraheze/puppet/pull/2456 [12:00:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:02:48] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.29, 9.27, 8.39 [12:05:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.73, 8.89, 7.94 [12:05:20] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.66, 3.09, 2.88 [12:06:44] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.26, 9.58, 8.69 [12:06:55] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 10.70, 9.54, 8.29 [12:08:42] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.91, 9.16, 8.64 [12:08:45] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.99, 3.77, 3.23 [12:08:50] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.59, 8.36, 8.00 [12:10:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.46, 3.29, 3.12 [12:12:39] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.43, 7.95, 8.26 [12:14:07] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.47, 3.69, 3.23 [12:14:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.69, 3.83, 3.39 [12:14:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.88, 1.55, 1.05 [12:15:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.77, 3.57, 3.12 [12:16:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.98, 4.06, 3.54 [12:16:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.58, 1.60, 1.13 [12:18:03] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.59, 3.19, 3.13 [12:18:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.35, 3.71, 3.47 [12:19:42] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.61, 3.25, 3.11 [12:20:25] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.67, 8.14, 7.56 [12:20:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.68, 8.12, 8.14 [12:20:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:22:27] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.28, 8.14, 8.15 [12:23:37] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.94, 1.87, 1.41 [12:25:31] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.35, 1.69, 1.41 [12:26:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.74, 3.60, 3.46 [12:27:38] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.44, 3.39, 3.22 [12:28:20] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 11.34, 9.52, 8.35 [12:28:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.26, 2.98, 3.24 [12:29:38] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.75, 3.09, 3.12 [12:30:19] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.51, 9.11, 8.34 [12:30:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:32:35] PROBLEM - db101 Current Load on db101 is WARNING: WARNING - load average: 6.88, 6.77, 6.17 [12:34:16] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.38, 8.28, 8.17 [12:36:30] RECOVERY - db101 Current Load on db101 is OK: OK - load average: 6.33, 6.60, 6.22 [12:36:57] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [12:40:31] ok : [RESOLVED] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:42:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.97, 3.69, 3.28 [12:43:03] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.57, 1.96, 1.41 [12:44:02] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.70, 8.40, 8.08 [12:44:33] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.38, 4.23, 3.42 [12:44:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.62, 4.17, 3.51 [12:45:03] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.04, 1.55, 1.32 [12:46:00] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 6.23, 7.59, 7.83 [12:48:32] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.84, 3.83, 3.49 [12:49:54] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 10.13, 8.95, 8.30 [12:51:52] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.77, 8.76, 8.29 [12:52:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.83, 3.89, 3.56 [12:52:48] PROBLEM - mwtask111 Current Load on mwtask111 is WARNING: WARNING - load average: 1.88, 1.69, 1.53 [12:53:02] alerting : [FIRING:1] (PHP-FPM Worker Usage High yes mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:53:19] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.83, 3.33, 3.02 [12:53:50] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.59, 8.48, 8.25 [12:54:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.14, 3.62, 3.50 [12:54:48] RECOVERY - mwtask111 Current Load on mwtask111 is OK: OK - load average: 1.41, 1.62, 1.53 [12:55:17] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.05, 3.15, 2.98 [12:56:06] RhinosF1, looks like TheBurningPrincess' global rename has failed on `arefepva.miraheze.org` so probably need to run `FixStuckGlobalRename.php` I think [12:56:15] re: -offtopic [12:56:19] and morning [12:56:23] only here briefly [12:56:30] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.69, 4.08, 3.67 [12:57:26] [discord] Hi [12:57:58]