[00:04:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.11, 3.65, 3.73 [00:05:57] PROBLEM - psychevos.org - LetsEncrypt on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [00:06:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.81, 3.29, 3.56 [00:08:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.92, 4.75, 4.08 [00:10:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.33, 3.84, 3.81 [00:12:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.28, 4.75, 4.15 [00:13:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.34, 18.90, 20.36 [00:14:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.28, 3.83, 3.87 [00:18:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.68, 5.03, 4.25 [00:22:09] PROBLEM - psychevos.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for psychevos.org could not be found [00:23:33] [02puppet] 07BlankEclair opened pull request 03#3895: T12105: Add hosted.weblate.org to img-src CSP - 13https://github.com/miraheze/puppet/pull/3895 [00:23:38] [02puppet] 07coderabbitai[bot] commented on pull request 03#3895: T12105: Add hosted.weblate.org to img-src CSP - 13https://github.com/miraheze/puppet/pull/3895#issuecomment-2285119735 [00:24:23] [02puppet] 07Universal-Omega closed pull request 03#3895: T12105: Add hosted.weblate.org to img-src CSP - 13https://github.com/miraheze/puppet/pull/3895 [00:24:24] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ca32ef171df2...404323f07408 [00:24:27] [02puppet] 07BlankEclair 03404323f - T12105: Add hosted.weblate.org to img-src CSP (#3895) [00:24:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.21, 3.69, 3.89 [00:26:04] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.59, 21.18, 20.83 [00:30:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.47, 3.62, 3.78 [00:31:51] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.02, 19.43, 20.22 [00:32:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.35, 3.13, 3.58 [00:34:32] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.59, 2.65, 3.36 [00:40:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.43, 3.66, 3.55 [00:42:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.11, 3.82, 3.60 [00:43:26] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.07, 19.27, 19.46 [00:45:21] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.27, 18.59, 19.22 [00:46:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.61, 3.86, 3.71 [00:48:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.63, 4.82, 4.08 [00:50:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.41, 3.94, 3.83 [00:51:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.25, 20.80, 19.81 [00:52:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 3.31, 4.17, 3.96 [00:53:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.58, 22.29, 20.48 [00:55:29] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:55:30] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:56:00] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:58:20] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:59:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.93, 22.44, 21.29 [01:00:29] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 14 minutes ago with 0 failures [01:00:29] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates). [01:02:10] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [01:03:20] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:03:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.67, 19.07, 20.21 [01:08:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 0.23, 2.76, 3.89 [01:12:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.31, 1.35, 3.05 [01:18:04] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.93, 22.03, 20.84 [01:19:59] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.29, 21.31, 20.72 [01:21:55] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.96, 20.08, 20.34 [01:25:47] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.79, 22.34, 21.14 [01:27:43] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.59, 21.20, 20.90 [01:35:25] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.75, 22.05, 21.21 [01:41:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.89, 23.03, 22.09 [02:03:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.29, 21.92, 21.59 [02:05:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.50, 21.74, 21.60 [02:11:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.43, 22.37, 21.65 [02:15:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.17, 21.72, 21.73 [02:27:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.32, 18.29, 19.96 [03:19:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.49, 22.75, 19.33 [03:21:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.38, 20.71, 19.04 [03:23:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.58, 19.21, 18.72 [05:10:32] PROBLEM - kodiak.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kodiak.wiki' expires in 15 day(s) (Thu 29 Aug 2024 04:47:47 AM GMT +0000). [05:10:43] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e9a67c44c297...2b7bfab67121 [05:10:44] [02ssl] 07WikiTideSSLBot 032b7bfab - Bot: Update SSL cert for kodiak.wiki [05:19:19] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.06, 4.23, 2.70 [05:21:15] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.94, 3.48, 2.59 [05:23:11] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.18, 2.90, 2.48 [05:27:45] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [05:29:00] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.75, 4.94, 3.41 [05:29:42] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [05:30:20] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:31:49] [02RemovePII] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-codesniffer-44.0.0 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/commit/b7d207c2f77d [05:31:50] [02RemovePII] 07dependabot[bot] 03b7d207c - Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 [05:31:52] [02RemovePII] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-44.0.0 - 13https://github.com/miraheze/RemovePII [05:31:54] [02RemovePII] 07dependabot[bot] opened pull request 03#89: Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 - 13https://github.com/miraheze/RemovePII/pull/89 [05:31:57] [02RemovePII] 07dependabot[bot] labeled pull request 03#89: Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 - 13https://github.com/miraheze/RemovePII/pull/89 [05:32:00] [02RemovePII] 07dependabot[bot] labeled pull request 03#89: Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 - 13https://github.com/miraheze/RemovePII/pull/89 [05:32:02] [02RemovePII] 07coderabbitai[bot] commented on pull request 03#89: Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 - 13https://github.com/miraheze/RemovePII/pull/89#issuecomment-2285370768 [05:34:45] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.99, 3.95, 3.50 [05:35:20] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:36:39] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.26, 4.65, 3.79 [05:39:33] miraheze/RemovePII - dependabot[bot] the build has errored. [05:42:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.21, 3.59, 3.66 [05:46:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.85, 2.90, 3.33 [05:52:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.63, 3.42, 3.44 [05:54:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.83, 2.93, 3.27 [06:00:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.60, 3.40, 3.46 [06:04:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.84, 2.81, 3.20 [06:10:11] RECOVERY - kodiak.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kodiak.wiki' will expire on Mon 11 Nov 2024 04:12:08 AM GMT +0000. [06:17:14] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.45, 4.48, 3.69 [06:17:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.52, 18.44, 16.49 [06:19:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.97, 21.50, 17.81 [06:21:05] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.68, 3.65, 3.54 [06:23:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.88, 21.92, 18.99 [06:24:55] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.57, 4.01, 3.70 [06:27:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.43, 19.30, 18.63 [06:34:02] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 146.12 ms [06:36:03] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 137.79 ms [06:42:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.76, 3.32, 3.79 [06:46:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.18, 4.23, 3.99 [06:50:20] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:55:20] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:55:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 29.05, 21.62, 18.67 [06:59:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.70, 21.52, 19.45 [07:00:20] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:01:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.44, 19.46, 18.93 [07:01:45] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [07:03:46] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 6.150 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [07:08:06] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [07:10:00] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.064 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [07:10:20] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:14:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 0.09, 1.92, 3.83 [07:16:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.08, 1.34, 3.38 [08:01:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.14, 20.20, 17.87 [08:03:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.79, 18.50, 17.56 [08:25:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.79, 20.68, 18.92 [08:27:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.33, 19.73, 18.78 [09:20:12] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.29, 20.35, 18.19 [09:20:40] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [09:20:47] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:21:09] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.69, 5.28, 3.27 [09:22:08] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.15, 23.11, 19.44 [09:22:35] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.064 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [09:22:44] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [09:24:58] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.55, 3.65, 3.08 [09:26:53] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.51, 3.10, 2.95 [09:27:55] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.74, 23.14, 20.86 [09:30:45] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.30, 3.74, 3.27 [09:31:46] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.63, 20.00, 20.05 [09:32:39] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.69, 3.78, 3.31 [09:34:36] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.43, 3.31, 3.17 [09:36:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 1.99, 3.12, 3.15 [09:59:04] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.17, 4.34, 3.66 [09:59:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.54, 21.43, 19.58 [10:00:04] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:00:05] PROBLEM - prometheus151 Puppet on prometheus151 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [10:00:42] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:01:20] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:01:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.26, 19.19, 18.97 [10:02:37] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.313 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [10:02:41] RECOVERY - prometheus151 Puppet on prometheus151 is OK: OK: Puppet is currently enabled, last run 16 minutes ago with 0 failures [10:02:41] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 51 packages available for upgrade (0 critical updates). [10:06:20] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:18:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.12, 3.41, 3.93 [10:24:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.15, 3.75, 3.84 [10:26:33] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.76, 3.34, 3.66 [10:30:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.26, 3.93, 3.78 [10:33:08] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [10:36:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.26, 3.53, 3.84 [10:39:16] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 4.823 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [10:40:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.38, 3.80, 3.87 [10:44:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.96, 3.20, 3.62 [10:46:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.63, 4.61, 4.08 [10:47:31] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:48:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.08, 3.83, 3.88 [10:52:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.11, 3.94, 3.87 [10:54:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.57, 3.68, 3.80 [10:57:31] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:58:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.71, 4.68, 4.10 [11:03:20] [Grafana] FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:06:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.05, 3.57, 3.95 [11:08:20] [Grafana] RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:10:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.05, 1.62, 3.05 [11:16:58] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:19:40] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.20, 22.64, 20.30 [11:21:35] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.86, 22.01, 20.39 [11:27:22] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.56, 19.89, 20.00 [11:40:12] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.66, 20.61, 20.05 [11:42:08] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.89, 18.99, 19.53 [11:45:56] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:16:46] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:18:05] PROBLEM - phorge171 php-fpm on phorge171 is CRITICAL: PROCS CRITICAL: 0 processes with command name 'php-fpm8.2' [12:18:59] PROBLEM - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 328 bytes in 0.013 second response time [12:19:31] PROBLEM - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 502 Bad Gateway [12:23:44] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:27:49] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001133173704 secs [12:38:59] RECOVERY - phorge171 issue-tracker.miraheze.org HTTPS on phorge171 is OK: HTTP OK: HTTP/1.1 200 OK - 19644 bytes in 0.058 second response time [12:39:31] RECOVERY - phorge171 phorge-static.wikitide.net HTTPS on phorge171 is OK: HTTP OK: Status line output matched "HTTP/1.1 200" - 17718 bytes in 0.051 second response time [12:40:05] RECOVERY - phorge171 php-fpm on phorge171 is OK: PROCS OK: 9 processes with command name 'php-fpm8.2' [12:43:42] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [13:00:26] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.08, 20.13, 18.50 [13:04:17] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 20.17, 20.18, 18.91 [13:04:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.87, 3.07, 1.20 [13:04:32] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [13:06:27] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.209 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [13:08:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.89, 2.90, 1.66 [13:10:53] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [13:13:59] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.60, 20.23, 19.50 [13:15:54] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.35, 21.89, 20.18 [13:19:45] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.21, 21.15, 20.33 [13:27:27] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.54, 24.04, 21.75 [13:33:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.52, 23.90, 22.55 [13:35:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.15, 25.41, 23.28 [13:39:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.87, 22.51, 22.68 [13:40:53] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for franchise.franchising.org.ua could not be found [13:45:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.29, 17.47, 20.33 [13:56:12] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.55, 22.32, 21.15 [14:00:03] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.53, 22.94, 21.71 [14:03:55] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 12.94, 18.25, 20.11 [14:15:30] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.91, 20.26, 20.17 [14:15:57] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:32:38] [02phorge-extensions] 07Universal-Omega closed pull request 03#49: Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 - 13https://github.com/miraheze/phorge-extensions/pull/49 [14:32:40] [02phorge-extensions] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/phorge-extensions/compare/e26887609042...b82dfa080ef6 [14:32:42] [02phorge-extensions] 07dependabot[bot] 03b82dfa0 - Update mediawiki/mediawiki-codesniffer requirement from 43.0.0 to 44.0.0 (#49) [14:32:44] [02phorge-extensions] 07Universal-Omega deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-44.0.0 [14:32:45] [02phorge-extensions] 07Universal-Omega deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-44.0.0 - 13https://github.com/miraheze/phorge-extensions [14:33:26] miraheze/phorge-extensions - Universal-Omega the build passed. [14:35:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.77, 18.97, 20.16 [15:00:41] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:46] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -4.681944847e-05 secs [15:32:15] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.58, 21.18, 20.34 [15:34:11] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.52, 21.18, 20.44 [15:41:53] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.17, 20.26, 20.39 [15:46:11] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:51:30] PROBLEM - ru-teirailway.f5.si - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ru-teirailway.f5.si' expires in 15 day(s) (Thu 29 Aug 2024 03:28:32 PM GMT +0000). [15:51:42] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2b7bfab67121...ea0e50ff08d8 [15:51:45] [02ssl] 07WikiTideSSLBot 03ea0e50f - Bot: Update SSL cert for ru-teirailway.f5.si [15:57:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.96, 19.93, 18.96 [16:03:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.43, 23.12, 20.53 [16:15:05] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:17:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.00, 23.06, 23.14 [16:21:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.06, 22.19, 22.67 [16:23:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.58, 22.26, 22.64 [17:03:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.03, 22.02, 23.34 [17:06:32] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [17:06:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.62, 3.92, 1.76 [17:06:36] PROBLEM - prometheus151 SSH on prometheus151 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:07:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.57, 23.49, 23.70 [17:08:27] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.068 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [17:08:33] RECOVERY - prometheus151 SSH on prometheus151 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u3 (protocol 2.0) [17:09:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.69, 22.23, 23.19 [17:11:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.39, 23.04, 23.36 [17:13:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.65, 22.85, 23.25 [17:13:58] [02mw-config] 07songnguxyz opened pull request 03#5636: T12394: Install UserProfileV2 - 13https://github.com/miraheze/mw-config/pull/5636 [17:13:59] [02mediawiki-repos] 07songnguxyz opened pull request 03#31: T12394: Install UserProfileV2 - 13https://github.com/miraheze/mediawiki-repos/pull/31 [17:14:06] [02mediawiki-repos] 07coderabbitai[bot] commented on pull request 03#31: T12394: Install UserProfileV2 - 13https://github.com/miraheze/mediawiki-repos/pull/31#issuecomment-2286735922 [17:14:08] [02mediawiki-repos] 07songnguxyz synchronize pull request 03#31: T12394: Install UserProfileV2 - 13https://github.com/miraheze/mediawiki-repos/pull/31 [17:15:06] miraheze/mw-config - songnguxyz the build passed. [17:15:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.30, 23.85, 23.59 [17:16:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.65, 3.83, 2.88 [17:17:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.54, 21.96, 22.93 [17:18:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.48, 4.76, 3.33 [17:24:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.46, 3.95, 3.46 [17:25:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.91, 20.44, 21.45 [17:26:32] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 3.32, 3.40, 3.30 [17:27:20] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.39, 20.14, 21.20 [17:28:17] [02mw-config] 07songnguxyz commented on pull request 03#5636: T12394: Install UserProfileV2 - 13https://github.com/miraheze/mw-config/pull/5636#issuecomment-2286762245 [17:28:23] [02mw-config] 07coderabbitai[bot] commented on pull request 03#5636: T12394: Install UserProfileV2 - 13https://github.com/miraheze/mw-config/pull/5636#issuecomment-2286762552 [17:28:31] [02mediawiki-repos] 07songnguxyz commented on pull request 03#31: T12394: Install UserProfileV2 - 13https://github.com/miraheze/mediawiki-repos/pull/31#issuecomment-2286762926 [17:28:35] [02mediawiki-repos] 07coderabbitai[bot] commented on pull request 03#31: T12394: Install UserProfileV2 - 13https://github.com/miraheze/mediawiki-repos/pull/31#issuecomment-2286763082 [17:28:36] [02mediawiki-repos] 07coderabbitai[bot] edited a comment on pull request 03#31: T12394: Install UserProfileV2 - 13https://github.com/miraheze/mediawiki-repos/pull/31#issuecomment-2286735922 [17:29:08] [02mw-config] 07coderabbitai[bot] edited pull request 03#5636: T12394: Install UserProfileV2 - 13https://github.com/miraheze/mw-config/pull/5636 [17:29:20] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.15, 22.43, 21.92 [17:29:34] [02mw-config] 07coderabbitai[bot] edited a comment on pull request 03#5636: T12394: Install UserProfileV2 - 13https://github.com/miraheze/mw-config/pull/5636#issuecomment-2286762552 [17:30:34] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 7.97, 5.19, 3.96 [17:30:39] [02mediawiki-repos] 07coderabbitai[bot] edited pull request 03#31: T12394: Install UserProfileV2 - 13https://github.com/miraheze/mediawiki-repos/pull/31 [17:30:43] [02mediawiki-repos] 07coderabbitai[bot] edited a comment on pull request 03#31: T12394: Install UserProfileV2 - 13https://github.com/miraheze/mediawiki-repos/pull/31#issuecomment-2286735922 [17:38:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.07, 3.43, 3.64 [17:40:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.23, 4.22, 3.90 [17:42:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.40, 3.61, 3.73 [17:44:17] RECOVERY - wiki.gowashtenaw.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gowashtenaw.org' will expire on Mon 11 Nov 2024 03:46:41 PM GMT +0000. [17:46:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 6.33, 4.32, 3.93 [17:47:27] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:50:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.86, 3.97, 3.93 [17:52:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.43, 4.13, 3.96 [17:54:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.53, 3.40, 3.72 [17:58:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.04, 3.82, 3.78 [18:00:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.47, 3.38, 3.64 [18:02:20] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [18:04:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.21, 3.62, 3.68 [18:10:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.86, 3.15, 3.55 [18:13:55] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 54 seconds ago with 0 failures [18:16:32] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.54, 3.78, 3.65 [18:18:11] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [18:22:11] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.056 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [18:29:33] [02mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/049db98ce5c0...02123783ceca [18:29:36] [02mw-config] 07Universal-Omega 030212378 - Enable wgSVGNativeRendering on a wiki [18:30:31] miraheze/mw-config - Universal-Omega the build passed. [18:30:32] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.76, 3.75, 3.97 [18:32:31] !log [@mwtask181] starting deploy of {'config': True} to all [18:32:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:32:45] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 14s [18:32:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:34:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.24, 4.02, 4.01 [18:36:31] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.43, 3.47, 3.79 [18:37:44] !log [@mwtask171] starting deploy of {'config': True} to all [18:37:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:37:56] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 11s [18:38:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:44:31] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.46, 3.39, 3.57 [18:46:31] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 2.81, 2.97, 3.38 [18:47:20] [Grafana] FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [18:53:45] !log [@test151] starting deploy of {'config': True} to test151 [18:53:46] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [18:53:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:58:15] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.23, 3.78, 3.40 [19:00:09] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.83, 3.99, 3.53 [19:02:04] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 5.17, 4.16, 3.63 [19:07:49] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 2.17, 3.78, 3.71 [19:09:44] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 4.92, 4.18, 3.86 [19:11:02] PROBLEM - prometheus151 PowerDNS Recursor on prometheus151 is CRITICAL: CRITICAL - Plugin timed out while executing system call [19:12:57] RECOVERY - prometheus151 PowerDNS Recursor on prometheus151 is OK: DNS OK: 0.218 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [19:18:04] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o