[00:02:16] PROBLEM - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:15:05] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 48488MiB (10% inode=99%); [00:16:16] RECOVERY - cp37 HTTP 4xx/5xx ERROR Rate on cp37 is OK: OK - NGINX Error Rate is 2% [00:17:05] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 51351MiB (11% inode=99%); [00:29:15] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 31.59 ms [00:31:13] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.58 ms [01:11:05] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49819MiB (10% inode=99%); [02:10:09] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [02:12:07] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.57 ms [02:36:12] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [02:42:20] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.58 ms [03:03:46] PROBLEM - mon181 Backups Grafana on mon181 is WARNING: FILE_AGE WARNING: /var/log/grafana-backup.log is 1382602 seconds old and 92 bytes [03:18:30] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 80%, RTA = 31.54 ms [03:22:34] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.60 ms [03:36:45] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [03:42:54] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.55 ms [04:52:14] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [04:54:12] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.59 ms [06:24:30] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [06:26:28] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 79.14 ms [06:30:45] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 80.25 ms [06:34:51] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 78.55 ms [06:39:01] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 76.76 ms [06:42:59] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 82.94 ms [06:47:15] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 79.72 ms [06:55:27] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 75.38 ms [07:54:49] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 70%, RTA = 76.04 ms [07:58:52] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 74.88 ms [08:02:57] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 83.39 ms [08:04:56] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 81.12 ms [08:09:06] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 73.79 ms [08:13:09] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 76.71 ms [08:31:21] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 74.10 ms [08:35:24] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 74.59 ms [08:39:43] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [08:49:57] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 63.11 ms [10:06:42] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 47.00 ms [10:12:51] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 39.32 ms [10:24:26] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 70%, RTA = 47.12 ms [10:32:37] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 42.79 ms [12:03:33] PROBLEM - neurologiahpm.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'neurologiahpm.com' expires in 14 day(s) (Tue 25 Feb 2025 11:32:58 AM GMT +0000). [12:03:44] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/af097e44f4692ee3fa65aaab1caba28d4ce9bd3b [12:03:44] 02ssl/03master 07WikiTideSSLBot 03af097e4 Bot: Update SSL cert for neurologiahpm.com [12:08:12] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 10.65, 6.09, 2.91 [12:10:02] [02ErrorPages] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/ErrorPages/commit/c775c0a9a3f6e9f8e09a79dccf117edaf342ff24 [12:10:02] 02ErrorPages/03master 07translatewiki.net 03c775c0a Localisation updates from https://translatewiki.net. [12:10:04] [02ImportDump] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/ImportDump/commit/285240634b280301e2854cbd51db2457bb519119 [12:10:05] 02ImportDump/03master 07translatewiki.net 032852406 Localisation updates from https://translatewiki.net. [12:10:07] [02DataDump] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/DataDump/commit/449d6868bfce4f7122183d8c5dbbd71a6b438ac9 [12:10:09] [02CreateWiki] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/CreateWiki/commit/0579345a95b1b083b250145e2ba9e2ade34f6baa [12:10:11] 02CreateWiki/03master 07translatewiki.net 030579345 Localisation updates from https://translatewiki.net. [12:10:12] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 20.88, 11.60, 5.32 [12:10:13] [02RequestSSL] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/RequestSSL/commit/6c50e2c188647ea714f1b8f48fb3526c4fbd151b [12:10:15] [02RottenLinks] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/RottenLinks/commit/b3c3a9a0055d787b1fbc5ddc99a3e0ecd473f5a1 [12:10:17] [02ManageWiki] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/ManageWiki/commit/e216a202c270824ae41cb89fb9d73d995c4d8049 [12:10:19] 02ManageWiki/03master 07translatewiki.net 03e216a20 Localisation updates from https://translatewiki.net. [12:10:21] [02WikiDiscover] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/WikiDiscover/commit/f5f902aab0bb74977f9c7a9869e9ff53282bce12 [12:10:22] 02DataDump/03master 07translatewiki.net 03449d686 Localisation updates from https://translatewiki.net. [12:10:24] [02MirahezeMagic] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/MirahezeMagic/commit/d55e8c584d9e5aefac2cff8a704e9f48c13fb02e [12:10:26] [02MatomoAnalytics] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/MatomoAnalytics/commit/ba190f0682c434453618096208d554dccd8633b0 [12:10:28] 02MatomoAnalytics/03master 07translatewiki.net 03ba190f0 Localisation updates from https://translatewiki.net. [12:10:29] 02RottenLinks/03master 07translatewiki.net 03b3c3a9a Localisation updates from https://translatewiki.net. [12:10:31] [02YouTube] 07translatewiki pushed 1 new commit to 03master 13https://github.com/miraheze/YouTube/commit/6daa530dc1fb57ce56399d9ee7f16eaf1283f768 [12:10:33] 02WikiDiscover/03master 07translatewiki.net 03f5f902a Localisation updates from https://translatewiki.net. [12:10:35] 02MirahezeMagic/03master 07translatewiki.net 03d55e8c5 Localisation updates from https://translatewiki.net. [12:10:36] 02RequestSSL/03master 07translatewiki.net 036c50e2c Localisation updates from https://translatewiki.net. [12:10:38] 02YouTube/03master 07translatewiki.net 036daa530 Localisation updates from https://translatewiki.net. [12:12:59] miraheze/ErrorPages - translatewiki the build passed. [12:13:16] miraheze/RottenLinks - translatewiki the build has errored. [12:14:21] miraheze/MirahezeMagic - translatewiki the build has errored. [12:14:44] miraheze/YouTube - translatewiki the build passed. [12:14:47] miraheze/DataDump - translatewiki the build passed. [12:14:56] miraheze/WikiDiscover - translatewiki the build passed. [12:15:24] miraheze/MatomoAnalytics - translatewiki the build passed. [12:15:28] miraheze/ManageWiki - translatewiki the build passed. [12:22:03] miraheze/ImportDump - translatewiki the build passed. [12:22:10] miraheze/RequestSSL - translatewiki the build passed. [12:22:12] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 4.45, 9.48, 9.00 [12:22:30] miraheze/CreateWiki - translatewiki the build passed. [12:26:13] PROBLEM - db171 Current Load on db171 is CRITICAL: LOAD CRITICAL - total load average: 42.99, 22.66, 13.96 [12:32:06] RECOVERY - neurologiahpm.com - LetsEncrypt on sslhost is OK: OK - Certificate 'neurologiahpm.com' will expire on Sun 11 May 2025 11:05:08 AM GMT +0000. [12:57:20] [Grafana] FIRING: The mediawiki job queue has more than 2000 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:02:20] [Grafana] RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:06:15] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [13:12:12] PROBLEM - db171 Current Load on db171 is WARNING: LOAD WARNING - total load average: 0.34, 2.27, 10.61 [13:14:12] RECOVERY - db171 Current Load on db171 is OK: LOAD OK - total load average: 0.56, 1.72, 9.39 [13:18:33] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.59 ms [13:56:54] PROBLEM - matomo151 HTTPS on matomo151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 7 - Failed to connect to matomo151.wikitide.net port 443 after 0 ms: Couldn't connect to server [14:23:36] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 42% [14:25:35] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 3% [14:26:47] RECOVERY - matomo151 HTTPS on matomo151 is OK: HTTP OK: HTTP/2 200 - 553 bytes in 0.206 second response time [14:39:05] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 27167MiB (5% inode=99%); [14:50:56] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 70%, RTA = 31.64 ms [14:52:55] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.59 ms [16:14:24] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 44%, RTA = 31.56 ms [16:18:27] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.59 ms [16:53:28] [02ssl] 07MacFan4000 pushed 1 new commit to 03master 13https://github.com/miraheze/ssl/commit/3798ee02370d77b4f53452c5a695836f343fec62 [16:53:28] 02ssl/03master 07MacFan4000 033798ee0 add niedersachsen.bezahlkarte-nein.de [17:50:11] PROBLEM - changeprop151 Current Load on changeprop151 is CRITICAL: LOAD CRITICAL - total load average: 29.96, 19.78, 10.10 [17:54:11] RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 3.07, 11.73, 9.11 [18:52:39] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [19:02:58] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.56 ms [19:07:21] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [19:23:50] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.55 ms [21:10:11] PROBLEM - changeprop151 Current Load on changeprop151 is WARNING: LOAD WARNING - total load average: 9.69, 14.71, 8.17 [21:12:11] RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 3.52, 10.69, 7.49 [22:37:00] [02phorge-extensions] 07Universal-Omega merged 07dependabot[bot]'s pull request #52: Update mediawiki/mediawiki-codesniffer requirement from 45.0.0 to 46.0.0 (03master...03dependabot/composer/mediawiki/mediawiki-codesniffer-46.0.0) 13https://github.com/miraheze/phorge-extensions/pull/52 [22:37:00] [02phorge-extensions] 07Universal-Omega pushed 1 new commit to 03master 13https://github.com/miraheze/phorge-extensions/commit/6d5ee88b94578b3cf0b82cc2e7a24a72b79cae6d [22:37:02] 02phorge-extensions/03master 07dependabot[bot] 036d5ee88 Update mediawiki/mediawiki-codesniffer requirement from 45.0.0 to 46.0.0 (#52)… [22:37:04] [02phorge-extensions] 07Universal-Omega 04deleted 03dependabot/composer/mediawiki/mediawiki-codesniffer-46.0.0 at 036587f05 13https://github.com/miraheze/phorge-extensions/commit/6587f05 [22:37:46] miraheze/phorge-extensions - Universal-Omega the build passed. [22:53:28] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100% [23:01:39] RECOVERY - ping6 on mattermost1 is OK: PING OK - Packet loss = 0%, RTA = 31.61 ms [23:41:21] [02python-functions] 07dependabot[bot] created 03dependabot/pip/dot-github/coverage-7.6.11 (+1 new commit) 13https://github.com/miraheze/python-functions/commit/7ceca6d4b687 [23:41:21] 02python-functions/03dependabot/pip/dot-github/coverage-7.6.11 07dependabot[bot] 037ceca6d Bump coverage from 7.6.10 to 7.6.11 in /.github… [23:41:22] [02python-functions] 07dependabot[bot] opened pull request #117: Bump coverage from 7.6.10 to 7.6.11 in /.github (03master...03dependabot/pip/dot-github/coverage-7.6.11) 13https://github.com/miraheze/python-functions/pull/117 [23:41:22] [02python-functions] 07dependabot[bot] added the label 'python' to pull request #117 (Bump coverage from 7.6.10 to 7.6.11 in /.github) 13https://github.com/miraheze/python-functions/pull/117 [23:41:24] [02python-functions] 07dependabot[bot] added the label 'dependencies' to pull request #117 (Bump coverage from 7.6.10 to 7.6.11 in /.github) 13https://github.com/miraheze/python-functions/pull/117 [23:41:29] [02python-functions] 07coderabbitai[bot] commented on pull request #117: --- […] 13https://github.com/miraheze/python-functions/pull/117#issuecomment-2649502090 [23:43:35] miraheze/python-functions - dependabot[bot] the build has errored. [23:44:46] PROBLEM - ping6 on mattermost1 is CRITICAL: PING CRITICAL - Packet loss = 100%