[00:00:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.07, 8.39, 8.10 [00:02:41] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.94, 6.62, 5.98 [00:04:45] RECOVERY - wiki.albrusmusic.ru - reverse DNS on sslhost is OK: SSL OK - wiki.albrusmusic.ru reverse DNS resolves to cp24.miraheze.org - CNAME OK [00:04:47] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.67, 5.85, 5.77 [00:04:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.87, 7.48, 7.84 [00:08:51] PROBLEM - swiftac111 Current Load on swiftac111 is WARNING: WARNING - load average: 7.17, 6.74, 5.42 [00:10:51] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 4.64, 5.99, 5.31 [00:10:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.49, 7.97, 7.92 [00:12:18] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.49, 2.96, 2.11 [00:14:12] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.98, 2.89, 2.18 [00:22:06] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [00:31:30] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.83, 3.29, 2.76 [00:32:10] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.01, 7.53, 6.78 [00:33:30] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1016013622 secs [00:33:30] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.75, 3.08, 2.75 [00:34:09] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.15, 7.00, 6.68 [00:36:09] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.89, 6.55, 6.55 [00:36:52] PROBLEM - wiki.albrusmusic.ru - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [00:37:30] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.87, 3.45, 2.96 [00:39:30] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.85, 3.18, 2.91 [00:42:39] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.31, 3.55, 3.18 [00:44:38] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 1.79, 2.85, 2.97 [00:45:40] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.62, 3.64, 2.82 [00:47:40] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.07, 3.00, 2.68 [00:58:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.20, 6.80, 7.99 [01:00:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.24, 7.18, 7.97 [01:02:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.88, 7.12, 7.85 [01:20:19] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [01:30:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.82, 7.36, 7.16 [01:32:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.23, 6.83, 6.98 [01:35:18] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.41, 7.34, 6.76 [01:36:47] RECOVERY - wiki.albrusmusic.ru - reverse DNS on sslhost is OK: SSL OK - wiki.albrusmusic.ru reverse DNS resolves to cp25.miraheze.org - CNAME OK [01:37:15] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.38, 6.63, 6.57 [01:38:59] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.96, 6.42, 6.75 [01:49:54] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.09, 6.72, 6.52 [01:49:55] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.92, 7.04, 6.87 [01:51:52] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.03, 6.47, 6.46 [01:51:54] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.52, 6.77, 6.78 [01:58:54] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.23, 3.34, 3.01 [02:05:43] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.98, 6.97, 6.75 [02:06:47] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.42, 3.87, 3.46 [02:07:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.02, 7.47, 6.96 [02:08:58] PROBLEM - wiki.albrusmusic.ru - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [02:09:41] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.24, 6.78, 6.76 [02:12:20] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.70, 7.65, 6.92 [02:12:42] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.45, 3.19, 3.30 [02:20:10] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.69, 6.49, 6.74 [02:29:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.89, 6.67, 6.56 [02:31:22] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.20, 6.44, 6.49 [02:32:09] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 9.05, 7.48, 7.00 [02:35:18] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.65, 8.18, 7.15 [02:35:22] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.76, 3.62, 3.35 [02:37:17] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.61, 7.57, 7.06 [02:37:21] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.75, 3.21, 3.23 [02:38:09] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 5.44, 7.33, 7.30 [02:43:13] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.97, 6.43, 6.76 [02:44:10] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.82, 5.83, 6.63 [02:49:08] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.70, 7.11, 6.94 [02:53:05] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.10, 7.53, 7.15 [02:55:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.60, 6.79, 6.92 [03:01:13] RECOVERY - db131 Backups SQL on db131 is OK: FILE_AGE OK: /var/log/sql-backup.log is 71 seconds old and 0 bytes [03:02:27] PROBLEM - mon141 Backups Grafana on mon141 is WARNING: FILE_AGE WARNING: /var/log/grafana-backup.log is 864141 seconds old and 785 bytes [03:02:59] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.28, 6.65, 6.80 [03:08:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.77, 7.32, 7.01 [03:09:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.61, 7.30, 6.91 [03:10:59] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.13, 6.40, 6.70 [03:17:20] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.59, 6.31, 6.64 [03:21:54] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.51, 7.20, 6.91 [03:25:51] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.57, 6.36, 6.64 [03:27:09] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.02, 6.44, 6.39 [03:29:07] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.33, 6.39, 6.38 [03:38:52] RECOVERY - wiki.albrusmusic.ru - reverse DNS on sslhost is OK: SSL OK - wiki.albrusmusic.ru reverse DNS resolves to cp25.miraheze.org - CNAME OK [03:50:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.31, 7.11, 6.72 [03:51:16] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.69, 3.29, 2.86 [03:52:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.01, 6.96, 6.53 [03:54:27] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.95, 6.69, 6.49 [03:55:14] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.35, 2.90, 2.81 [04:00:26] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.79, 6.30, 6.54 [04:04:36] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 122.24 ms [04:10:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [04:29:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.38, 7.18, 6.62 [04:30:57] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.14, 7.56, 6.70 [04:31:03] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.54, 6.55, 6.46 [04:36:49] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.07, 6.80, 6.62 [04:42:00] PROBLEM - wiki.albrusmusic.ru - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.408 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [04:50:28] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.84, 6.82, 6.61 [04:51:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.91, 7.15, 6.80 [04:53:49] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.54, 7.50, 6.96 [04:55:47] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.51, 7.52, 7.04 [04:58:18] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.56, 6.58, 6.64 [05:04:09] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.34, 6.87, 6.70 [05:10:09] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 6.28, 6.76, 6.74 [05:25:27] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.39, 6.37, 6.75 [05:30:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [05:33:21] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.74, 6.92, 6.83 [05:35:19] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.59, 6.46, 6.68 [05:35:20] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.67, 3.56, 3.11 [05:35:33] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.07, 6.82, 6.59 [05:37:18] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.15, 3.03, 2.97 [05:37:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.33, 6.09, 6.36 [05:47:10] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.48, 7.02, 6.63 [05:53:07] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.31, 7.59, 7.06 [05:55:06] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.02, 7.64, 7.13 [05:57:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.76, 7.60, 7.16 [06:04:59] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.43, 6.31, 6.68 [06:11:02] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.90, 6.72, 6.16 [06:16:54] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.50, 6.38, 6.21 [06:16:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.65, 6.81, 6.73 [06:18:14] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 103.72 ms [06:18:59] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.17, 6.60, 6.66 [07:15:36] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [07:20:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.53, 6.91, 6.33 [07:22:26] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.40, 6.64, 6.30 [07:32:43] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.49, 3.17, 2.95 [07:34:41] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.39, 3.31, 3.03 [07:45:37] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['NS.ANKH.FR.eu.org.', 'NS1.eu.org.', 'NS1.ERIOMEM.NET.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [08:30:44] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.74, 6.87, 6.33 [08:34:38] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.56, 5.87, 6.06 [08:43:14] [02mw-config] 07RhinosF1 opened pull request 03#5356: Update GlobalSettings.php - 13https://github.com/miraheze/mw-config/pull/5356 [08:44:23] miraheze/mw-config - RhinosF1 the build passed. [09:18:20] [02mw-config] 07RhinosF1 opened pull request 03#5357: VE: use parsoid in direct mode - 13https://github.com/miraheze/mw-config/pull/5357 [09:18:22] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.01, 7.52, 6.83 [09:19:16] [02mw-config] 07RhinosF1 edited pull request 03#5356: Parsoid: enable query string - 13https://github.com/miraheze/mw-config/pull/5356 [09:19:26] miraheze/mw-config - RhinosF1 the build passed. [09:22:19] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.54, 7.50, 6.99 [09:26:17] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.11, 6.60, 6.73 [09:52:45] PROBLEM - db142 Backups SQL on db142 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209715 seconds old and 86822 bytes [10:03:12] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.69, 3.12, 2.60 [10:05:10] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.36, 2.75, 2.52 [10:11:39] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.29, 2.95, 1.58 [10:15:40] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 2.64, 3.16, 2.02 [10:29:23] PROBLEM - itadminwiki.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'itadminwiki.pl' expires in 15 day(s) (Mon 20 Nov 2023 10:09:35 GMT +0000). [10:29:37] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bbdc93c99ab2...99ef8578252b [10:29:40] [02miraheze/ssl] 07MirahezeSSLBot 0399ef857 - Bot: Update SSL cert for itadminwiki.pl [10:58:17] RECOVERY - itadminwiki.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'itadminwiki.pl' will expire on Fri 02 Feb 2024 09:29:29 GMT +0000. [12:13:30] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5010347962 secs [13:17:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.84, 6.81, 6.54 [13:19:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.06, 8.12, 7.03 [13:21:38] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.10, 7.72, 7.02 [13:29:32] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.10, 6.24, 6.61 [13:33:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.85, 7.10, 6.92 [13:37:26] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.72, 6.58, 6.73 [13:49:04] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.13, 9.58, 7.82 [13:51:01] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 6.83, 8.63, 7.69 [13:51:14] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.16, 7.00, 6.75 [13:53:12] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.08, 6.40, 6.57 [14:27:08] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.96, 9.22, 8.10 [14:29:06] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 9.98, 9.26, 8.23 [14:35:10] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.58, 3.25, 3.00 [14:35:40] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 11.21, 8.00, 7.01 [14:37:10] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.19, 3.29, 3.05 [14:37:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.39, 7.66, 7.01 [14:45:34] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.67, 6.40, 6.65 [14:51:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.01, 7.08, 6.86 [14:53:27] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.41, 6.76, 6.77 [14:54:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/379d4e6cd170...979e47aa2be2 [14:54:23] [02miraheze/puppet] 07paladox 03979e47a - nutcracker: increase memcache timeout to 500 [15:03:17] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.32, 7.12, 6.96 [15:07:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.85, 6.36, 6.67 [15:15:42] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.48, 3.15, 2.98 [15:19:39] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.34, 3.09, 3.02 [15:32:32] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 9.69, 10.23, 9.30 [15:33:28] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.16, 3.85, 3.28 [15:35:26] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.95, 3.81, 3.33 [15:36:26] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 7.48, 9.48, 9.26 [15:39:53] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/9051a3b9dd0e [15:39:55] [02miraheze/mw-config] 07paladox 039051a3b - Set wgTranslateTranslationDefaultService to false [15:39:58] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [15:39:59] [02mw-config] 07paladox opened pull request 03#5358: Set wgTranslateTranslationDefaultService to false - 13https://github.com/miraheze/mw-config/pull/5358 [15:40:19] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.00, 10.24, 9.64 [15:40:26] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9051a3b9dd0e...5843f971cfe4 [15:40:27] [02miraheze/mw-config] 07github-actions 035843f97 - CI: lint code to MediaWiki standards [15:40:30] [02mw-config] 07github-actions[bot] synchronize pull request 03#5358: Set wgTranslateTranslationDefaultService to false - 13https://github.com/miraheze/mw-config/pull/5358 [15:41:09] [02mw-config] 07paladox closed pull request 03#5358: Set wgTranslateTranslationDefaultService to false - 13https://github.com/miraheze/mw-config/pull/5358 [15:41:12] miraheze/mw-config - paladox the build passed. [15:41:12] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/97dd0e73bcd7...10ca4ff4f198 [15:41:13] [02miraheze/mw-config] 07paladox 0310ca4ff - Set wgTranslateTranslationDefaultService to false (#5358) [15:41:15] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [15:41:16] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [15:41:25] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.19, 10.39, 9.27 [15:41:57] [02mw-config] 07paladox closed pull request 03#5245: GlobalCache: implement Parsoid Parser Caching - 13https://github.com/miraheze/mw-config/pull/5245 [15:41:58] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/10ca4ff4f198...77e4893a1147 [15:41:59] [02miraheze/mw-config] 07RhinosF1 0377e4893 - GlobalCache: implement Parsoid Parser Caching (#5245) [15:42:09] [02mw-config] 07paladox closed pull request 03#5356: Parsoid: enable query string - 13https://github.com/miraheze/mw-config/pull/5356 [15:42:12] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/77e4893a1147...50f518bcbc6f [15:42:12] miraheze/mw-config - paladox the build passed. [15:42:13] [02miraheze/mw-config] 07RhinosF1 0350f518b - Parsoid: enable query string (#5356) [15:42:16] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.97, 9.84, 9.58 [15:42:21] [02mw-config] 07paladox closed pull request 03#5357: VE: use parsoid in direct mode - 13https://github.com/miraheze/mw-config/pull/5357 [15:42:22] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/50f518bcbc6f...fba60a621253 [15:42:24] [02miraheze/mw-config] 07RhinosF1 03fba60a6 - VE: use parsoid in direct mode (#5357) [15:42:40] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [15:42:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:42:52] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 12s [15:42:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:42:58] !log [paladox@mwtask141] starting deploy of {'config': True} to all [15:43:02] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.58, 3.21, 2.51 [15:43:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:43:06] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [15:43:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:43:19] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [15:43:21] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [15:43:23] miraheze/mw-config - paladox the build passed. [15:43:24] !log [paladox@test131] starting deploy of {'config': True} to all [15:43:25] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [15:43:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:43:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:43:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:43:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:45:00] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 3.28, 3.17, 2.58 [15:45:20] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.80, 11.02, 9.76 [15:46:08] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.55, 10.94, 10.11 [15:50:03] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 8.21, 9.86, 9.88 [15:51:14] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.71, 3.85, 3.55 [15:53:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/8dfc48b9506c [15:53:06] [02miraheze/puppet] 07paladox 038dfc48b - mediawiki: only run parsoidCachePrewarm on the jobrunner-hi [15:53:08] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [15:53:08] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 8.99, 9.88, 9.68 [15:53:11] [02puppet] 07paladox opened pull request 03#3480: mediawiki: only run parsoidCachePrewarm on the jobrunner-hi - 13https://github.com/miraheze/puppet/pull/3480 [15:54:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8dfc48b9506c...a62dc08a305d [15:54:40] [02miraheze/puppet] 07paladox 03a62dc08 - Update jobrunner-hi.json.erb [15:54:41] [02puppet] 07paladox synchronize pull request 03#3480: mediawiki: only run parsoidCachePrewarm on the jobrunner-hi - 13https://github.com/miraheze/puppet/pull/3480 [15:54:47] [02puppet] 07paladox closed pull request 03#3480: mediawiki: only run parsoidCachePrewarm on the jobrunner-hi - 13https://github.com/miraheze/puppet/pull/3480 [15:54:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/979e47aa2be2...cc4ce19db4ce [15:54:53] [02miraheze/puppet] 07paladox 03cc4ce19 - mediawiki: only run parsoidCachePrewarm on the jobrunner-hi (#3480) [15:54:55] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [15:54:58] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [15:55:11] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.78, 3.90, 3.64 [16:00:21] !log [paladox@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/w/maintenance/runJobs.php [16:00:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:02:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cc4ce19db4ce...3cfbcf2c145e [16:02:38] [02miraheze/puppet] 07paladox 033cfbcf2 - jobrunner-hi: give parsoidCachePrewarm it's own job group [16:02:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.95, 6.69, 6.32 [16:03:11] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.14, 3.51, 3.51 [16:04:59] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.72, 6.26, 6.20 [16:05:13] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.40, 3.40, 3.46 [16:09:10] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.37, 3.04, 3.31 [16:10:22] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/0f5bb967f0fe [16:10:23] [02miraheze/mw-config] 07paladox 030f5bb96 - Switch warmCache off for wgParsoidCacheConfig [16:10:25] [02mw-config] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [16:10:26] [02mw-config] 07paladox opened pull request 03#5359: Switch warmCache off for wgParsoidCacheConfig - 13https://github.com/miraheze/mw-config/pull/5359 [16:10:35] [02mw-config] 07paladox closed pull request 03#5359: Switch warmCache off for wgParsoidCacheConfig - 13https://github.com/miraheze/mw-config/pull/5359 [16:10:38] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/fba60a621253...a0ec5a79450c [16:10:40] [02miraheze/mw-config] 07paladox 03a0ec5a7 - Switch warmCache off for wgParsoidCacheConfig (#5359) [16:10:42] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-4 [16:10:42] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [16:10:44] [02mw-config] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/mw-config [16:10:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:10:53] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [16:10:59] !log [paladox@mwtask141] starting deploy of {'config': True} to all [16:11:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:11:07] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [16:11:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:11:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:11:22] miraheze/mw-config - paladox the build passed. [16:11:37] miraheze/mw-config - paladox the build passed. [16:18:47] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3cfbcf2c145e...0bb9ecf41dbe [16:18:49] [02miraheze/puppet] 07paladox 030bb9ecf - Increase parseroidcache runners to 2 [16:23:16] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a0ec5a79450c...c48e5a44e958 [16:23:18] [02miraheze/mw-config] 07paladox 03c48e5a4 - Revert "Switch warmCache off for wgParsoidCacheConfig (#5359)" [16:24:13] miraheze/mw-config - paladox the build passed. [16:26:14] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=poor4wikiwiki --no-updates --username-prefix=wikia:grim-tides /home/paladox/poor_4fandomcom-20231103-history.xml (START) [16:26:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:30:06] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=poor4wikiwiki /home/paladox/dump --search-recursively (START) [16:30:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:30:46] !log [paladox@mwtask141] screen -d -m bash -c mwscript importImages.php poor4wikiwiki -y /home/paladox/dump --search-recursively; rm -rf /home/paladox/dump [16:30:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:31:12] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [16:31:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:31:22] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [16:31:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:31:28] !log [paladox@mwtask141] starting deploy of {'config': True} to all [16:31:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:31:38] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 10s [16:31:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:35:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [16:36:02] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True} to all [16:36:03] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 1s [16:36:05] !log [paladox@test131] starting deploy of {'config': True} to all [16:36:06] !log [paladox@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [16:36:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:36:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:36:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:36:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:35] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=poor4wikiwiki --no-updates --username-prefix=wikia:grim-tides /home/paladox/poor_4fandomcom-20231103-history.xml (END - exit=0) [16:37:36] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=nitromepediawiki (START) [16:37:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:50] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 11.54, 9.35, 7.77 [16:38:14] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 13.36, 11.38, 9.90 [16:38:29] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 15.20, 11.71, 9.63 [16:39:50] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 9.10, 8.90, 7.78 [16:39:50] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 6.94, 6.76, 5.89 [16:41:50] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 8.28, 7.30, 6.19 [16:42:24] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.18, 11.75, 10.14 [16:43:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.35, 7.27, 6.62 [16:43:44] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.82, 3.75, 3.35 [16:43:50] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 6.58, 7.07, 6.25 [16:44:09] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.99, 6.43, 5.79 [16:44:21] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 12.16, 11.60, 10.25 [16:44:51] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.60, 3.45, 3.14 [16:45:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.11, 7.12, 6.64 [16:46:03] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 8.76, 11.75, 10.97 [16:46:10] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.79, 6.21, 5.79 [16:46:19] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.36, 11.44, 10.35 [16:46:45] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.72, 3.14, 3.07 [16:47:27] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.90, 7.54, 6.84 [16:47:42] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.61, 3.74, 3.40 [16:47:50] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 6.14, 6.73, 6.31 [16:48:00] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 11.27, 12.17, 11.24 [16:49:26] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.93, 7.84, 7.05 [16:49:40] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.99, 3.52, 3.37 [16:49:57] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 7.44, 10.58, 10.78 [16:50:13] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 12.96, 11.91, 10.79 [16:51:25] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.31, 8.03, 7.22 [16:51:32] !log [paladox@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=poor4wikiwiki /home/paladox/dump --search-recursively (END - exit=0) [16:51:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:51:38] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.50, 3.86, 3.51 [16:51:50] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 8.36, 7.38, 6.65 [16:52:11] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 9.41, 11.03, 10.60 [16:53:23] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.41, 7.71, 7.19 [16:53:37] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.12, 3.59, 3.45 [16:53:50] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 7.43, 7.34, 6.73 [16:53:52] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.17, 10.60, 10.69 [16:55:35] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.33, 3.09, 3.28 [16:55:49] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 10.76, 10.57, 10.67 [16:55:50] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 9.04, 7.87, 6.99 [16:56:05] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 14.18, 12.06, 11.10 [16:58:03] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.21, 11.76, 11.12 [16:59:20] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 4.09, 5.80, 6.53 [16:59:30] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.43, 3.41, 3.37 [17:01:30] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.26, 3.64, 3.45 [17:01:40] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 12.89, 10.95, 10.74 [17:01:58] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 13.89, 12.18, 11.35 [17:03:23] PROBLEM - mw133 Current Load on mw133 is WARNING: WARNING - load average: 11.65, 10.09, 9.20 [17:05:19] RECOVERY - mw133 Current Load on mw133 is OK: OK - load average: 8.93, 9.79, 9.21 [17:05:26] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.55, 3.30, 3.38 [17:07:32] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 9.06, 11.18, 11.11 [17:09:47] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 8.98, 10.79, 11.22 [17:13:23] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.14, 8.72, 10.06 [17:15:40] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 16.63, 12.72, 11.79 [17:17:16] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.78, 3.20, 1.76 [17:17:39] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 10.15, 11.76, 11.56 [17:19:15] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.93, 3.44, 2.03 [17:21:15] PROBLEM - os141 Current Load on os141 is CRITICAL: LOAD CRITICAL - total load average: 4.75, 3.79, 2.32 [17:22:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0bb9ecf41dbe...43b9a56bd06d [17:22:25] [02miraheze/puppet] 07paladox 0343b9a56 - jobrunner-hi: increase parseroidcache runners to 4 [17:24:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/22b78fd3f9c0 [17:24:42] [02miraheze/puppet] 07paladox 0322b78fd - jobrunner: run runJobs with run.php [17:24:45] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [17:24:47] [02puppet] 07paladox opened pull request 03#3481: jobrunner: run runJobs with run.php - 13https://github.com/miraheze/puppet/pull/3481 [17:25:09] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/22b78fd3f9c0...667c09bd1bf2 [17:25:11] [02miraheze/puppet] 07paladox 03667c09b - Update jobrunner.json.erb [17:25:14] [02puppet] 07paladox synchronize pull request 03#3481: jobrunner: run runJobs with run.php - 13https://github.com/miraheze/puppet/pull/3481 [17:25:17] [02puppet] 07paladox closed pull request 03#3481: jobrunner: run runJobs with run.php - 13https://github.com/miraheze/puppet/pull/3481 [17:25:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/43b9a56bd06d...9c8db35accd9 [17:25:19] [02miraheze/puppet] 07paladox 039c8db35 - jobrunner: run runJobs with run.php (#3481) [17:25:20] PROBLEM - os141 Current Load on os141 is WARNING: LOAD WARNING - total load average: 3.51, 3.71, 2.64 [17:25:21] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [17:25:22] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://github.com/miraheze/puppet [17:26:58] PROBLEM - mw131 Current Load on mw131 is CRITICAL: CRITICAL - load average: 14.06, 10.74, 9.98 [17:27:20] RECOVERY - os141 Current Load on os141 is OK: LOAD OK - total load average: 0.83, 2.66, 2.39 [17:27:39] PROBLEM - mw132 Current Load on mw132 is CRITICAL: CRITICAL - load average: 13.91, 10.85, 10.74 [17:28:55] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 9.91, 10.29, 9.90 [17:29:39] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.49, 11.11, 10.86 [17:32:50] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 7.44, 9.31, 9.63 [17:35:39] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 7.30, 9.00, 10.06 [17:45:00] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9c8db35accd9...a146170ee0e2 [17:45:02] [02miraheze/puppet] 07paladox 03a146170 - jobrunner-hi: increase parsoidCachePrewarm memory limits to 1.4g [17:47:41] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.90, 7.17, 6.78 [17:49:30] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.63, 3.26, 2.92 [17:51:30] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.11, 2.77, 2.78 [17:53:37] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.36, 6.50, 6.63 [17:57:38] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.60, 3.25, 3.11 [18:01:35] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.04, 3.86, 3.37 [18:03:29] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.97, 7.01, 6.62 [18:05:28] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.60, 6.81, 6.59 [18:05:32] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.89, 3.49, 3.35 [18:07:27] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.01, 6.46, 6.49 [18:07:30] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.33, 3.08, 3.22 [18:22:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.15, 6.80, 6.50 [18:24:13] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.28, 6.58, 6.46 [18:32:15] PROBLEM - mw131 Current Load on mw131 is WARNING: WARNING - load average: 11.19, 9.34, 8.66 [18:33:26] PROBLEM - mw132 Current Load on mw132 is WARNING: WARNING - load average: 11.01, 9.78, 9.10 [18:34:12] RECOVERY - mw131 Current Load on mw131 is OK: OK - load average: 8.09, 9.18, 8.71 [18:35:04] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.13, 6.83, 6.62 [18:35:23] RECOVERY - mw132 Current Load on mw132 is OK: OK - load average: 9.51, 9.75, 9.18 [18:37:03] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.57, 6.40, 6.49 [18:54:31] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c48e5a44e958...129afc362c21 [18:54:32] [02miraheze/mw-config] 07paladox 03129afc3 - Disable warmup for bluepageswiki and allthetropeswiki [18:54:51] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [18:54:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:55:04] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 12s [18:55:10] !log [paladox@mwtask141] starting deploy of {'config': True} to all [18:55:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:55:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:55:19] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 8s [18:55:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:55:33] miraheze/mw-config - paladox the build passed. [18:56:18] !log [@test131] starting deploy of {'config': True} to all [18:56:19]