[00:00:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.85, 11.38, 10.64 [00:01:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.41, 10.88, 10.08 [00:02:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.56, 10.63, 10.43 [00:02:59] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.08, 4.47, 3.61 [00:04:02] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 2.15, 3.49, 2.09 [00:04:04] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.52, 10.84, 6.71 [00:04:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.65, 9.85, 10.17 [00:06:03] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.89, 2.56, 1.91 [00:06:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.57, 10.09, 10.00 [00:07:53] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.67, 3.88, 2.88 [00:09:18] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.58, 1.68, 0.95 [00:09:49] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.69, 3.39, 2.82 [00:11:19] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 1.15, 1.43, 0.95 [00:14:03] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 3.39, 4.77, 3.29 [00:16:02] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 1.20, 3.57, 3.03 [00:18:03] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.94, 2.70, 2.77 [00:20:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.34, 10.77, 10.23 [00:20:47] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.57, 10.40, 9.89 [00:24:44] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.26, 10.19, 9.96 [00:25:03] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.45, 3.49, 3.97 [00:26:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.81, 9.56, 10.03 [00:28:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.37, 2.90, 3.87 [00:30:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.16, 10.26, 10.02 [00:31:02] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.67, 3.51, 3.73 [00:32:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.67, 2.10, 3.30 [00:32:37] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.02, 11.45, 10.48 [00:32:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.58, 10.39, 10.18 [00:33:02] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.38, 2.93, 3.48 [00:33:12] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 5.41, 4.03, 3.41 [00:34:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.35, 11.06, 10.46 [00:35:03] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.71, 4.14, 3.82 [00:36:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.99, 11.07, 10.54 [00:39:08] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 2.57, 3.58, 3.47 [00:40:32] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.20, 12.00, 11.36 [00:41:08] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.96, 3.11, 3.31 [00:42:31] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.74, 12.27, 11.53 [00:44:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 8.20, 10.97, 11.16 [00:46:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.05, 9.61, 10.16 [00:54:29] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.48, 9.41, 10.14 [01:03:19] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.12, 10.67, 9.95 [01:05:17] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.68, 10.50, 9.99 [01:05:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.01, 11.47, 10.77 [01:07:14] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.80, 9.65, 9.74 [01:11:08] !log [universalomega@mwtask141] 3644 non-existing wikis removed from Matomo with the cleanupMatomos.php script that has now finished [01:11:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:11:19] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.06, 9.09, 9.95 [01:20:48] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.51, 11.61, 10.29 [01:22:08] !log [universalomega@mwtask141] Run https://phabricator.miraheze.org/T10412#209891 to delete all Swift containers that do not correspond to any wiki (191 containers deleted) [01:22:09] [url] ⚓ T10412 Cleanup database mess after cloud14 outages | phabricator.miraheze.org [01:22:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:22:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.20, 11.21, 10.30 [01:23:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.38, 3.97, 3.22 [01:23:21] PROBLEM - swiftproxy111 HTTPS on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 401 Unauthorized [01:23:32] PROBLEM - swiftproxy111 HTTP on swiftproxy111 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 401 Unauthorized [01:24:03] PROBLEM - swiftproxy131 HTTPS on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 401 Unauthorized [01:24:06] PROBLEM - swiftproxy131 HTTP on swiftproxy131 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host: HTTP/1.1 401 Unauthorized [01:24:43] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.02, 10.04, 10.00 [01:25:00] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.19, 3.28, 3.06 [01:28:58] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.28, 3.75, 3.34 [01:29:54] [02ManageWiki] 07Justman10000 opened pull request 03#379: Update - 13https://github.com/miraheze/ManageWiki/pull/379 [01:30:55] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.74, 3.35, 3.24 [01:37:21] RECOVERY - swiftproxy111 HTTPS on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.055 second response time [01:37:32] RECOVERY - swiftproxy111 HTTP on swiftproxy111 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.018 second response time [01:38:03] RECOVERY - swiftproxy131 HTTPS on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.027 second response time [01:38:06] RECOVERY - swiftproxy131 HTTP on swiftproxy131 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.020 second response time [01:47:13] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.83, 2.66, 3.69 [01:49:11] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.60, 11.12, 10.09 [01:50:20] [02ManageWiki] 07Universal-Omega closed pull request 03#379: Update - 13https://github.com/miraheze/ManageWiki/pull/379 [01:50:46] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.35, 11.01, 9.95 [01:51:13] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.10, 2.27, 3.28 [01:51:32] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.06, 3.86, 3.37 [01:52:45] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.17, 11.21, 10.17 [01:53:05] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.59, 11.50, 10.50 [01:53:29] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.33, 3.54, 3.30 [01:55:27] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.08, 3.12, 3.18 [01:57:00] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.76, 10.11, 10.13 [02:00:40] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.58, 9.53, 9.84 [02:01:13] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.38, 3.52, 3.41 [02:03:13] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.87, 4.02, 3.61 [02:04:35] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.33, 10.90, 10.33 [02:05:14] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.49, 3.64, 3.51 [02:06:34] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.25, 11.39, 10.56 [02:08:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.82, 10.73, 10.41 [02:10:31] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.46, 9.73, 10.09 [02:12:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.04, 10.40, 9.93 [02:15:14] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.02, 2.85, 3.31 [02:18:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.07, 10.82, 10.38 [02:18:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.74, 9.80, 9.86 [02:20:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.58, 10.39, 10.28 [02:22:29] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.22, 9.27, 9.89 [02:28:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.78, 10.91, 10.34 [02:30:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.82, 11.14, 10.29 [02:32:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.20, 11.27, 10.46 [02:33:14] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.69, 3.15, 2.90 [02:34:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.11, 11.21, 10.59 [02:34:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.71, 10.08, 10.12 [02:35:14] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.58, 4.73, 3.48 [02:36:29] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.99, 9.46, 10.04 [02:38:48] PROBLEM - swiftobject121 Current Load on swiftobject121 is WARNING: WARNING - load average: 7.43, 5.95, 4.49 [02:40:48] RECOVERY - swiftobject121 Current Load on swiftobject121 is OK: OK - load average: 4.21, 5.49, 4.52 [02:41:14] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.17, 3.42, 3.38 [02:43:14] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.64, 2.91, 3.20 [02:46:32] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.99, 3.90, 3.25 [02:48:29] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.18, 4.08, 3.39 [02:52:14] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.52, 3.60, 3.48 [02:54:14] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.28, 4.04, 3.64 [02:54:20] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.83, 3.71, 3.44 [02:56:15] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.80, 3.57, 3.52 [02:56:17] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.18, 3.91, 3.54 [02:58:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.41, 3.40, 3.40 [02:58:15] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.66, 4.18, 3.76 [03:00:05] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=pokemonwiki (END - exit=0) [03:00:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:31] RECOVERY - db121 Backups SQL on db121 is OK: FILE_AGE OK: /var/log/sql-backup.log is 30 seconds old and 0 bytes [03:01:08] PROBLEM - db112 Backups SQL on db112 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209666 seconds old and 90 bytes [03:02:53] !log [universalomega@mwtask141] Renamed pokemon2wiki to pokemonwiki [03:02:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:04:09] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.91, 5.65, 4.27 [03:04:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9bbb0139da42...31970c9d2c06 [03:04:19] [02miraheze/mw-config] 07Universal-Omega 0331970c9 - pokemon2wiki -> pokemonwiki [03:05:25] miraheze/mw-config - Universal-Omega the build passed. [03:10:23] !log [@mwtask141] starting deploy of {'config': True} to all [03:10:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:10:30] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [03:10:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:13:31] !log [universalomega@mwtask141] Renamed edcarewiki to edsharewiki [03:13:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:14:02] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=edsharewiki (END - exit=0) [03:14:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.13, 3.53, 3.86 [03:14:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:16:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.70, 3.62, 3.82 [03:18:15] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.21, 3.21, 3.85 [03:18:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.43, 10.18, 9.25 [03:20:15] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.66, 3.65, 3.92 [03:22:15] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.64, 2.89, 3.61 [03:22:19] !log [universalomega@mwtask141] Renamed parodylogoswiki to jlwwiki [03:22:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:22:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.52, 9.68, 9.29 [03:27:03] !log [@test131] starting deploy of {'config': True} to all [03:27:04] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [03:27:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:27:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:28:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.44, 3.24, 3.87 [03:29:17] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=jlwwiki (END - exit=0) [03:29:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:30:00] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=bungiehalowikiwiki (END - exit=0) [03:30:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.30, 3.72, 3.97 [03:30:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:30:15] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.77, 2.57, 3.24 [03:30:32] !log [universalomega@mwtask141] Renamed bungiehalopediawiki to bungiehalowikiwiki [03:30:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.47, 10.31, 9.72 [03:30:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:32:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.29, 3.06, 3.69 [03:32:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.56, 9.91, 9.64 [03:34:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.27, 3.37, 3.71 [03:36:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.01, 2.80, 3.46 [03:38:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.22, 2.73, 3.33 [03:39:18] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=brandonwmwiki (END - exit=0) [03:39:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:40:00] !log [universalomega@mwtask141] Renamed bwmwiki to brandonwmwiki [03:40:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:46:10] !log [universalomega@mwtask141] Renamed kkvocaloidwiki to wikianderwiki [03:46:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:49:51] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=anderwiki (END - exit=65280) [03:49:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:50:26] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=wikianderwiki (END - exit=0) [03:50:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:51:57] !log [universalomega@mwtask141] sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=wideohousewikiwiki (END - exit=0) [03:52:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:52:45] !log [universalomega@mwtask141] Renamed aevumgamewiki to wideohousewikiwiki [03:52:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:06:17] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.12, 3.83, 3.14 [04:10:17] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.91, 3.38, 3.13 [04:18:35] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.96, 10.84, 9.14 [04:18:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.69, 11.24, 9.61 [04:20:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.62, 10.94, 9.39 [04:20:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.57, 11.01, 9.71 [04:21:19] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.75, 6.82, 5.76 [04:23:16] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.91, 7.44, 6.10 [04:24:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.04, 9.86, 9.60 [04:26:30] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.80, 9.93, 9.45 [04:26:55] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.18, 3.57, 2.89 [04:27:08] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.55, 7.82, 6.59 [04:28:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.88, 4.01, 2.90 [04:28:51] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 1.76, 2.90, 2.73 [04:30:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.87, 3.62, 2.90 [04:30:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.09, 10.26, 9.92 [04:32:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.58, 3.13, 2.80 [04:32:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.20, 8.91, 9.47 [04:36:17] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.74, 3.53, 3.25 [04:42:17] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.03, 3.10, 3.20 [04:43:08] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 9.07, 7.89, 7.18 [04:44:04] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.30, 5.07, 3.50 [04:45:08] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.12, 7.55, 7.15 [04:49:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.63, 10.92, 10.17 [04:50:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.85, 3.75, 3.39 [04:51:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.50, 10.98, 10.27 [04:52:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.01, 4.28, 3.65 [04:53:22] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 14.18, 12.04, 10.73 [04:54:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.15, 3.46, 3.42 [04:55:19] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.08, 11.29, 10.62 [04:56:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.18, 4.24, 3.71 [04:57:08] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 9.73, 7.70, 7.23 [04:58:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.54, 3.13, 3.36 [04:59:08] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.93, 7.70, 7.28 [04:59:08] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.73, 3.77, 3.15 [05:01:08] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 10.78, 9.06, 7.84 [05:01:11] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.71, 9.61, 10.15 [05:03:08] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 2.81, 3.53, 3.21 [05:04:45] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 63.48, 47.15, 21.44 [05:06:18] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.75, 3.85, 3.17 [05:07:08] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.65, 3.20, 3.16 [05:08:18] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.30, 3.36, 3.08 [05:09:08] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.81, 7.94, 7.76 [05:11:08] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.37, 8.05, 7.82 [05:15:08] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.36, 7.76, 7.78 [05:18:18] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.96, 3.71, 3.24 [05:20:18] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.20, 3.23, 3.13 [05:20:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.36, 11.28, 10.28 [05:21:08] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.37, 7.43, 7.51 [05:23:08] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.50, 6.74, 7.25 [05:24:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.48, 11.38, 10.56 [05:28:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.83, 4.10, 3.49 [05:28:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.07, 9.95, 10.20 [05:29:08] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.29, 5.84, 6.70 [05:30:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.21, 3.82, 3.47 [05:32:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.75, 3.02, 3.21 [05:35:08] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.07, 6.73, 6.82 [05:38:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.64, 4.34, 3.68 [05:40:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.81, 1.90, 3.99 [05:40:19] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.93, 3.93, 3.62 [05:41:28] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.04, 11.26, 10.44 [05:43:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.36, 10.88, 10.43 [05:44:19] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.47, 2.82, 3.26 [05:45:08] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.40, 7.42, 7.05 [05:45:22] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.19, 10.02, 10.15 [05:46:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.36, 2.35, 3.43 [05:47:08] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 5.97, 6.96, 6.93 [05:48:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.31, 1.89, 3.13 [05:49:08] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 6.35, 6.62, 6.80 [05:53:08] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.17, 7.55, 7.14 [05:55:08] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.46, 7.23, 7.06 [05:57:08] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.84, 8.01, 7.38 [06:01:08] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.96, 7.56, 7.35 [06:09:17] !log [reception@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/reception/stopforumspam.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --disable stopforumspam (START) [06:09:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:11:41] !log [reception@mwtask141] sudo -u www-data /usr/local/bin/foreachwikiindblist /home/reception/stopforumspam.json /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --disable stopforumspam (END - exit=0) [06:11:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:16:20] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1102450192 secs [06:17:08] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.18, 7.69, 7.40 [06:17:49] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.45, 3.93, 3.05 [06:19:46] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.95, 3.38, 2.97 [06:21:08] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.72, 7.44, 7.38 [06:22:20] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.07993364334 secs [06:23:08] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 9.06, 7.95, 7.56 [06:25:09] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 4.00, 3.54, 2.96 [06:28:54] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 2.19, 4.69, 2.74 [06:30:48] [02mw-config] 07Reception123 commented on pull request 03#5107: remove StopForumSpam from ManageWiki (enabled globally) - 13https://github.com/miraheze/mw-config/pull/5107#issuecomment-1416936379 [06:30:53] [02mw-config] 07Reception123 closed pull request 03#5107: remove StopForumSpam from ManageWiki (enabled globally) - 13https://github.com/miraheze/mw-config/pull/5107 [06:30:54] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.49, 3.20, 2.43 [06:30:55] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/31970c9d2c06...5061e8781ddf [06:30:57] [02miraheze/mw-config] 07Reception123 035061e87 - remove StopForumSpam from ManageWiki (enabled globally) (#5107) [06:30:58] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-4 - 13https://github.com/miraheze/mw-config [06:31:01] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-4 [06:31:53] miraheze/mw-config - Reception123 the build passed. [06:33:08] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.74, 7.72, 7.81 [06:33:08] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.27, 3.13, 3.06 [06:34:19] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.02, 4.06, 3.41 [06:40:05] !log [@mwtask141] starting deploy of {'config': True} to all [06:40:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:40:13] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [06:40:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:40:20] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.70, 3.35, 3.43 [06:46:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.34, 2.63, 3.15 [06:55:08] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.48, 6.15, 6.80 [06:56:57] !log [@test131] starting deploy of {'config': True} to all [06:56:58] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [06:57:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:57:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:59:08] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.56, 6.76, 6.93 [07:07:08] PROBLEM - db121 Current Load on db121 is CRITICAL: CRITICAL - load average: 8.51, 7.64, 7.23 [07:07:20] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.16, 3.42, 2.92 [07:07:32] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1185137033 secs [07:09:08] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.11, 7.18, 7.12 [07:10:09] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.11, 10.93, 9.63 [07:11:20] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.57, 3.30, 3.08 [07:14:03] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.62, 9.71, 9.48 [07:17:08] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 4.91, 5.94, 6.57 [07:25:32] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.09928512573 secs [07:33:47] [02mw-config] 07Universal-Omega commented on pull request 03#5089: T10323: Add more links to Special:AdminLinks for paneidoversewiki - 13https://github.com/miraheze/mw-config/pull/5089#issuecomment-1417021019 [07:36:22] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.44, 2.88, 2.47 [07:38:22] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.17, 2.60, 2.42 [07:43:17] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=radiversewiki --no-updates --username-prefix=w /home/reception/ImportDump/radiversewiki-20230205031754.xml (START) [07:43:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:44:15] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=radiversewiki --no-updates --username-prefix=w /home/reception/ImportDump/radiversewiki-20230205031754.xml (END - exit=0) [07:44:16] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=radiversewiki (START) [07:44:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:44:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:46:22] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=radiversewiki (END - exit=0) [07:46:23] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=radiversewiki --active --update (END - exit=0) [07:46:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:46:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:47:12] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.42, 9.72, 8.91 [07:49:09] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.99, 9.24, 8.84 [08:02:22] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.74, 3.67, 2.69 [08:06:22] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.20, 3.66, 2.95 [08:08:22] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.43, 3.13, 2.84 [08:12:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.66, 9.96, 9.24 [08:14:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.49, 10.83, 9.62 [08:16:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.57, 10.82, 9.75 [08:21:23] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.84, 10.45, 9.45 [08:21:23] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.05, 6.29, 5.54 [08:22:39] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.80, 10.03, 9.81 [08:23:19] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 6.39, 6.20, 5.59 [08:23:22] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.49, 9.74, 9.31 [08:48:03] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.74, 3.46, 2.63 [08:49:17] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:49:59] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.83, 3.21, 2.64 [08:51:18] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 6.705 second response time [08:52:24] PROBLEM - mw122 MediaWiki Rendering on mw122 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:53:54] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.22, 3.46, 2.89 [08:54:21] RECOVERY - mw122 MediaWiki Rendering on mw122 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 1.538 second response time [08:57:47] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.05, 3.54, 3.04 [08:59:08] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 7.19, 6.77, 5.99 [08:59:43] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.05, 3.39, 3.05 [09:03:08] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.14, 6.49, 6.11 [09:03:38] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.36, 3.46, 3.17 [09:05:34] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.75, 3.22, 3.12 [09:07:08] PROBLEM - db121 Current Load on db121 is WARNING: WARNING - load average: 6.29, 7.22, 6.57 [09:09:29] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.86, 3.66, 3.32 [09:11:08] RECOVERY - db121 Current Load on db121 is OK: OK - load average: 5.41, 6.43, 6.42 [09:11:25] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.90, 3.37, 3.25 [09:20:39] PROBLEM - jobchron121 Puppet on jobchron121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_JobRunner] [09:31:17] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=dgck81lnnwiki --no-updates --username-prefix=dgck81lnnwiki /home/reception/ImportDump/dgck81lnnwiki-20230205085846.xml (START) [09:31:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:33:26] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=dgck81lnnwiki --no-updates --username-prefix=dgck81lnnwiki /home/reception/ImportDump/dgck81lnnwiki-20230205085846.xml (END - exit=0) [09:33:27] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=dgck81lnnwiki (START) [09:33:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:33:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:34:52] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=dgck81lnnwiki (END - exit=0) [09:34:53] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=dgck81lnnwiki --active --update (END - exit=0) [09:34:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:35:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:35:24] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.02, 4.64, 3.08 [09:39:24] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.92, 3.79, 3.16 [09:41:24] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.16, 3.24, 3.03 [09:46:34] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [09:47:28] PROBLEM - db101 Backups SQL on db101 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864214 seconds old and 48 bytes [10:37:25] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.16, 3.67, 2.56 [10:49:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 0.99, 3.58, 3.77 [10:51:25] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.53, 2.55, 3.37 [10:55:00] !log sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=hkrailwiki --summary="Imported from https://hkrail.miraheze.org" --user="Miraheze maintenance script" --search-recursively /home/reception/hkrailwiki_images [10:55:01] [url] 香港鐵路百科 | hkrail.miraheze.org [10:55:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:02:06] [02ManageWiki] 07Justman10000 commented on pull request 03#379: Update - 13https://github.com/miraheze/ManageWiki/pull/379#issuecomment-1417460853 [11:14:04] [02mw-config] 07ugochimobi opened pull request 03#5109: Fix for T10430 wgKartographerDfltStyle - 13https://github.com/miraheze/mw-config/pull/5109 [11:15:00] miraheze/mw-config - ugochimobi the build passed. [11:28:54] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.95, 3.09, 2.00 [11:29:34] [02ManageWiki] 07Reception123 commented on pull request 03#379: Update - 13https://github.com/miraheze/ManageWiki/pull/379#issuecomment-1417535664 [11:30:48] [02ManageWiki] 07Reception123 edited a comment on pull request 03#379: Update - 13https://github.com/miraheze/ManageWiki/pull/379#issuecomment-1417535664 [11:30:54] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.21, 2.72, 2.00 [11:35:27] PROBLEM - wiki.albrusmusic.ru - reverse DNS on sslhost is WARNING: Timeout: The DNS operation timed out after 5.406742095947266 seconds [11:36:26] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.74, 3.52, 2.37 [11:39:52] !log UPDATE content SET content_model = 7 WHERE content_model = 6; (hkrailwiki) [11:39:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:42:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.30, 3.95, 3.08 [11:44:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.08, 3.34, 2.96 [12:04:14] RECOVERY - wiki.albrusmusic.ru - reverse DNS on sslhost is OK: SSL OK - wiki.albrusmusic.ru reverse DNS resolves to cp23.miraheze.org - CNAME OK [12:25:41] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 17.26, 13.15, 10.38 [12:25:58] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.68, 12.14, 10.18 [12:31:28] [02mw-config] 07Reception123 closed pull request 03#4931: Add rights to ManageWiki blacklist - 13https://github.com/miraheze/mw-config/pull/4931 [12:31:31] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5061e8781ddf...9a687db6f255 [12:31:32] [02miraheze/mw-config] 07AgentIsai 039a687db - Add rights to ManageWiki blacklist (#4931) [12:32:39] miraheze/mw-config - Reception123 the build passed. [12:34:29] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.63, 2.92, 2.12 [12:35:38] [02mw-config] 07Reception123 closed pull request 03#5109: Fix for T10430 wgKartographerDfltStyle - 13https://github.com/miraheze/mw-config/pull/5109 [12:35:40] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9a687db6f255...b0e51d6fd134 [12:35:42] [02miraheze/mw-config] 07ugochimobi 03b0e51d6 - Fix for T10430 wgKartographerDfltStyle (#5109) [12:36:36] miraheze/mw-config - Reception123 the build passed. [12:36:40] [02mw-config] 07Reception123 synchronize pull request 03#5108: set wgKartographerSimpleStyleMarkers to false for gratisdatawiki - 13https://github.com/miraheze/mw-config/pull/5108 [12:36:59] [02mw-config] 07Reception123 synchronize pull request 03#5108: set wgKartographerSimpleStyleMarkers to false for gratisdatawiki - 13https://github.com/miraheze/mw-config/pull/5108 [12:37:36] miraheze/mw-config - Reception123 the build passed. [12:37:41] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.73, 11.53, 11.16 [12:37:56] miraheze/mw-config - Reception123 the build passed. [12:38:07] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b0e51d6fd134...b0f0b5fb335d [12:38:08] [02miraheze/mw-config] 07ugochimobi 03b0f0b5f - set wgKartographerSimpleStyleMarkers to false for gratisdatawiki (#5108) [12:38:09] [02mw-config] 07Reception123 closed pull request 03#5108: set wgKartographerSimpleStyleMarkers to false for gratisdatawiki - 13https://github.com/miraheze/mw-config/pull/5108 [12:38:19] [02mw-config] 07OwenBaines commented on pull request 03#5036: T9540: set `$wgMirahezeReportsBlockAlertKeywords` - 13https://github.com/miraheze/mw-config/pull/5036#issuecomment-1417720535 [12:38:29] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.64, 2.87, 2.30 [12:39:07] miraheze/mw-config - Reception123 the build passed. [12:40:12] !log [@mwtask141] starting deploy of {'config': True} to all [12:40:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:40:19] !log [@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [12:40:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:41:30] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 7.71, 10.39, 11.06 [12:45:19] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/cf502f08800a...fbd6da1af8a4 [12:45:22] [02miraheze/mw-config] 07Reception123 03fbd6da1 - Update LocalSettings.php [12:45:25] [02mw-config] 07Reception123 synchronize pull request 03#5036: T9540: set `$wgMirahezeReportsBlockAlertKeywords` - 13https://github.com/miraheze/mw-config/pull/5036 [12:46:33] miraheze/mw-config - Reception123 the build passed. [12:47:26] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.93, 9.06, 10.18 [12:51:05] [02miraheze/mw-config] 07Reception123 deleted branch 03Universal-Omega-patch-2 [12:51:06] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b0f0b5fb335d...e7f4c4e2b349 [12:51:09] [02miraheze/mw-config] 07Universal-Omega 03e7f4c4e - T9540: set `$wgMirahezeReportsBlockAlertKeywords` (#5036) [12:51:11] [02mw-config] 07Reception123 closed pull request 03#5036: T9540: set `$wgMirahezeReportsBlockAlertKeywords` - 13https://github.com/miraheze/mw-config/pull/5036 [12:51:12] [02mw-config] 07Reception123 deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [12:51:21] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.01, 10.59, 10.53 [12:52:09] miraheze/mw-config - Reception123 the build passed. [12:53:18] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.95, 10.39, 10.47 [12:53:20] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.97, 12.03, 11.06 [12:55:02] !log [reception@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [12:55:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:55:09] !log [reception@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 7s [12:55:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:56:45] !log [@test131] starting deploy of {'config': True} to all [12:56:46] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [12:56:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:56:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:57:12] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.28, 11.21, 10.76 [12:57:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.52, 11.93, 11.28 [12:59:16] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.30, 12.47, 11.56 [13:01:07] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.51, 11.28, 10.94 [13:01:15] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.20, 11.76, 11.41 [13:03:04] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.16, 11.66, 11.13 [13:03:13] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.27, 11.73, 11.43 [13:05:01] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.04, 10.85, 10.89 [13:07:02] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 13.80, 11.11, 5.51 [13:08:29] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.34, 3.25, 2.25 [13:09:09] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.68, 11.43, 11.55 [13:10:52] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.04, 10.82, 10.75 [13:11:39] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [13:12:03] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c17bb118b7c4...2486c276dda6 [13:12:06] [02miraheze/puppet] 07Reception123 032486c27 - add fastly.jsdelivr.net to CSP [13:13:06] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.24, 11.88, 11.65 [13:14:30] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.40, 3.78, 2.89 [13:14:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.25, 11.03, 10.93 [13:16:30] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.80, 5.48, 3.63 [13:16:47] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 0.88, 3.02, 3.98 [13:18:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.54, 11.79, 11.26 [13:20:30] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.62, 3.69, 3.36 [13:20:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.99, 11.02, 11.06 [13:20:41] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.06, 2.08, 3.38 [13:22:30] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.59, 2.94, 3.12 [13:22:40] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 13.35, 11.62, 11.25 [13:28:55] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.53, 11.46, 11.89 [13:34:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.80, 11.94, 11.98 [13:34:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.26, 12.14, 12.01 [13:36:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.53, 11.57, 11.82 [13:39:39] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [13:48:42] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.13, 10.60, 10.95 [13:50:40] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.25, 10.48, 10.88 [13:56:36] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.33, 11.77, 11.21 [13:56:41] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.69, 12.24, 11.31 [14:06:33] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.62, 4.22, 2.87 [14:08:33] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.86, 3.34, 2.71 [14:27:08] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.93, 3.36, 2.57 [14:29:08] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 3.10, 3.17, 2.59 [14:38:35] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.60, 4.72, 3.32 [14:40:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.82, 3.58, 3.07 [14:42:35] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.16, 3.22, 3.00 [14:48:36] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.26, 3.41, 3.11 [14:50:36] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.65, 4.29, 3.53 [14:52:36] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.60, 3.26, 3.24 [14:55:09] [02mw-config] 07ugochimobi commented on pull request 03#5019: Wikibase client and repo settings that are currently missing - 13https://github.com/miraheze/mw-config/pull/5019#issuecomment-1418002228 [15:00:40] RECOVERY - cloud11 IPMI Sensors on cloud11 is OK: IPMI Status: OK [15:04:40] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 7 = Critical] [15:08:37] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.09, 3.24, 2.91 [15:10:37] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.22, 3.31, 2.99 [15:28:54] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 6.26, 4.05, 2.56 [15:29:35] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=kaju10wiki --no-updates --username-prefix=ja /home/reception/ImportDump/kaju10wiki-20230205143507.xml (START) [15:29:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:29:42] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=kaju10wiki --no-updates --username-prefix=ja /home/reception/ImportDump/kaju10wiki-20230205143507.xml (END - exit=0) [15:29:43] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=kaju10wiki (START) [15:29:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:29:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:30:27] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=kaju10wiki (END - exit=0) [15:30:28] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=kaju10wiki --active --update (END - exit=0) [15:30:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:30:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:30:54] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.49, 3.36, 2.48 [16:26:19] [02mw-config] 07Universal-Omega reviewed pull request 03#5101 commit - 13https://github.com/miraheze/mw-config/pull/5101#discussion_r1096734876 [16:28:54] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.61, 2.93, 2.10 [16:30:55] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.52, 3.46, 2.39 [16:30:56] [02mw-config] 07Universal-Omega reviewed pull request 03#5101 commit - 13https://github.com/miraheze/mw-config/pull/5101#discussion_r1096735542 [16:36:54] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.96, 3.91, 2.96 [16:37:38] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.77, 3.32, 2.54 [16:40:54] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.26, 2.64, 2.67 [16:45:34] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#57: T9999: add `$wgIncidentReportingInactiveServices` - 13https://github.com/miraheze/IncidentReporting/pull/57 [16:45:36] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03add-wgIncidentReportingInactiveServices [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/50a2d279f62d...2ddaf241b0d3 [16:45:39] [02miraheze/IncidentReporting] 07Universal-Omega 032ddaf24 - Update IncidentReportingFormFactory.php [16:45:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.62, 3.99, 3.32 [16:47:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.32, 3.31, 3.14 [16:48:21] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#57: T9999: add `$wgIncidentReportingInactiveServices` - 13https://github.com/miraheze/IncidentReporting/pull/57 [16:48:23] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03add-wgIncidentReportingInactiveServices [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/2ddaf241b0d3...58d9cccd11a6 [16:48:26] [02miraheze/IncidentReporting] 07Universal-Omega 0358d9ccc - Update SpecialIncidentReports.php [16:49:04] miraheze/IncidentReporting - Universal-Omega the build passed. [16:52:09] miraheze/IncidentReporting - Universal-Omega the build passed. [16:52:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.81, 11.26, 11.91 [16:54:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.98, 11.85, 12.04 [16:54:49] [02IncidentReporting] 07Universal-Omega synchronize pull request 03#57: T9999: add `$wgIncidentReportingInactiveServices` - 13https://github.com/miraheze/IncidentReporting/pull/57 [16:54:50] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03add-wgIncidentReportingInactiveServices [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/58d9cccd11a6...3ea16c0d224a [16:54:51] [02miraheze/IncidentReporting] 07Universal-Omega 033ea16c0 - Update extension.json [16:58:12] miraheze/IncidentReporting - Universal-Omega the build passed. [16:59:25] [02mw-config] 07Universal-Omega reviewed pull request 03#5101 commit - 13https://github.com/miraheze/mw-config/pull/5101#discussion_r1096739676 [17:04:08] PROBLEM - wiki.zulaclub.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zulaclub.net All nameservers failed to answer the query. [17:08:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 3.56, 5.25, 4.19 [17:14:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.44, 11.28, 11.93 [17:14:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.89, 3.83, 3.96 [17:18:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.46, 10.86, 11.69 [17:20:40] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.01, 2.71, 3.35 [17:24:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 14.18, 10.79, 11.12 [17:24:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.17, 10.57, 11.19 [17:24:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.42, 3.74, 3.62 [17:26:40] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.92, 3.39, 3.50 [17:28:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.90, 10.97, 11.17 [17:30:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 15.20, 12.04, 11.50 [17:30:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.48, 3.57, 3.52 [17:32:33] PROBLEM - index.pdcommunity.ir - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query pdcommunity.ir. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:32:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.82, 3.36, 3.45 [17:33:17] RECOVERY - wiki.zulaclub.net - reverse DNS on sslhost is OK: SSL OK - wiki.zulaclub.net reverse DNS resolves to cp23.miraheze.org - CNAME OK [17:34:41] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.50, 3.09, 3.35 [17:36:56] PROBLEM - wiki.phgalaxy.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.phgalaxy.eu.org All nameservers failed to answer the query. [17:38:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.08, 11.56, 11.99 [17:38:41] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.02, 5.67, 4.50 [17:40:29] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.27, 11.37, 11.81 [17:42:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.42, 11.82, 11.95 [17:43:28] PROBLEM - phgalaxy.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - phgalaxy.eu.org All nameservers failed to answer the query. [17:44:29] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.13, 11.65, 11.82 [17:46:03] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 3.87, 2.58, 1.79 [17:46:41] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.25, 2.96, 3.84 [17:48:03] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 1.55, 2.16, 1.73 [17:50:41] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.41, 2.29, 3.37 [18:00:10] PROBLEM - graylog121 Current Load on graylog121 is WARNING: WARNING - load average: 3.66, 3.59, 3.05 [18:01:27] RECOVERY - index.pdcommunity.ir - reverse DNS on sslhost is OK: SSL OK - index.pdcommunity.ir reverse DNS resolves to cp23.miraheze.org - CNAME OK [18:04:03] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.54, 3.22, 3.03 [18:07:43] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.63, 5.41, 4.01 [18:13:43] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.29, 3.45, 3.59 [18:15:43] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.32, 2.69, 3.29 [18:18:35] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [18:23:43] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.87, 3.10, 3.16 [18:28:35] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [18:29:43] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.46, 2.87, 3.07 [18:29:47] [02mw-config] 07Reception123 reviewed pull request 03#5101 commit - 13https://github.com/miraheze/mw-config/pull/5101#discussion_r1096784325 [18:30:22] [02mw-config] 07Reception123 commented on pull request 03#5101: set wgIncidentReportingInactiveServices - 13https://github.com/miraheze/mw-config/pull/5101#issuecomment-1418208747 [18:30:25] [02mw-config] 07Reception123 closed pull request 03#5101: set wgIncidentReportingInactiveServices - 13https://github.com/miraheze/mw-config/pull/5101 [18:33:10] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.03, 3.33, 2.89 [18:35:08] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.90, 3.12, 2.86 [18:35:43] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.50, 4.48, 3.65 [18:43:44] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.20, 3.39, 3.58 [18:47:44] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.97, 2.78, 3.29 [19:07:44] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.19, 3.49, 3.23 [19:11:20] !log [reception@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/invalidateUserSessions.php --wiki=loginwiki --user=BrandonWM (END - exit=0) [19:11:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:13:47] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 3.08, 3.39, 3.28 [19:18:41] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1767446995 secs [19:32:41] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.07500517368 secs [19:36:48] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.82, 4.42, 3.50 [19:44:48] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.53, 3.92, 3.84 [19:54:48] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.53, 2.52, 3.17 [20:04:48] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.67, 3.86, 3.20 [20:08:05] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [20:14:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.21, 3.83, 3.72 [20:18:05] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [20:22:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.76, 2.78, 3.31 [20:25:17] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.45, 3.74, 3.03 [20:27:14] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.64, 2.97, 2.84 [20:37:49] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.57, 4.02, 3.52 [20:39:49] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.93, 3.61, 3.43 [20:41:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.31, 2.75, 3.13 [21:02:41] PROBLEM - cp33 NTP time on cp33 is WARNING: NTP WARNING: Offset 0.1791372299 secs [21:05:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.75, 4.28, 3.46 [21:13:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.64, 3.53, 3.57 [21:14:41] RECOVERY - cp33 NTP time on cp33 is OK: NTP OK: Offset 0.087798208 secs [21:17:52] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.03, 3.05, 3.38 [21:21:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.83, 3.59, 3.53 [21:23:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [21:23:52] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.44, 3.09, 3.35 [21:28:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [21:38:52] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.37, 4.25, 3.60 [21:40:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.39, 3.51, 3.41 [21:42:52] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.95, 3.25, 3.32 [21:52:53] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.38, 3.78, 3.36 [21:54:55] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.25, 4.01, 2.91 [21:56:52] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.24, 3.72, 2.93 [21:56:53] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.30, 3.53, 3.39 [21:58:49] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.27, 3.17, 2.82 [21:59:26] [02mw-config] 07redbluegreenhat opened pull request 03#5110: T10440: Add some Pivot settings to ManageWiki and set "showActionsForAnon" = false for wanderingstarswiki - 13https://github.com/miraheze/mw-config/pull/5110 [22:00:32] miraheze/mw-config - redbluegreenhat the build passed. [22:02:53] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.44, 4.04, 3.65 [22:20:15] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 38.25, 51.59, 25.04 [22:23:52] [02MirahezeMagic] 07dmehus opened pull request 03#404: Add MirahezeMagic interface message for local+global flood groups - 13https://github.com/miraheze/MirahezeMagic/pull/404 [22:24:57] [02MirahezeMagic] 07dmehus synchronize pull request 03#404: Add MirahezeMagic interface message for local+global flood groups - 13https://github.com/miraheze/MirahezeMagic/pull/404 [22:28:54] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.84, 3.10, 3.95 [22:33:34] miraheze/MirahezeMagic - dmehus the build passed. [22:33:45] I noticed in the pending checks, there's no longer a specific task to notify IRC. Is that because it's embedded within whatever notification system we're using? [22:34:16] Nevermind, spoke too soon! [22:34:25] miraheze/MirahezeMagic - dmehus the build passed. [22:34:33] > Quibble and Phan / notify-irc (pull_request) In progress — This check has started... [22:34:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.82, 3.16, 3.64 [22:38:55] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.41, 3.94, 3.91 [22:42:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.09, 3.71, 3.79 [22:44:55] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.58, 3.27, 3.61 [22:46:55] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.36, 3.34, 3.57 [22:50:55] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.41, 3.74, 3.71 [22:54:55] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.07, 2.76, 3.30 [23:06:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.84, 2.57, 3.76 [23:08:03] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.30, 3.57, 4.04 [23:08:56] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.92, 3.83, 3.37 [23:10:03] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.56, 2.79, 3.69 [23:10:56] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.03, 3.10, 3.15 [23:20:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.92, 2.47, 3.20 [23:26:56] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.99, 3.67, 3.25 [23:28:56] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.47, 3.27, 3.16 [23:36:56] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 7.96, 4.91, 3.72 [23:38:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 8.38, 10.16, 11.70 [23:44:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 15.28, 12.20, 11.93 [23:46:57] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.38, 3.84, 3.94 [23:52:38] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.73, 3.95, 3.35 [23:52:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.33, 11.62, 11.94 [23:54:35] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.36, 3.52, 3.28 [23:54:39] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.82, 12.13, 12.08 [23:56:32] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.65, 3.33, 3.24 [23:56:39] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.96, 11.77, 11.94 [23:56:57] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.69, 2.97, 3.40