[00:00:56] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 8.15, 9.82, 10.17 [00:01:40] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 6.53, 4.38, 2.85 [00:02:19] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 5.22, 3.72, 2.88 [00:03:16] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=rlabwiki --no-updates /home/macfan/ImportDump/rlabwiki-20231109093551.xml (END - exit=0) [00:03:17] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=rlabwiki (START) [00:03:39] RECOVERY - os131 Disk Space on os131 is OK: DISK OK - free space: / 22234MiB (11% inode=99%); [00:03:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:03:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:03:57] PROBLEM - db112 Current Load on db112 is CRITICAL: CRITICAL - load average: 17.26, 9.31, 4.30 [00:04:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-12 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/e5b26a78034f [00:04:20] [02miraheze/puppet] 07paladox 03e5b26a7 - prometheus-jobqueue-stats: Fix performance issues [00:04:22] [02puppet] 07paladox created branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [00:04:23] [02puppet] 07paladox opened pull request 03#3501: prometheus-jobqueue-stats: Fix performance issues - 13https://github.com/miraheze/puppet/pull/3501 [00:05:49] [02puppet] 07paladox closed pull request 03#3501: prometheus-jobqueue-stats: Fix performance issues - 13https://github.com/miraheze/puppet/pull/3501 [00:05:50] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/fda7df0c3528...b5aecb7d615d [00:05:52] [02miraheze/puppet] 07paladox 03b5aecb7 - prometheus-jobqueue-stats: Fix performance issues (#3501) [00:05:55] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-12 [00:05:57] [02puppet] 07paladox deleted branch 03paladox-patch-12 - 13https://github.com/miraheze/puppet [00:06:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.75, 6.27, 6.75 [00:06:22] PROBLEM - swiftac111 Current Load on swiftac111 is CRITICAL: CRITICAL - load average: 8.15, 6.95, 5.38 [00:07:02] PROBLEM - db131 Current Load on db131 is CRITICAL: CRITICAL - load average: 11.47, 8.95, 5.59 [00:07:39] PROBLEM - os131 Disk Space on os131 is WARNING: DISK WARNING - free space: / 21748MiB (10% inode=99%); [00:07:49] RECOVERY - jobchron121 Puppet on jobchron121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:08:26] RECOVERY - swiftac111 Current Load on swiftac111 is OK: OK - load average: 4.99, 6.16, 5.30 [00:09:53] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/868c9ee7f510...5493d20d9d66 [00:09:55] [02miraheze/mw-config] 07paladox 035493d20 - Enable warmup again on select wikis [00:10:08] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [00:10:27] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 18s [00:10:29] !log [paladox@mwtask141] starting deploy of {'config': True} to all [00:10:37] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [00:10:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:10:53] miraheze/mw-config - paladox the build passed. [00:11:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:11:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:11:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:12:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.22, 6.77, 6.86 [00:13:30] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=rlabwiki (END - exit=0) [00:13:31] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=rlabwiki --active --update (END - exit=0) [00:13:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:13:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:14:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.52, 6.47, 6.75 [00:14:45] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 4.83, 7.38, 6.31 [00:14:49] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 14.19, 11.25, 10.32 [00:16:41] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 4.28, 6.53, 6.14 [00:17:48] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5493d20d9d66...5cd658eaf0a9 [00:17:49] [02miraheze/mw-config] 07paladox 035cd658e - Enable warmup on more wikis [00:17:49] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [00:17:59] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [00:18:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:00] !log [paladox@mwtask141] starting deploy of {'config': True} to all [00:18:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:08] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [00:18:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:18:47] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 9.64, 11.39, 10.67 [00:19:07] miraheze/mw-config - paladox the build passed. [00:19:23] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [00:19:57] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 2.33, 6.35, 7.94 [00:20:34] PROBLEM - db131 Current Load on db131 is CRITICAL: CRITICAL - load average: 16.85, 11.12, 7.94 [00:20:43] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b5aecb7d615d...11bc582a6697 [00:20:44] [02miraheze/puppet] 07paladox 0311bc582 - Lower memcache size slightly [00:20:46] RECOVERY - mwtask141 Current Load on mwtask141 is OK: OK - load average: 7.17, 9.85, 10.19 [00:21:28] PROBLEM - graylog131 PowerDNS Recursor on graylog131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [00:22:16] PROBLEM - os131 Puppet on os131 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [00:23:27] RECOVERY - graylog131 PowerDNS Recursor on graylog131 is OK: DNS OK: 1.839 second response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [00:24:22] PROBLEM - os131 Puppet on os131 is UNKNOWN: NRPE: Unable to read output [00:25:57] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 2.91, 4.85, 6.78 [00:26:43] !log [@test131] starting deploy of {'config': True} to all [00:26:44] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 0s [00:27:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/11bc582a6697...e4ed5300ec11 [00:27:07] [02miraheze/puppet] 07paladox 03e4ed530 - Revert "Lower memcache size slightly" [00:35:38] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.04, 10.51, 9.75 [00:41:35] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.36, 11.36, 10.42 [00:46:02] PROBLEM - swiftobject113 Current Load on swiftobject113 is WARNING: WARNING - load average: 7.49, 6.50, 5.66 [00:48:02] RECOVERY - swiftobject113 Current Load on swiftobject113 is OK: OK - load average: 4.95, 5.90, 5.55 [00:52:59] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.67, 7.06, 6.56 [00:54:56] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.65, 6.58, 6.44 [00:57:28] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.43, 10.76, 10.55 [01:00:45] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/5cd658eaf0a9...63481546523a [01:00:46] [02miraheze/mw-config] 07paladox 036348154 - Enable warmup on all [01:01:09] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [01:01:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:01:17] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [01:01:19] !log [paladox@mwtask141] starting deploy of {'config': True} to all [01:01:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:01:26] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 7s [01:01:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:01:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:01:38] miraheze/mw-config - paladox the build passed. [01:04:08] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 2.57, 4.77, 7.51 [01:08:08] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 3.69, 4.24, 6.68 [01:11:21] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 9.78, 11.57, 11.49 [01:12:08] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 6.01, 5.98, 6.86 [01:13:20] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.05, 11.86, 11.61 [01:14:08] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 3.53, 5.20, 6.47 [01:17:34] PROBLEM - os131 Disk Space on os131 is CRITICAL: DISK CRITICAL - free space: / 11678MiB (5% inode=99%); [01:18:19] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.18, 3.29, 3.90 [01:20:55] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 1.73, 2.60, 3.97 [01:21:33] RECOVERY - os131 Disk Space on os131 is OK: DISK OK - free space: / 25682MiB (12% inode=99%); [01:22:55] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.22, 3.18, 4.02 [01:24:56] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.67, 3.35, 3.98 [01:26:19] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.28, 3.61, 3.74 [01:26:55] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 6.09, 4.36, 4.27 [01:27:36] !log [@test131] starting deploy of {'config': True} to all [01:27:37] !log [@test131] finished deploy of {'config': True} to all - SUCCESS in 1s [01:27:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:28:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:31:01] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 7.19, 6.18, 5.79 [01:33:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.37, 7.52, 6.67 [01:34:52] PROBLEM - db131 Current Load on db131 is CRITICAL: CRITICAL - load average: 9.09, 7.39, 6.34 [01:35:26] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=lucydndwiki --no-updates --username-prefix=wikia:lucydnd /home/macfan/ImportDump/lucydndwiki-20231109103703.xml (START) [01:35:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.55, 7.12, 6.62 [01:35:39] PROBLEM - os131 Disk Space on os131 is WARNING: DISK WARNING - free space: / 20611MiB (10% inode=99%); [01:35:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:36:20] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=lucydndwiki --no-updates --username-prefix=wikia:lucydnd /home/macfan/ImportDump/lucydndwiki-20231109103703.xml (END - exit=0) [01:36:21] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=lucydndwiki (START) [01:36:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:36:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:09] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.15, 11.10, 11.81 [01:37:32] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=lucydndwiki (END - exit=0) [01:37:33] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=lucydndwiki --active --update (END - exit=0) [01:37:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:39:05] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=megaverseoftheboxwiki --no-updates --username-prefix=wikia:megaverse-of-the-box /home/macfan/ImportDump/megaverseoftheboxwiki-20231109143638.xml (START) [01:39:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:39:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 3.93, 5.87, 6.27 [01:41:07] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.29, 11.28, 11.70 [01:42:37] PROBLEM - matomo121 Current Load on matomo121 is WARNING: LOAD WARNING - total load average: 3.82, 3.44, 3.17 [01:44:32] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 4.10, 7.89, 7.68 [01:44:37] RECOVERY - matomo121 Current Load on matomo121 is OK: LOAD OK - total load average: 2.86, 3.28, 3.15 [01:45:22] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=megaverseoftheboxwiki --no-updates --username-prefix=wikia:megaverse-of-the-box /home/macfan/ImportDump/megaverseoftheboxwiki-20231109143638.xml (END - exit=0) [01:45:23] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=megaverseoftheboxwiki (START) [01:45:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:45:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:48:53] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.03, 7.16, 6.61 [01:50:19] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 3.90, 5.03, 6.42 [01:50:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.46, 7.20, 6.69 [01:52:19] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.35, 3.36, 3.88 [01:52:47] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.83, 6.56, 6.51 [01:52:57] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.11, 3.29, 3.94 [02:00:19] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.35, 3.35, 3.64 [02:00:56] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.63, 3.59, 3.74 [02:02:55] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.93, 3.68, 3.75 [02:04:57] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 4.14, 3.88, 3.82 [02:05:33] RECOVERY - os131 Disk Space on os131 is OK: DISK OK - free space: / 25971MiB (12% inode=99%); [02:06:19] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.67, 3.96, 3.87 [02:06:58] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 3.86, 3.76, 3.78 [02:10:19] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.88, 4.26, 3.99 [02:10:56] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.82, 4.47, 4.05 [02:12:33] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=megaverseoftheboxwiki (END - exit=0) [02:12:34] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=megaverseoftheboxwiki --active --update (END - exit=0) [02:12:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:13:01] PROBLEM - db131 Current Load on db131 is CRITICAL: CRITICAL - load average: 9.83, 8.42, 6.82 [02:13:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:16:51] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 8.71, 10.80, 11.99 [02:17:34] PROBLEM - os131 Disk Space on os131 is WARNING: DISK WARNING - free space: / 21093MiB (10% inode=99%); [02:19:34] PROBLEM - os131 PowerDNS Recursor on os131 is CRITICAL: CRITICAL - Plugin timed out while executing system call [02:20:57] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.99, 6.52, 6.32 [02:21:30] RECOVERY - os131 PowerDNS Recursor on os131 is OK: DNS OK: 0.718 seconds response time. miraheze.org returns 2001:41d0:801:2000::3a18,2001:41d0:801:2000::5d68,51.195.201.140,51.89.139.24 [02:22:54] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.79, 6.20, 6.22 [02:28:42] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.81, 7.02, 6.64 [02:30:39] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 9.35, 7.88, 6.99 [02:32:36] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 5.67, 7.22, 6.87 [02:32:43] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.27, 10.87, 10.90 [02:32:59] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=physicalanthropologywiki --no-updates --username-prefix=wikia:physicalanthropologywiki /home/macfan/ImportDump/physicalanthropologywiki-20231109154734.xml (START) [02:33:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:34:14] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 3.38, 6.49, 7.80 [02:34:33] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.09, 7.81, 7.14 [02:36:19] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.77, 7.28, 6.57 [02:36:30] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.37, 7.27, 7.02 [02:40:08] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.65, 6.54, 6.46 [02:40:08] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 2.55, 4.39, 6.48 [02:42:19] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.93, 3.35, 3.99 [02:42:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [02:45:50] PROBLEM - swiftobject101 Current Load on swiftobject101 is CRITICAL: CRITICAL - load average: 8.33, 7.11, 6.66 [02:47:45] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.99, 7.34, 6.79 [02:48:56] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 1.92, 2.88, 3.95 [02:52:19] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.10, 2.65, 3.35 [02:53:33] RECOVERY - os131 Disk Space on os131 is OK: DISK OK - free space: / 23783MiB (11% inode=99%); [02:54:57] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 1.67, 2.30, 3.37 [02:55:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.58, 6.28, 6.54 [02:58:19] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.75, 3.22, 3.35 [02:58:57] PROBLEM - os131 Current Load on os131 is CRITICAL: LOAD CRITICAL - total load average: 5.55, 3.81, 3.74 [02:59:33] PROBLEM - os131 Disk Space on os131 is WARNING: DISK WARNING - free space: / 20219MiB (10% inode=99%); [03:00:08] PROBLEM - db131 Current Load on db131 is CRITICAL: CRITICAL - load average: 12.66, 7.59, 5.90 [03:00:19] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.54, 3.73, 3.53 [03:04:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.71, 6.45, 6.78 [03:04:29] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.16, 10.71, 11.96 [03:08:27] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.88, 11.44, 11.92 [03:12:08] PROBLEM - db131 Current Load on db131 is WARNING: WARNING - load average: 3.47, 7.07, 7.48 [03:12:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [03:16:08] RECOVERY - db131 Current Load on db131 is OK: OK - load average: 3.46, 5.03, 6.52 [03:16:19] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 2.61, 3.55, 3.95 [03:16:52] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=physicalanthropologywiki --no-updates --username-prefix=wikia:physicalanthropologywiki /home/macfan/ImportDump/physicalanthropologywiki-20231109154734.xml (END - exit=0) [03:16:53] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=physicalanthropologywiki (START) [03:16:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:17:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:20:50] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.26, 6.82, 6.67 [03:22:47] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.45, 6.50, 6.56 [03:25:02] PROBLEM - os131 Current Load on os131 is WARNING: LOAD WARNING - total load average: 2.78, 3.42, 3.93 [03:29:33] RECOVERY - os131 Disk Space on os131 is OK: DISK OK - free space: / 22674MiB (11% inode=99%); [03:32:20] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.03, 3.17, 3.40 [03:32:24] RECOVERY - ns1 NTP time on ns1 is OK: NTP OK: Offset 0.008543223143 secs [03:32:55] RECOVERY - os131 Current Load on os131 is OK: LOAD OK - total load average: 2.17, 2.66, 3.38 [03:41:33] PROBLEM - os131 Disk Space on os131 is WARNING: DISK WARNING - free space: / 21636MiB (10% inode=99%); [03:44:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 7.98, 7.25, 6.68 [03:44:55] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.78, 3.01, 3.08 [03:46:21] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [03:46:50] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.47, 2.76, 2.98 [03:48:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.73, 6.77, 6.61 [03:49:33] RECOVERY - os131 Disk Space on os131 is OK: DISK OK - free space: / 23885MiB (11% inode=99%); [03:52:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.37, 7.12, 6.75 [03:54:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.51, 6.68, 6.63 [04:04:43] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=physicalanthropologywiki (END - exit=0) [04:04:44] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=physicalanthropologywiki --active --update (END - exit=0) [04:04:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:04:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:07:33] PROBLEM - os131 Disk Space on os131 is WARNING: DISK WARNING - free space: / 20812MiB (10% inode=99%); [04:09:24] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=trotuserverwiki --no-updates --username-prefix=w /home/macfan/ImportDump/trotuserverwiki-20231109162838.xml (START) [04:09:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:09:33] RECOVERY - os131 Disk Space on os131 is OK: DISK OK - free space: / 22295MiB (11% inode=99%); [04:10:06] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=trotuserverwiki --no-updates --username-prefix=w /home/macfan/ImportDump/trotuserverwiki-20231109162838.xml (END - exit=0) [04:10:07] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=trotuserverwiki (START) [04:10:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:10:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:12:21] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [04:15:33] PROBLEM - os131 Disk Space on os131 is WARNING: DISK WARNING - free space: / 21792MiB (10% inode=99%); [04:16:40] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=trotuserverwiki (END - exit=0) [04:16:41] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=trotuserverwiki --active --update (END - exit=0) [04:16:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:16:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:18:04] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [04:18:41] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=umbrawiki --no-updates --username-prefix=wikia:pokemon-umbra /home/macfan/ImportDump/umbrawiki-20231109195752.xml (START) [04:18:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:21:03] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=umbrawiki --no-updates --username-prefix=wikia:pokemon-umbra /home/macfan/ImportDump/umbrawiki-20231109195752.xml (END - exit=0) [04:21:04] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=umbrawiki (START) [04:21:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:21:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:25:34] RECOVERY - os131 Disk Space on os131 is OK: DISK OK - free space: / 22575MiB (11% inode=99%); [04:31:26] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=umbrawiki (END - exit=0) [04:31:27] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=umbrawiki --active --update (END - exit=0) [04:31:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:31:34] PROBLEM - os131 Disk Space on os131 is WARNING: DISK WARNING - free space: / 20965MiB (10% inode=99%); [04:31:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:33:08] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=eposimulationwiki --no-updates --username-prefix=w /home/macfan/ImportDump/eposimulationwiki-20231110002446.xml (START) [04:33:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:36:27] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=eposimulationwiki --no-updates --username-prefix=w /home/macfan/ImportDump/eposimulationwiki-20231110002446.xml (END - exit=0) [04:36:28] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=eposimulationwiki (START) [04:36:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:36:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:40:36] PROBLEM - mail121 Current Load on mail121 is CRITICAL: LOAD CRITICAL - total load average: 2.79, 2.05, 0.90 [04:42:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [04:44:36] RECOVERY - mail121 Current Load on mail121 is OK: LOAD OK - total load average: 0.72, 1.63, 1.01 [04:47:42] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=eposimulationwiki (END - exit=0) [04:47:43] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=eposimulationwiki --active --update (END - exit=0) [04:47:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:47:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:54:25] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=robloxbaldiwikiwiki --no-updates /home/macfan/ImportDump/robloxbaldiwikiwiki-20231110144659.xml (START) [04:54:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:56:59] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=robloxbaldiwikiwiki --no-updates /home/macfan/ImportDump/robloxbaldiwikiwiki-20231110144659.xml (END - exit=0) [04:57:00] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=robloxbaldiwikiwiki (START) [04:57:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:57:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:00:26] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=robloxbaldiwikiwiki (END - exit=0) [05:00:27] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=robloxbaldiwikiwiki --active --update (END - exit=0) [05:00:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:00:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:16:24] PROBLEM - ns1 NTP time on ns1 is WARNING: NTP WARNING: Offset 0.1017264128 secs [06:44:10] PROBLEM - pyramidgames.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address pyramidgames.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [06:46:04] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [07:00:30] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [07:14:10] PROBLEM - pyramidgames.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pyramidgames.wiki' expires in 8 day(s) (Wed 22 Nov 2023 14:28:59 GMT +0000). [07:18:04] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [07:32:19] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.54, 3.60, 2.93 [07:34:20] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.66, 3.36, 2.93 [07:40:19] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.59, 3.45, 3.09 [07:42:19] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.38, 3.35, 3.10 [07:48:23] PROBLEM - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [08:12:21] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [08:17:59] RECOVERY - wiki.tmyt105.leyhp.com - reverse DNS on sslhost is OK: SSL OK - wiki.tmyt105.leyhp.com reverse DNS resolves to cp24.miraheze.org - CNAME OK [08:20:30] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [08:36:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.08, 11.04, 11.94 [08:42:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.gab.pt.eu.org. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [08:57:02] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [08:57:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:57:10] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [08:57:13] !log [paladox@mwtask141] starting deploy of {'config': True} to all [08:57:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:57:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:57:18] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 5s [08:57:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:10:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.32, 12.00, 11.52 [09:12:21] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [09:16:22] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [09:31:32] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.67, 6.64, 5.95 [09:33:29] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.12, 6.40, 5.94 [09:42:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [09:48:09] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:17:43] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [10:43:33] PROBLEM - os131 Disk Space on os131 is CRITICAL: DISK CRITICAL - free space: / 11984MiB (5% inode=99%); [10:44:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.31, 11.24, 11.93 [10:45:33] PROBLEM - os131 Disk Space on os131 is WARNING: DISK WARNING - free space: / 12165MiB (6% inode=99%); [10:48:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.05, 11.58, 11.91 [10:49:13] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [10:56:55] PROBLEM - cloud11 IPMI Sensors on cloud11 is UNKNOWN:> Your server seems to be powered off. (Execution of FreeIPMI returned an empty output or only 1 header row!)-> /usr/sbin/ipmi-sensors was executed with the following parameters: [HIDDEN]sudo /usr/sbin/ipmi-sensors --exclude-sensor-types Memory --quiet-cache --sdr-cache-recreate --interpret-oem-data --output-sensor-state --ignore-not-available-sensors --output-sensor-thresholds [10:58:55] PROBLEM - cloud11 IPMI Sensors on cloud11 is CRITICAL: IPMI Status: Critical [Cntlr 2 Bay 8 = Critical] [11:02:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.45, 11.20, 11.79 [11:06:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.84, 11.92, 11.93 [11:08:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.74, 11.82, 11.90 [11:11:33] PROBLEM - os131 Disk Space on os131 is CRITICAL: DISK CRITICAL - free space: / 11969MiB (5% inode=99%); [11:16:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.47, 11.89, 11.85 [11:18:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.97, 11.64, 11.77 [11:27:33] PROBLEM - os131 Disk Space on os131 is WARNING: DISK WARNING - free space: / 12225MiB (6% inode=99%); [11:35:17] [Grafana] !sre FIRING: There has been a rise in the MediaWiki exception rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [11:36:27] PROBLEM - db142 Current Load on db142 is CRITICAL: CRITICAL - load average: 8.39, 7.54, 3.39 [11:37:33] PROBLEM - os131 Disk Space on os131 is CRITICAL: DISK CRITICAL - free space: / 11727MiB (5% inode=99%); [11:38:27] RECOVERY - db142 Current Load on db142 is OK: OK - load average: 1.74, 5.27, 3.06 [11:40:03] [Grafana] !sre RESOLVED: MediaWiki Exception Rate https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [11:59:33] PROBLEM - os131 Disk Space on os131 is WARNING: DISK WARNING - free space: / 12455MiB (6% inode=99%); [12:06:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.36, 11.58, 11.21 [12:08:35] PROBLEM - db131 Backups SQL on db131 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864171 seconds old and 129622 bytes [12:10:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.64, 11.81, 11.39 [12:12:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.20, 12.02, 11.53 [12:16:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.16, 7.35, 6.32 [12:16:19] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.54, 3.23, 2.96 [12:17:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.02, 5.90, 5.40 [12:18:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is WARNING: WARNING - load average: 6.98, 6.89, 6.27 [12:18:19] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.07, 2.81, 2.84 [12:19:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.48, 5.37, 5.27 [12:20:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 6.55, 6.72, 6.28 [12:45:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.65, 6.21, 5.46 [12:47:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.48, 5.83, 5.41 [13:03:05] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.73, 3.42, 2.92 [13:04:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 9.94, 11.20, 11.92 [13:05:00] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.17, 3.35, 2.96 [13:38:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.37, 11.56, 11.46 [13:40:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.75, 11.62, 11.50 [13:42:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [13:50:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.19, 11.75, 11.58 [13:56:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.14, 11.84, 11.72 [13:58:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.96, 12.29, 11.91 [14:02:19] PROBLEM - graylog131 Current Load on graylog131 is CRITICAL: LOAD CRITICAL - total load average: 4.07, 3.45, 2.98 [14:04:01] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.15, 6.53, 5.94 [14:04:19] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 2.57, 3.26, 2.98 [14:04:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 10.39, 11.46, 11.70 [14:05:55] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.84, 6.13, 5.85 [14:10:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.13, 11.66, 11.70 [14:12:56] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.22, 6.93, 6.39 [14:14:53] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.41, 6.30, 6.22 [14:28:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.83, 12.00, 11.99 [14:30:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 11.99, 12.05, 12.01 [14:32:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.45, 11.84, 11.94 [14:42:22] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns.ankh.fr.eu.org.', 'ns1.eu.org.', 'ns1.eriomem.net.'], 'CNAME': 'bouncingwiki.miraheze.org.'} [14:44:10] PROBLEM - pyramidgames.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'pyramidgames.wiki' expires in 7 day(s) (Wed 22 Nov 2023 14:28:59 GMT +0000). [14:46:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.37, 11.55, 11.63 [14:48:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.52, 11.55, 11.62 [14:55:10] PROBLEM - www.pyramidgames.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'pyramidgames.wiki' expires in 7 day(s) (Wed 22 Nov 2023 14:28:59 GMT +0000). [15:08:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 12.08, 11.45, 11.39 [15:12:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.90, 11.86, 11.59 [15:16:04] RECOVERY - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.twistwoodtaleswiki.com' will expire on Wed 20 Dec 2023 22:08:34 GMT +0000. [15:32:07] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.20, 6.27, 5.80 [15:32:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 10.77, 7.46, 6.69 [15:34:01] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.41, 5.90, 5.72 [15:34:15] RECOVERY - swiftobject122 Current Load on swiftobject122 is OK: OK - load average: 5.36, 6.57, 6.46 [15:44:45] PROBLEM - graylog131 Current Load on graylog131 is WARNING: LOAD WARNING - total load average: 3.61, 3.22, 2.96 [15:46:41] RECOVERY - graylog131 Current Load on graylog131 is OK: LOAD OK - total load average: 3.36, 3.21, 2.98 [15:47:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 6.10, 6.87, 6.32 [15:49:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 4.31, 5.97, 6.06 [16:23:30] PROBLEM - swiftobject101 Current Load on swiftobject101 is WARNING: WARNING - load average: 7.55, 6.70, 6.18 [16:27:30] RECOVERY - swiftobject101 Current Load on swiftobject101 is OK: OK - load average: 5.44, 6.10, 6.06 [16:34:14] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=waywardrealmswiki --no-updates --username-prefix=wikia:thewaywardrealms /home/macfan/ImportDump/waywardrealmswiki-20231110152337.xml (START) [16:34:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:35:55] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=waywardrealmswiki --no-updates --username-prefix=wikia:thewaywardrealms /home/macfan/ImportDump/waywardrealmswiki-20231110152337.xml (END - exit=0) [16:35:56] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=waywardrealmswiki (START) [16:36:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:36:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:16] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=waywardrealmswiki (END - exit=0) [16:37:17] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=waywardrealmswiki --active --update (END - exit=0) [16:37:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:41:05] PROBLEM - ewgf.wiki - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ewgf.wiki could not be found [16:42:21] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [16:42:22] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=gascwiki --no-updates --username-prefix=w /home/macfan/ImportDump/gascwiki-20231110174822.xml (START) [16:42:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:43:00] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=gascwiki --no-updates --username-prefix=w /home/macfan/ImportDump/gascwiki-20231110174822.xml (END - exit=0) [16:43:01] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=gascwiki (START) [16:43:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:43:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:46:23] PROBLEM - mwtask141 Current Load on mwtask141 is CRITICAL: CRITICAL - load average: 13.36, 11.91, 11.42 [16:46:39] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/rebuildall.php --wiki=gascwiki (END - exit=0) [16:46:39] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=gascwiki --active --update (END - exit=0) [16:46:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:46:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:48:01] PROBLEM - ewgf.wiki - LetsEncrypt on sslhost is CRITICAL: connect to address ewgf.wiki and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:48:04] PROBLEM - www.twistwoodtaleswiki.com - LetsEncrypt on sslhost is CRITICAL: connect to address www.twistwoodtaleswiki.com and port 443: Network is unreachableHTTP CRITICAL - Unable to open TCP socket [16:48:24] PROBLEM - ns1 NTP time on ns1 is CRITICAL: NTP CRITICAL: Offset 0.5570384264 secs [16:48:49] !log [macfan@mwtask141] sudo -u www-data php /srv/mediawiki/w/maintenance/run.php /srv/mediawiki/w/maintenance/importDump.php --wiki=pubertrpwiki --no-updates --username-prefix=wikia:pubertrp /home/macfan/ImportDump/pubertrpwiki-20231110183821.xml (START) [16:48:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:50:15] PROBLEM - swiftobject122 Current Load on swiftobject122 is CRITICAL: CRITICAL - load average: 8.65, 7.25, 6.47 [16:50:23] PROBLEM - mwtask141 Current Load on mwtask141 is WARNING: WARNING - load average: 11.27, 11.71, 11.47 [16:51:56]