[00:01:18] miraheze/ImportDump - Universal-Omega the build has errored. [00:02:12] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.89, 3.75, 3.64 [00:02:47] miraheze/ImportDump - Universal-Omega the build has errored. [00:04:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.07, 2.93, 2.10 [00:04:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.40, 4.92, 3.75 [00:05:40] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.58, 4.25, 3.39 [00:06:01] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [00:08:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.55, 1.62, 1.75 [00:10:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.34, 1.73, 1.76 [00:12:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.94, 1.41, 1.64 [00:13:35] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.09, 3.61, 3.60 [00:16:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.78, 3.65, 3.78 [00:17:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:17:08] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ImportDump/compare/6c6c063f47f2...9960d2cda031 [00:17:10] [02miraheze/ImportDump] 07Universal-Omega 039960d2c - Create ImportDumpRequestQueuePager.php [00:17:11] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [00:17:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.40, 2.99, 3.35 [00:17:52] PROBLEM - es101 Current Load on es101 is CRITICAL: CRITICAL - load average: 4.11, 3.29, 2.41 [00:18:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.47, 3.92, 3.85 [00:19:22] miraheze/ImportDump - Universal-Omega the build has errored. [00:19:51] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 2.63, 3.05, 2.43 [00:20:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.06, 3.14, 3.57 [00:22:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.49, 3.39, 3.59 [00:25:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.12, 3.67, 3.44 [00:26:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.64, 3.87, 3.79 [00:27:25] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.10, 3.53, 3.42 [00:29:24] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.22, 3.79, 3.50 [00:32:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:33:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.03, 3.64, 3.52 [00:34:23] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.42, 10.45, 9.16 [00:34:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.05, 4.14, 3.94 [00:36:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.31, 9.91, 9.11 [00:37:19] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 3.76, 3.58 [00:39:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:43:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.13, 3.95, 3.77 [00:43:55] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/9960d2cda031...92d8bf7fbc6d [00:43:57] [02miraheze/ImportDump] 07Universal-Omega 0392d8bf7 - Update ImportDumpRequestQueuePager.php [00:43:58] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [00:46:09] miraheze/ImportDump - Universal-Omega the build has errored. [00:49:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.06, 3.00, 3.39 [00:49:51] PROBLEM - es101 Current Load on es101 is WARNING: WARNING - load average: 3.61, 3.17, 2.85 [00:51:51] RECOVERY - es101 Current Load on es101 is OK: OK - load average: 3.16, 3.16, 2.89 [00:53:11] PROBLEM - cloud11 Puppet on cloud11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [00:55:05] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.80, 1.59, 1.21 [00:57:03] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.18, 1.45, 1.20 [00:57:10] [02mw-config] 07MacFan4000 commented on pull request 03#4594: [#T9071](https://phabricator.miraheze.org/T9071) - 13https://github.com/miraheze/mw-config/pull/4594#issuecomment-1101891237 [00:59:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:06:09] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/92d8bf7fbc6d...7be00c9b14d2 [01:06:10] [02miraheze/ImportDump] 07Universal-Omega 037be00c9 - Update extension.json [01:06:12] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:06:16] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/7be00c9b14d2...1ef27efcbbe8 [01:06:17] [02miraheze/ImportDump] 07Universal-Omega 031ef27ef - Update SpecialImportDumpRequestQueue.php [01:06:19] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:06:23] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:06:25] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/1ef27efcbbe8...bd922c15e9b0 [01:06:26] [02miraheze/ImportDump] 07Universal-Omega 03bd922c1 - inject services [01:07:13] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:07:15] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/bd922c15e9b0...855db3e504bb [01:07:16] [02miraheze/ImportDump] 07Universal-Omega 03855db3e - Update SpecialRequestImportDump.php [01:08:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.11, 3.22, 3.93 [01:10:12] miraheze/ImportDump - Universal-Omega the build has errored. [01:10:29] [02miraheze/ImportDump] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±2] 13https://github.com/miraheze/ImportDump/compare/855db3e504bb...c48be8ed0632 [01:10:31] [02miraheze/ImportDump] 07github-actions 03c48be8e - CI: lint code to MediaWiki standards [01:10:32] miraheze/ImportDump - Universal-Omega the build has errored. [01:10:32] [02ImportDump] 07github-actions[bot] synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:10:49] miraheze/ImportDump - Universal-Omega the build has errored. [01:11:37] miraheze/ImportDump - Universal-Omega the build has errored. [01:16:29] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/c48be8ed0632...54e6b3f74a74 [01:16:30] [02miraheze/ImportDump] 07Universal-Omega 0354e6b3f - Update ImportDumpRequestQueuePager.php [01:16:32] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:16:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.28, 2.38, 3.30 [01:18:53] miraheze/ImportDump - Universal-Omega the build has errored. [01:19:39] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/54e6b3f74a74...bb1985853ecf [01:19:41] [02miraheze/ImportDump] 07Universal-Omega 03bb19858 - Update SpecialImportDumpRequestQueue.php [01:19:42] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:21:10] RECOVERY - cloud11 Puppet on cloud11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [01:22:22] miraheze/ImportDump - Universal-Omega the build has errored. [01:23:43] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [01:23:44] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/bb1985853ecf...f7b44b833b0a [01:23:46] [02miraheze/ImportDump] 07Universal-Omega 03f7b44b8 - Update mediawiki-tests.yml [01:24:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:26:47] miraheze/ImportDump - Universal-Omega the build has errored. [01:27:27] [02mw-config] 07Universal-Omega closed pull request 03#4599: Add SemanticMediaWiki to ManageWikiExtensions - 13https://github.com/miraheze/mw-config/pull/4599 [01:27:28] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [01:27:30] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [01:27:31] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±5] 13https://github.com/miraheze/mw-config/compare/2ec4e890d700...df7f90536556 [01:27:33] [02miraheze/mw-config] 07Universal-Omega 03df7f905 - Add SemanticMediaWiki to ManageWikiExtensions (#4599) [01:28:37] miraheze/mw-config - Universal-Omega the build passed. [01:29:24] [02mediawiki] 07Universal-Omega closed pull request 03#6303: Bump extensions/PortableInfobox from `2a6077a` to `5f13d93` - 13https://github.com/miraheze/mediawiki/pull/6303 [01:29:25] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PortableInfobox-5f13d93 [01:29:27] [02mediawiki] 07Universal-Omega deleted branch 03dependabot/submodules/REL1_37/extensions/PortableInfobox-5f13d93 - 13https://github.com/miraheze/mediawiki [01:29:28] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/9a57c579b6a9...17a617fedb0a [01:29:30] [02miraheze/mediawiki] 07dependabot[bot] 0317a617f - Bump extensions/PortableInfobox from `2a6077a` to `5f13d93` (#6303) [01:29:30] !log [@test101] starting deploy of {'config': True} to all [01:29:31] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [01:29:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:29:38] [02mw-config] 07Universal-Omega closed pull request 03#4600: Add `$wgPortableInfoboxResponsiblyOpenCollapsed` to ManageWikiSettings.php - 13https://github.com/miraheze/mw-config/pull/4600 [01:29:39] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/df7f90536556...9e9899de8bd3 [01:29:41] [02miraheze/mw-config] 07lens0021 039e9899d - Add `$wgPortableInfoboxResponsiblyOpenCollapsed` to ManageWikiSettings.php (#4600) [01:29:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.16, 3.25, 3.93 [01:29:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:30:40] miraheze/mw-config - Universal-Omega the build passed. [01:34:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:39:32] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:39:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.78, 3.57, 3.64 [01:40:05] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/17a617fedb0a...3f9ebdab8fef [01:40:07] [02miraheze/mediawiki] 07Universal-Omega 033f9ebda - Update ManageWiki [01:40:34] !log [universalomega@test101] starting deploy of {'world': True} to all [01:40:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:40:41] !log [universalomega@mwtask111] starting deploy of {'world': True} to all [01:40:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:41:29] PROBLEM - mwtask111 Puppet on mwtask111 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 18 minutes ago with 0 failures [01:43:37] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.12, 3.42, 3.56 [01:45:36] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.89, 2.85, 3.33 [01:45:58] !log [universalomega@test101] finished deploy of {'world': True} to all - SUCCESS in 324s [01:46:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:48:43] !log [universalomega@mwtask111] finished deploy of {'world': True} to all - SUCCESS in 481s [01:48:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:48:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.24, 3.10, 2.80 [01:49:12] !log [@test101] starting deploy of {'config': True} to all [01:49:13] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [01:49:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:49:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:49:32] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [01:50:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.81, 2.91, 2.77 [02:01:29] RECOVERY - mwtask111 Puppet on mwtask111 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [02:01:41] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [02:01:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:02:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.44, 2.63, 1.81 [02:04:04] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 143s [02:04:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:08:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.95, 2.00, 1.76 [02:11:00] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [02:12:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.15, 1.63, 1.67 [02:13:17] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.93, 3.65, 2.81 [02:15:15] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.80, 3.16, 2.73 [02:26:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.53, 3.16, 2.93 [02:26:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.53, 3.40, 3.19 [02:26:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/9e9899de8bd3...bae1afc5c83e [02:26:31] [02miraheze/mw-config] 07Universal-Omega 03bae1afc - Set smwgUpgradeKey [02:26:40] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [02:26:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:26:53] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 12s [02:26:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:27:33] miraheze/mw-config - Universal-Omega the build passed. [02:28:12] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.37, 2.96, 2.88 [02:28:19] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.05, 3.18, 3.15 [02:29:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/bae1afc5c83e...82fab92ee124 [02:29:20] [02miraheze/mw-config] 07Universal-Omega 0382fab92 - try enableSemantics [02:29:24] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [02:29:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:29:33] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [02:29:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:30:01] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=altversewiki (END - exit=0) [02:30:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:30:18] miraheze/mw-config - Universal-Omega the build passed. [02:30:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/82fab92ee124...facdcf3032e7 [02:30:25] [02miraheze/mw-config] 07Universal-Omega 03facdcf3 - Update SemanticMediaWiki.php [02:30:26] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [02:30:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:30:35] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [02:30:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:31:32] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/facdcf3032e7...3e4d8c8d4a6e [02:31:32] miraheze/mw-config - Universal-Omega the build passed. [02:31:33] [02miraheze/mw-config] 07Universal-Omega 033e4d8c8 - Update SemanticMediaWiki.php [02:31:36] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [02:31:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:31:45] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [02:31:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:32:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.51, 3.13, 2.84 [02:32:32] miraheze/mw-config - Universal-Omega the build passed. [02:32:35] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=altversewiki (END - exit=0) [02:32:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:33:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/3e4d8c8d4a6e...de197c8de995 [02:33:46] [02miraheze/mw-config] 07Universal-Omega 03de197c8 - Update SemanticMediaWiki.php [02:33:51] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [02:33:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:34:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.07, 3.11, 2.86 [02:34:01] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [02:34:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:34:48] miraheze/mw-config - Universal-Omega the build passed. [02:35:41] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=altversewiki --delete (END - exit=0) [02:35:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:36:13] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=altversewiki (END - exit=0) [02:36:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:36:49] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=semanticmediawiki --disable (END - exit=65280) [02:36:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:11] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=altversewiki semanticmediawiki --disable (END - exit=0) [02:37:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:40] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [02:37:40] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [02:38:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/de197c8de995...e831bcf8af36 [02:38:23] [02miraheze/mw-config] 07Universal-Omega 03e831bcf - Update SemanticMediaWiki.php [02:38:29] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [02:38:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:38:39] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [02:38:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:39:00] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 6 seconds ago with 0 failures [02:39:31] miraheze/mw-config - Universal-Omega the build passed. [02:40:41] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e831bcf8af36...0254065d4620 [02:40:43] [02miraheze/mw-config] 07Universal-Omega 030254065 - Update SemanticMediaWiki.php [02:41:05] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/c3ebdd025af3 [02:41:07] [02miraheze/mw-config] 07Universal-Omega 03c3ebdd0 - Move SMW define [02:41:08] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [02:41:10] [02mw-config] 07Universal-Omega opened pull request 03#4602: Move SMW define - 13https://github.com/miraheze/mw-config/pull/4602 [02:41:45] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [02:41:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:41:50] miraheze/mw-config - Universal-Omega the build passed. [02:41:56] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [02:42:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:42:15] miraheze/mw-config - Universal-Omega the build passed. [02:42:23] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.19, 9.88, 8.37 [02:42:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.90, 10.20, 8.77 [02:44:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.07, 9.09, 8.26 [02:44:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.13, 9.17, 8.58 [02:44:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/0d94a205fc75 [02:44:54] [02miraheze/mw-config] 07Universal-Omega 030d94a20 - Manually load SMW [02:44:56] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [02:44:57] [02mw-config] 07Universal-Omega opened pull request 03#4603: Manually load SMW - 13https://github.com/miraheze/mw-config/pull/4603 [02:45:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/0d94a205fc75...75398a82da9e [02:45:55] [02miraheze/mw-config] 07Universal-Omega 0375398a8 - Update LocalSettings.php [02:45:56] [02mw-config] 07Universal-Omega synchronize pull request 03#4603: Manually load SMW - 13https://github.com/miraheze/mw-config/pull/4603 [02:46:13] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/75398a82da9e...a9875969f9af [02:46:14] [02miraheze/mw-config] 07Universal-Omega 03a987596 - Update SemanticMediaWiki.php [02:46:16] [02mw-config] 07Universal-Omega synchronize pull request 03#4603: Manually load SMW - 13https://github.com/miraheze/mw-config/pull/4603 [02:46:30] [02mw-config] 07Universal-Omega closed pull request 03#4603: Manually load SMW - 13https://github.com/miraheze/mw-config/pull/4603 [02:46:31] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [02:46:33] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/c3ebdd025af3...eafe69991957 [02:46:34] [02miraheze/mw-config] 07Universal-Omega 03eafe699 - Manually load SMW (#4603) [02:46:36] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-4 [02:46:37] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [02:46:37] [02mw-config] 07Universal-Omega synchronize pull request 03#4602: Move SMW define - 13https://github.com/miraheze/mw-config/pull/4602 [02:46:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:46:50] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 12s [02:46:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:47:33] miraheze/mw-config - Universal-Omega the build passed. [02:48:10] !log [@test101] starting deploy of {'config': True} to all [02:48:11] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [02:48:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:48:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:49:12] [02miraheze/mw-config] 07Universal-Omega pushed 033 commits to 03Universal-Omega-patch-4 [+0/-0/±3] 13https://github.com/miraheze/mw-config/compare/0d94a205fc75^...a9875969f9af [02:49:13] [02miraheze/mw-config] 07Universal-Omega 030d94a20 - Manually load SMW [02:49:14] [02miraheze/mw-config] 07Universal-Omega 0375398a8 - Update LocalSettings.php [02:49:16] [02miraheze/mw-config] 07Universal-Omega 03a987596 - Update SemanticMediaWiki.php [02:49:18] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [02:49:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4603-Universal-Omega-patch-4 [+0/-0/±2] 13https://github.com/miraheze/mw-config/commit/7a484c6b198b [02:49:21] [02miraheze/mw-config] 07Universal-Omega 037a484c6 - Revert "Manually load SMW (#4603)" [02:49:22] [02mw-config] 07Universal-Omega created branch 03revert-4603-Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [02:49:23] [02mw-config] 07Universal-Omega opened pull request 03#4604: Revert "Manually load SMW (#4603)" - 13https://github.com/miraheze/mw-config/pull/4604 [02:49:29] [02mw-config] 07Universal-Omega closed pull request 03#4604: Revert "Manually load SMW (#4603)" - 13https://github.com/miraheze/mw-config/pull/4604 [02:49:31] [02mw-config] 07Universal-Omega deleted branch 03revert-4603-Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [02:49:32] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4603-Universal-Omega-patch-4 [02:49:34] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/eafe69991957...80a42537fa97 [02:49:35] [02miraheze/mw-config] 07Universal-Omega 0380a4253 - Revert "Manually load SMW (#4603)" (#4604) [02:49:37] [02mw-config] 07Universal-Omega synchronize pull request 03#4602: Move SMW define - 13https://github.com/miraheze/mw-config/pull/4602 [02:49:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4604-revert-4603-Universal-Omega-patch-4 [+0/-0/±2] 13https://github.com/miraheze/mw-config/commit/b4ca2876f5c0 [02:49:40] [02miraheze/mw-config] 07Universal-Omega 03b4ca287 - Revert "Revert "Manually load SMW (#4603)" (#4604)" [02:49:41] [02mw-config] 07Universal-Omega created branch 03revert-4604-revert-4603-Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [02:49:43] [02mw-config] 07Universal-Omega opened pull request 03#4605: Revert "Revert "Manually load SMW (#4603)"" - 13https://github.com/miraheze/mw-config/pull/4605 [02:49:44] [02mw-config] 07Universal-Omega edited pull request 03#4605: Revert "Revert "Manually load SMW (#4603)"" - 13https://github.com/miraheze/mw-config/pull/4605 [02:50:13] [02mw-config] 07Universal-Omega synchronize pull request 03#4605: Revert "Revert "Manually load SMW (#4603)"" - 13https://github.com/miraheze/mw-config/pull/4605 [02:50:15] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4604-revert-4603-Universal-Omega-patch-4 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b4ca2876f5c0...8e95ec579324 [02:50:16] [02miraheze/mw-config] 07Universal-Omega 038e95ec5 - Update Defines.php [02:50:30] [02mw-config] 07Universal-Omega edited pull request 03#4605: Manually load SMW - 13https://github.com/miraheze/mw-config/pull/4605 [02:50:35] miraheze/mw-config - Universal-Omega the build passed. [02:50:36] [02mw-config] 07Universal-Omega edited pull request 03#4605: Manually load SMW - 13https://github.com/miraheze/mw-config/pull/4605 [02:50:42] [02mw-config] 07Universal-Omega closed pull request 03#4605: Manually load SMW - 13https://github.com/miraheze/mw-config/pull/4605 [02:50:44] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4604-revert-4603-Universal-Omega-patch-4 [02:50:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/0254065d4620...44b4545fb441 [02:50:47] [02miraheze/mw-config] 07Universal-Omega 0344b4545 - Manually load SMW (#4605) [02:50:48] [02mw-config] 07Universal-Omega deleted branch 03revert-4604-revert-4603-Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [02:50:57] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [02:51:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:51:06] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [02:51:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:51:18] miraheze/mw-config - Universal-Omega the build passed. [02:51:20] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [02:51:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:51:31] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [02:51:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:51:53] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=cosmoswiki semanticmediawiki --disable (END - exit=0) [02:51:55] miraheze/mw-config - Universal-Omega the build passed. [02:51:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:52:02] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=cosmoswiki semanticmediawiki (END - exit=0) [02:52:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:54:06] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/44b4545fb441...a8615bc1f524 [02:54:08] [02miraheze/mw-config] 07Universal-Omega 03a8615bc - Update SemanticMediaWiki.php [02:54:14] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [02:54:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:54:23] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [02:54:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:55:14] miraheze/mw-config - Universal-Omega the build passed. [02:55:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a8615bc1f524...159cbeeaafed [02:55:39] [02miraheze/mw-config] 07Universal-Omega 03159cbee - Update SemanticMediaWiki.php [02:55:41] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [02:55:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:55:55] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 14s [02:56:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:56:39] miraheze/mw-config - Universal-Omega the build passed. [03:00:04] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=semanticmediawiki (END - exit=65280) [03:00:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:00:24] !log [universalomega@mwtask111] sudo -u www-data php /srv/mediawiki/w/extensions/ManageWiki/maintenance/toggleExtension.php --wiki=altversewiki semanticmediawiki (END - exit=0) [03:00:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:05:31] !log mwscript extensions/SemanticMediaWiki/setupStore.php altversewiki -y --no-log on mw* and mwtask111 [03:05:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:07:20] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [03:09:23] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/608be0a58ae8 [03:09:24] [02miraheze/mw-config] 07Universal-Omega 03608be0a - Don't manually enable [03:09:26] [02mw-config] 07Universal-Omega created branch 03Universal-Omega-patch-5 - 13https://github.com/miraheze/mw-config [03:09:27] [02mw-config] 07Universal-Omega opened pull request 03#4606: Don't manually enable - 13https://github.com/miraheze/mw-config/pull/4606 [03:09:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/608be0a58ae8...ea2735f59364 [03:09:48] [02miraheze/mw-config] 07Universal-Omega 03ea2735f - Update SemanticMediaWiki.php [03:09:50] [02mw-config] 07Universal-Omega synchronize pull request 03#4606: Don't manually enable - 13https://github.com/miraheze/mw-config/pull/4606 [03:10:01] [02mw-config] 07Universal-Omega closed pull request 03#4606: Don't manually enable - 13https://github.com/miraheze/mw-config/pull/4606 [03:10:02] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/159cbeeaafed...951ee9b6ed7a [03:10:04] [02miraheze/mw-config] 07Universal-Omega 03951ee9b - Don't manually enable (#4606) [03:10:05] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 [03:10:07] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-5 - 13https://github.com/miraheze/mw-config [03:10:15] [02mw-config] 07Universal-Omega closed pull request 03#4602: Move SMW define - 13https://github.com/miraheze/mw-config/pull/4602 [03:10:20] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [03:10:21] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://github.com/miraheze/mw-config [03:10:33] miraheze/mw-config - Universal-Omega the build passed. [03:10:56] miraheze/mw-config - Universal-Omega the build passed. [03:11:09] miraheze/mw-config - Universal-Omega the build passed. [03:14:40] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/951ee9b6ed7a...eb2ce5482077 [03:14:40] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 3.63, 3.32, 1.95 [03:14:41] [02miraheze/mw-config] 07Universal-Omega 03eb2ce54 - Add note to SMW [03:15:42] miraheze/mw-config - Universal-Omega the build passed. [03:15:53] !log [universalomega@mwtask111] starting deploy of {'pull': 'config', 'config': True} to all [03:16:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:16:02] !log [universalomega@mwtask111] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [03:16:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:18:57] !log [@test101] starting deploy of {'config': True} to all [03:18:58] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [03:19:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:19:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:22:40] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.59, 1.56, 1.63 [03:50:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.48, 3.19, 3.00 [03:52:42] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.03, 3.14, 3.00 [03:57:29] !log enable SMW related rights in the global system administrator user group, by using cosmoswiki [03:57:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:00:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 3.64, 3.15 [04:00:30] [02mw-config] 07dmehus opened pull request 03#4607: Tweak wording of SMW help page - 13https://github.com/miraheze/mw-config/pull/4607 [04:01:32] miraheze/mw-config - dmehus the build passed. [04:01:59] [02mw-config] 07dmehus synchronize pull request 03#4607: Tweak wording of SMW help page - 13https://github.com/miraheze/mw-config/pull/4607 [04:02:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.28, 3.48, 3.15 [04:02:58] miraheze/mw-config - dmehus the build passed. [04:04:00] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.17, 3.12, 3.06 [04:21:46] [02mw-config] 07Universal-Omega reviewed pull request 03#4607 commit - 13https://github.com/miraheze/mw-config/pull/4607#discussion_r852580916 [04:22:07] dmehus: ^ [04:23:03] The way the script is formatted, it isn't running in typical maintenance script formatting, so doesn't look right to call it like that. [04:26:25] [02mw-config] 07dmehus synchronize pull request 03#4594: [#T9071](https://phabricator.miraheze.org/T9071) - 13https://github.com/miraheze/mw-config/pull/4594 [04:27:28] miraheze/mw-config - dmehus the build passed. [04:34:58] [02mw-config] 07dmehus synchronize pull request 03#4594: [#T9071](https://phabricator.miraheze.org/T9071) - 13https://github.com/miraheze/mw-config/pull/4594 [04:35:55] CosmicAlpha, looking [04:36:03] ok [04:36:04] miraheze/mw-config - dmehus the build passed. [04:36:06] I will fix that [04:36:37] wow how'd you do that [04:36:43] where it says "Commit suggestion"? [04:37:05] [02mw-config] 07dmehus synchronize pull request 03#4607: Tweak wording of SMW help page - 13https://github.com/miraheze/mw-config/pull/4607 [04:38:06] miraheze/mw-config - dmehus the build passed. [04:43:57] dmehus: using suggestion instead of a code language in the same format you'd do code blocks. [04:44:02] Thanks for fixing [04:45:10] CosmicAlpha: np, that's a cool feature :) [04:45:11] [02mw-config] 07Universal-Omega closed pull request 03#4607: Tweak wording of SMW help page - 13https://github.com/miraheze/mw-config/pull/4607 [04:45:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/eb2ce5482077...54959e7268d2 [04:45:14] [02miraheze/mw-config] 07dmehus 0354959e7 - Tweak wording of SMW help page (#4607) [04:46:17] miraheze/mw-config - Universal-Omega the build passed. [04:46:34] dmehus: yeah it is cool, and useful! [04:48:00] !log [@test101] starting deploy of {'config': True} to all [04:48:01] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [04:48:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:48:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:48:20] CosmicAlpha: why every mediawiki server [04:50:16] RhinosF1, yeah, that's a bit strange, I think CosmicAlpha thinks something to do with caching? [04:50:18] RhinosF1: I honestly don't know exactly, it has something to do with cache I think, so that it picks up the smwgUpgradeKey value, and reads it. I didn't have the issue on test101, since it was just that server but on production, it didn't work until I ran the script on all servers. [04:51:32] CosmicAlpha: can we try and find out? We could try and then add support to deploy tool [04:51:50] I don't really want us ever running things on more than one server if we can avoid [04:52:27] !log [@mwtask111] starting deploy of {'config': True} to all [04:52:30] RhinosF1, yeah, less room for error, right? that'd be ideal if that could be added to the deploy tool [04:52:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:52:36] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 8s [04:52:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:52:43] dmehus: and it's quicker [04:52:49] true [04:52:54] The more servers we get it really doesn't scale tbh [04:53:02] yeah... [04:53:06] trying to run scripts on every server at once is a pain [04:53:28] that's partly why the deploy tool exists [04:53:45] RhinosF1: I'll try and find out yes. But it isn't files, so the deploy tool won't work. It's some sort of cache, I think, somehow, but not files I could find. [04:53:50] So there's no need to run the same steps on every server [04:54:03] I don't understand it tbh. [04:54:18] I just know it had to be done on all of them. [04:54:20] CosmicAlpha: right, that makes 0 sense. Tbh, you could make it run the script over ssh via the deploy tool. [04:54:28] It has a shell user on each server [04:54:43] and runs as www-data anyway [04:54:43] Yeah true. [04:55:37] It has me confused though. It isn't files that's refreshed on each server, and our caching is memcached, or database, not local server. So I really don't understand why it had to be done on all of them. [04:56:46] I mean there is obviously an explanation and some sort of caching is all that makes any logical sense to me at all. [04:56:49] But even that doesn't make sense. [04:56:59] No it doesn't [04:57:14] Which tbh it should have before pushing to prod [04:57:32] I don't think the answer to why we do something should be it works [04:57:32] Well how would I have known that? [04:57:37] Ask [04:57:41] Go upstream [04:57:58] Ask what? How am I supposed to anticipate this? [05:00:02] I think it could've been difficult to anticipate because of our configuration. The issue is our test101.miraheze.org is one server, and it seems to be an issue in production where we have four MediaWiki servers. We're fortunate, though, it didn't seem to cause more than a few minutes of downtime and no issues on `altversewiki` and @Centrist16 was very appreciative and understanding as we worked through it [05:00:09] But you've known a few hours and wouldn't have done anything if I hadn't have questioned it CosmicAlpha [05:00:25] Only way to truly replicate would be to have four test servers [05:00:41] dmehus: yeah, more test environments would be ice but sadly prod barely has enough resources [05:00:50] RhinosF1, true :( [05:01:41] Yeah I knew after pushing to prod, not betore. I still don't intend to do anything now that you did say something RhinosF1. I'm not reverting. It works fine, and is already used on a wiki. It's just a few extra steps for us right now until we figure something out. [05:02:10] I'll find out why, which I have intended to do. [05:03:03] We need a solution [05:03:05] We just need to be super cautious. I'll drop Raidarr a note to stress the importance of ensuring an SRE is available and not mobile before enabling [05:03:14] It shouldn't be ran on all servers long term [05:03:24] Neither should we be doing anything just because [05:03:51] RhinosF1: I know that. I never intended to just drop it and leave it like that. I'll try and find something. [05:04:04] Hopefully we won't be enabling it on too many wikis? I wouldn't over-promote it. It should be a passive enable; only if users ask [05:07:19] Hopefully not to many. It's one of the reasons I set the Cargo and PageSchemas conflicts, to limit some other intensive database extensions being used alongside. Wikibase actually has a technical conflict with namespaces. [05:07:56] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [05:07:56] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [05:13:44] CosmicAlpha: yeah, that's good [05:37:35] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [05:58:47] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.60, 2.91, 2.45 [06:00:46] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.17, 3.40, 2.67 [06:02:45] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.13, 3.05, 2.64 [06:07:15] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [06:19:49] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.11, 1.73, 1.15 [06:21:49] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.64, 1.71, 1.22 [06:23:49] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.34, 1.57, 1.22 [06:28:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.47, 3.49, 2.62 [06:30:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.03, 2.84, 2.48 [06:48:39] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.05, 1.70, 1.50 [06:50:33] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.90, 1.81, 1.57 [06:54:23] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.49, 1.67, 1.56 [07:03:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:05:01] [02mw-config] 07dmehus commented on pull request 03#4601: Re-enable CreateRedirect; Disable EditNotify - 13https://github.com/miraheze/mw-config/pull/4601#issuecomment-1102168312 [07:08:31] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [07:08:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [07:13:49] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.89, 1.72, 1.60 [07:17:49] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.55, 1.99, 1.73 [07:21:49] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.81, 1.99, 1.80 [07:27:49] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.41, 1.58, 1.68 [07:38:10] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [07:44:52] PROBLEM - vedopedia.witches-empire.com - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File [07:44:52] ib/python3/dist-packages/dns/resolver.py", line 1043, in resolve timeout = self._compute_timeout(start, lifetime) File "/usr/lib/python3/dist-packages/dns/resolver.py", line 950, in _compute_timeout raise Timeout(timeout=duration)dns.exception.Timeout: The DNS operation timed out after 5.40727162361145 seconds [08:09:35] PROBLEM - cp20 HTTPS on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:09:55] PROBLEM - ping6 on cp20 is CRITICAL: PING CRITICAL - Packet loss = 100% [08:09:58] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb [08:10:09] PROBLEM - cp20 SSH on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [08:10:19] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 2 datacenters are down: 51.195.220.68/cpweb, 2001:41d0:801:2000::4c25/cpweb [08:11:25] PROBLEM - cp20 NTP time on cp20 is WARNING: NTP WARNING: Offset 0.1042574942 secs [08:11:33] RECOVERY - cp20 HTTPS on cp20 is OK: HTTP OK: HTTP/1.1 301 Moved Permanently - 3384 bytes in 0.017 second response time [08:11:52] PROBLEM - cp20 Varnish Backends on cp20 is CRITICAL: 7 backends are down. mw101 mw102 mw111 mw112 mw121 mw122 mediawiki [08:11:58] RECOVERY - ping6 on cp20 is OK: PING OK - Packet loss = 0%, RTA = 2.74 ms [08:12:05] RECOVERY - cp20 SSH on cp20 is OK: SSH OK - OpenSSH_8.4p1 Debian-5 (protocol 2.0) [08:13:58] RECOVERY - vedopedia.witches-empire.com - reverse DNS on sslhost is OK: SSL OK - vedopedia.witches-empire.com reverse DNS resolves to cp21.miraheze.org - CNAME OK [08:13:58] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [08:14:19] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [08:15:46] RECOVERY - cp20 Varnish Backends on cp20 is OK: All 12 backends are healthy [08:17:26] RECOVERY - cp20 NTP time on cp20 is OK: NTP OK: Offset 0.09977573156 secs [08:30:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.38, 3.42, 2.94 [08:33:59] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.15, 3.38, 3.05 [08:37:08] PROBLEM - vedopedia.witches-empire.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:06:29] RECOVERY - vedopedia.witches-empire.com - LetsEncrypt on sslhost is OK: OK - Certificate 'vedopedia.witches-empire.com' will expire on Sat 11 Jun 2022 05:15:21 GMT +0000. [09:08:47] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.26, 3.79, 3.14 [09:10:46] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.89, 3.04, 2.94 [09:11:01] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [09:11:01] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:18:43] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.05, 3.66, 3.20 [09:19:54] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.27, 3.36, 2.77 [09:20:42] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.93, 3.69, 3.29 [09:21:53] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.04, 3.27, 2.81 [09:22:41] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.29, 3.35, 3.22 [09:26:30] [02mw-config] 07lens0021 opened pull request 03#4608: Add EchoMaxMentionsInEditSummary to ManagaWiki - 13https://github.com/miraheze/mw-config/pull/4608 [09:27:19] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.80, 3.84, 3.18 [09:27:36] miraheze/mw-config - lens0021 the build passed. [09:29:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.54, 3.00, 2.95 [09:40:33] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.34, 4.46, 3.65 [09:40:41] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [09:40:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.63, 3.66, 3.26 [09:42:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.45, 4.03, 3.44 [09:44:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.36, 3.38, 3.27 [09:54:28] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.95, 3.90 [09:56:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.60, 4.09, 3.97 [09:59:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.11, 3.68, 3.42 [10:00:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.37, 3.57, 3.82 [10:01:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.62, 3.32, 3.32 [10:06:23] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.95, 2.43, 3.21 [10:10:27] Reception123: you around? [10:10:59] or UO? CosmicAlpha [10:27:12] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.99, 1.74, 1.34 [10:29:12] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.84, 1.38, 1.25 [10:41:56] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 86, in check_records cname = str(dns_resolver.resolve(hostname, 'CNAME')[0]) File "/usr/li [10:41:56] 3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query dnd.bellinrattin.it. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:11:36] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [11:26:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:31:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:41:00] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [11:41:15] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [11:41:16] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [11:49:40] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.75, 3.53, 2.71 [11:53:39] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.22, 3.21, 2.78 [11:54:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [11:59:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:01:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.48, 3.52, 2.91 [12:03:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.12, 2.92, 2.77 [12:05:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:09:00] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [12:12:32] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.48, 3.14, 2.79 [12:12:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.19, 3.19, 2.57 [12:14:32] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.40, 3.12, 2.82 [12:14:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.56, 2.87, 2.53 [12:19:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.47, 9.90, 9.04 [12:21:16] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.56, 9.66, 9.08 [12:25:37] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.22, 1.71, 1.31 [12:27:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.92, 4.32, 3.44 [12:27:45] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.67, 3.68, 2.92 [12:28:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.28, 3.70, 3.03 [12:29:36] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.04, 1.80, 1.48 [12:29:44] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.31, 3.20, 2.83 [12:30:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.98, 3.13, 2.91 [12:31:24] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.00, 3.37, 3.27 [12:31:36] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.78, 1.52, 1.42 [12:35:22] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.83, 3.71, 3.44 [12:37:21] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.62, 3.96, 3.55 [12:37:52] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.12, 3.56, 3.21 [12:38:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.73, 3.50, 3.08 [12:39:21] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.62, 3.47 [12:39:48] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.17, 3.37, 3.18 [12:40:01] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:40:35] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.72, 3.27, 3.05 [12:41:20] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.16, 3.35, 3.40 [12:42:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.08, 10.22, 8.83 [12:44:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.66, 9.23, 8.63 [12:45:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.19, 3.56, 3.49 [12:48:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [12:49:16] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.66, 3.55, 3.47 [12:51:09] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/dfa6a5208dd3 [12:51:10] [02miraheze/mw-config] 07Reception123 03dfa6a52 - rcenhancedfilters & usenewrc settings for squareenixwiki and metroid (T9100) [12:51:12] [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config [12:51:13] [02mw-config] 07Reception123 opened pull request 03#4609: rcenhancedfilters & usenewrc settings for squareenixwiki and metroid … - 13https://github.com/miraheze/mw-config/pull/4609 [12:51:16] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.64, 3.60, 3.50 [12:51:54] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.63, 10.66, 9.88 [12:52:24] miraheze/mw-config - Reception123 the build passed. [12:53:15] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.39, 3.23, 3.36 [12:54:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.13, 10.40, 9.27 [12:57:51] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.67, 10.03, 9.93 [12:58:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.52, 10.36, 9.59 [12:59:15] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.88, 9.89, 9.23 [13:00:17] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.98, 3.71, 3.27 [13:00:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.37, 9.53, 9.40 [13:01:14] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.95, 9.39, 9.15 [13:02:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.30, 10.03, 9.08 [13:04:10] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.16, 3.58, 3.29 [13:04:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 10.07, 10.12, 9.23 [13:06:06] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.51, 3.09, 3.14 [13:08:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:10:16] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [13:18:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [13:28:00] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.19, 4.14, 3.56 [13:28:49] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.22, 9.47, 8.92 [13:28:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.25, 4.56, 3.50 [13:29:07] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.30, 4.16, 3.34 [13:30:18] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.21, 1.77, 1.43 [13:30:44] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.70, 9.33, 8.92 [13:32:17] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.19, 1.52, 1.38 [13:34:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.78, 3.47, 3.45 [13:35:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.75, 3.53, 3.37 [13:35:54] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.08, 10.46, 9.79 [13:36:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.41, 3.24, 3.37 [13:37:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.32, 10.64, 9.76 [13:37:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.35, 3.61, 3.78 [13:38:41] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.39, 10.63, 9.94 [13:39:01] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.15, 3.10, 3.26 [13:39:47] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.83, 9.99, 9.78 [13:42:33] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.80, 10.04, 9.85 [13:43:54] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.36, 3.68, 3.75 [13:45:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.88, 10.69, 10.04 [13:46:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.46, 10.15, 9.47 [13:47:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.62, 10.16, 9.41 [13:47:53] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.25, 3.47, 3.65 [13:48:15] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.74, 9.79, 9.26 [13:49:28] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.08, 11.31, 10.44 [13:49:38] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.44, 11.35, 10.54 [13:51:10] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.98, 9.90, 9.54 [13:51:25] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.98, 10.59, 10.26 [13:51:37] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.06, 10.31, 10.28 [13:52:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.94, 9.53, 9.51 [13:53:36] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.14, 10.06, 10.20 [13:53:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.40, 3.01, 3.39 [13:55:05] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.45, 10.43, 9.85 [13:57:04] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.85, 10.10, 9.82 [13:59:12] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.53, 9.79, 10.07 [14:00:08] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.02, 10.06, 9.84 [14:00:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 1.55, 3.15, 2.05 [14:01:07] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.83, 1.81, 1.59 [14:02:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.78, 3.58, 3.33 [14:03:06] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.80, 1.40, 1.46 [14:04:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.16, 10.47, 10.04 [14:04:49] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 0.82, 1.81, 1.74 [14:04:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.59, 3.24, 3.24 [14:04:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.05, 10.17, 10.08 [14:06:55] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.16, 9.84, 9.97 [14:08:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.67, 1.40, 1.60 [14:09:36] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [14:10:00] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 10.09, 10.12, 10.03 [14:10:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.65, 10.36, 9.73 [14:12:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.90, 11.36, 10.19 [14:13:00] PROBLEM - cloud10 Puppet on cloud10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [14:14:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.81, 10.64, 10.06 [14:14:39] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.06, 10.57, 10.24 [14:16:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.24, 10.12, 9.94 [14:16:36] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.23, 10.11, 10.11 [14:19:01] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.26, 1.82, 1.53 [14:19:51] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.75, 11.34, 10.50 [14:21:01] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.28, 1.53, 1.45 [14:21:50] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.69, 11.11, 10.50 [14:33:43] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.87, 9.48, 10.04 [14:34:53] [02mw-config] 07MacFan4000 commented on pull request 03#4594: [#T9071](https://phabricator.miraheze.org/T9071) - 13https://github.com/miraheze/mw-config/pull/4594#issuecomment-1102732901 [14:37:26] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.43, 9.88, 9.44 [14:39:25] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.82, 9.15, 9.23 [14:41:00] RECOVERY - cloud10 Puppet on cloud10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:41:11] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [14:50:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.46, 10.19, 9.81 [14:50:30] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.23, 11.62, 10.50 [14:51:31] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.67, 3.50, 3.20 [14:52:28] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/b3960c6dd545 [14:52:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.82, 11.09, 10.44 [14:52:30] [02miraheze/puppet] 07paladox 03b3960c6 - varnish: set proxy_buffering to off in nginx config [14:52:31] [02puppet] 07paladox created branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [14:52:33] [02puppet] 07paladox opened pull request 03#2510: varnish: set proxy_buffering to off in nginx config - 13https://github.com/miraheze/puppet/pull/2510 [14:53:30] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.04, 3.83, 3.36 [14:53:59] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.42, 9.78, 9.78 [14:54:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.30, 11.65, 10.73 [14:55:14] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.24, 9.83, 9.37 [14:55:30] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.66, 3.35 [14:57:12] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 6.99, 8.95, 9.11 [14:58:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.12, 10.60, 10.55 [14:59:28] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.05, 3.84, 3.45 [15:00:21] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.83, 3.67, 3.16 [15:00:51] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.81, 10.47, 9.84 [15:01:27] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.28, 3.76, 3.49 [15:02:20] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.03, 3.36, 3.11 [15:03:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.52, 4.16, 3.67 [15:04:43] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.92, 9.59, 9.66 [15:04:54] ugochimobi: please see https://phabricator.miraheze.org/T8917 [15:04:55] [url] ⚓ T8917 Missing thumbnails for SVG on commonswiki | phabricator.miraheze.org [15:05:04] And try uploading on a WMF wiki (or non Miraheze) [15:05:10] And if that works then it's not a Miraheze problem [15:05:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.51, 10.54, 10.20 [15:07:03] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.38, 10.35, 9.58 [15:07:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.87, 3.84, 3.40 [15:08:15] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.81, 3.59, 3.24 [15:09:01] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.23, 9.97, 9.54 [15:09:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.22, 10.01, 9.47 [15:10:14] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.11, 3.64, 3.29 [15:10:46] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.74, 10.30, 9.91 [15:10:51] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [15:11:29] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.44, 4.36, 3.65 [15:12:13] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.27, 3.54, 3.30 [15:12:45] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.52, 10.03, 9.86 [15:13:21] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.85, 11.31, 10.61 [15:13:47] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.40, 9.37, 9.41 [15:14:12] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.96, 3.27, 3.23 [15:15:18] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.79, 10.97, 10.56 [15:15:22] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.69, 3.58, 3.50 [15:16:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.97, 9.70, 10.15 [15:16:41] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.13, 10.53, 10.10 [15:18:40] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.30, 11.19, 10.40 [15:19:11] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.89, 11.55, 10.87 [15:19:15] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.20, 2.99, 3.29 [15:20:26] Reception123: ah, okay, let me see then [15:20:34] let me use Wikimedia Commons [15:20:39] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.78, 11.42, 10.58 [15:20:39] Thanks! [15:20:49] Let me know the result [15:23:05] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.90, 11.52, 11.00 [15:23:09] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.49, 10.22, 9.79 [15:23:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.48, 3.53, 3.95 [15:24:36] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.50, 11.70, 10.87 [15:24:47] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.60, 10.22, 9.56 [15:25:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.94, 4.06 [15:26:03] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.87, 3.70, 3.36 [15:26:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.28, 11.47, 10.89 [15:26:40] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.29, 1.74, 1.36 [15:26:56] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 2.79, 1.65, 1.07 [15:27:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.70, 3.75, 3.97 [15:27:35] Reception123: it didn't work [15:27:44] I was suspecting the files anyway [15:27:45] https://commons.wikimedia.org/wiki/File:GDechoIcon.svg [15:27:45] [url] File:GDechoIcon.svg - Wikimedia Commons | commons.wikimedia.org [15:28:02] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.20, 3.43, 3.29 [15:28:33] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.81, 11.69, 11.02 [15:28:40] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.46, 1.62, 1.36 [15:28:56] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.81, 1.38, 1.05 [15:30:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.26, 11.30, 10.95 [15:30:42] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.91, 9.57, 9.54 [15:31:59] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.42, 3.04, 3.16 [15:32:49] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.00, 12.06, 10.75 [15:33:16] [02puppet] 07paladox closed pull request 03#2510: varnish: set proxy_buffering to off in nginx config - 13https://github.com/miraheze/puppet/pull/2510 [15:33:18] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9 [15:33:19] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/17002323079e...450e477c4ea5 [15:33:21] [02miraheze/puppet] 07paladox 03450e477 - varnish: set proxy_buffering to off in nginx config (#2510) [15:33:22] [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [15:34:45] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.66, 11.05, 10.56 [15:35:15] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.47, 3.98, 3.94 [15:36:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 7.04, 4.46, 3.67 [15:37:15] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.58, 3.76, 3.86 [15:38:25] ugochimobi: ok, in that case it is indeed the files so there's nothing Miraheze can do about that [15:38:39] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 5.98, 8.69, 9.93 [15:38:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.94, 3.75, 3.51 [15:39:46] Reception123: yeah, there's nothing we can do, I'd upload the png instead [15:40:33] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.85, 11.75, 10.97 [15:42:25] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.67, 11.08, 10.74 [15:42:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.08, 10.92, 10.75 [15:42:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.27, 11.06, 10.17 [15:42:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 3.22, 3.19, 3.31 [15:43:12] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.01, 3.51, 3.66 [15:43:16] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.72, 10.50, 10.10 [15:44:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.96, 10.13, 9.92 [15:45:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.77, 3.67, 3.70 [15:45:15] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.35, 9.47, 9.76 [15:46:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.04, 11.08, 10.87 [15:48:18] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.13, 11.08, 10.85 [15:49:10] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.43, 2.85, 3.34 [15:50:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.74, 10.50, 10.69 [15:50:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.41, 10.56, 10.10 [15:51:10] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.39, 10.99, 10.26 [15:52:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.09, 11.28, 10.40 [15:54:23] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.20, 11.32, 10.92 [15:55:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.56, 3.50 [15:55:07] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.22, 11.08, 10.52 [15:55:15] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.22, 10.53, 9.60 [15:56:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.79, 11.20, 10.92 [15:56:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.28, 11.36, 10.67 [15:59:06] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.09, 3.90, 3.66 [16:00:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.50, 11.52, 11.22 [16:01:05] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.91, 3.92, 3.70 [16:02:55] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.99, 9.53, 9.63 [16:03:04] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.08, 3.87, 3.70 [16:04:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.35, 11.72, 11.36 [16:04:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.44, 4.28, 3.49 [16:06:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.32, 9.60, 10.10 [16:06:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.40, 3.96, 3.27 [16:08:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.81, 3.97, 3.53 [16:08:59] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 10.00, 9.95, 10.20 [16:10:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.35, 10.37, 10.32 [16:11:29] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/f7b44b833b0a...721fc9261dcd [16:11:31] [02miraheze/ImportDump] 07Universal-Omega 03721fc92 - Update en.json [16:11:32] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:11:41] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/721fc9261dcd...1a225b1b0fff [16:11:43] [02miraheze/ImportDump] 07Universal-Omega 031a225b1 - Update qqq.json [16:11:44] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:12:30] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.30, 10.34, 9.91 [16:12:32] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.74, 3.22, 3.17 [16:12:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.62, 10.51, 10.40 [16:13:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.25, 3.54, 3.80 [16:14:23] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.49, 11.27, 10.89 [16:14:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.24, 9.79, 10.13 [16:14:43] miraheze/ImportDump - Universal-Omega the build has errored. [16:14:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.98, 2.96, 3.25 [16:16:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.54, 11.16, 10.91 [16:16:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.84, 10.04, 9.93 [16:16:53] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.26, 9.93, 10.17 [16:18:26] Reception123: can the script be ran now for gratisdatawiki? [16:18:38] the one for populating wikibase sites table.... [16:19:28] I'm mobile only but I can probably do so this evening [16:19:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.94, 10.93, 10.22 [16:20:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.93, 10.12, 10.21 [16:20:49] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.86, 10.48, 10.36 [16:22:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.18, 10.95, 10.50 [16:23:38] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.08, 9.82, 9.94 [16:24:00] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/1a225b1b0fff...551d5cd3d449 [16:24:02] [02miraheze/ImportDump] 07Universal-Omega 03551d5cd - Update ImportDumpRequestQueuePager.php [16:24:03] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:24:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.60, 9.86, 10.17 [16:24:46] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 7.43, 9.58, 10.07 [16:24:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.48, 3.81, 3.75 [16:26:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.73, 4.25, 3.48 [16:26:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.55, 3.68, 3.71 [16:27:21] miraheze/ImportDump - Universal-Omega the build has errored. [16:27:41] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.64, 4.28, 3.68 [16:28:03] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.28, 10.91, 10.31 [16:29:08] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/551d5cd3d449...a25de3de4f1a [16:29:09] [02miraheze/ImportDump] 07Universal-Omega 03a25de3d - Update SpecialImportDumpRequestQueue.php [16:29:11] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:29:37] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.82, 3.60, 3.49 [16:29:58] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.22, 10.49, 10.22 [16:30:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.65, 11.70, 10.93 [16:30:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.62, 3.61, 3.42 [16:30:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.70, 10.28, 10.10 [16:30:41] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 12.00, 10.55, 10.24 [16:31:34] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.49, 3.27, 3.38 [16:31:53] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.23, 9.85, 10.03 [16:32:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.36, 11.84, 11.10 [16:32:23] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.22, 10.95, 10.64 [16:32:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.73, 3.05, 3.24 [16:32:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.90, 9.73, 9.92 [16:33:24] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/a25de3de4f1a...395ffad95515 [16:33:25] [02miraheze/ImportDump] 07Universal-Omega 03395ffad - Update SpecialRequestImportDump.php [16:33:27] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:34:01] miraheze/ImportDump - Universal-Omega the build has errored. [16:34:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.65, 10.74, 10.60 [16:34:53] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.24, 2.93, 3.39 [16:36:17] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.70, 10.90, 10.41 [16:36:38] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.72, 9.41, 9.88 [16:36:55] miraheze/ImportDump - Universal-Omega the build has errored. [16:38:50] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.61, 3.64, 3.57 [16:40:21] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.29, 3.58, 3.45 [16:42:18] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.62, 3.20, 3.32 [16:44:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.57, 9.50, 10.17 [16:46:13] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.70, 3.53 [16:47:26] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.24, 4.17, 3.68 [16:48:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.89, 10.32, 10.37 [16:49:54] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.79, 9.89, 10.20 [16:50:06] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.78, 4.40, 3.83 [16:50:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.94, 9.11, 9.89 [16:50:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.64, 9.48, 10.07 [16:51:23] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.92, 3.99, 3.71 [16:55:38] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [16:55:39] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/395ffad95515...26148b99710a [16:55:41] [02miraheze/ImportDump] 07Universal-Omega 0326148b9 - Update SpecialRequestImportDump.php [16:55:55] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.39, 3.88, 3.78 [16:56:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.20, 10.85, 10.49 [16:58:43] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.33, 3.47, 4.00 [16:59:19] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 1.95, 2.79, 3.28 [16:59:49] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 1.67, 2.86, 3.39 [17:00:37] miraheze/ImportDump - Universal-Omega the build passed. [17:00:42] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 3.57, 3.69, 4.02 [17:02:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.60, 10.52, 10.17 [17:02:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.33, 9.77, 10.14 [17:02:41] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.72, 3.31, 3.84 [17:04:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.68, 9.63, 9.90 [17:06:10] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.90, 10.85, 9.69 [17:08:08] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.84, 10.43, 9.68 [17:10:38] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.39, 3.47, 3.58 [17:11:30] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.64, 3.21, 3.12 [17:12:05] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.98, 10.01, 9.70 [17:12:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.28, 10.36, 10.10 [17:12:38] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.59, 3.02, 3.40 [17:13:22] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [17:13:22] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [17:13:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.35, 2.89, 3.01 [17:14:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 7.24, 9.11, 9.67 [17:15:05] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.22, 10.38, 9.95 [17:16:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.70, 4.21, 3.82 [17:17:02] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.91, 10.82, 10.16 [17:18:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:18:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.82, 3.94, 3.76 [17:18:58] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.06, 10.16, 10.00 [17:19:15] PROBLEM - cp31 Current Load on cp31 is CRITICAL: CRITICAL - load average: 2.40, 1.71, 1.15 [17:19:59] ugochimobi: I did it already yesterday. It didn't fix your issue. [17:20:08] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.48, 10.49, 9.51 [17:20:11] Reception123: ^ [17:21:12] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 1.44, 1.51, 1.14 [17:22:07] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.80, 10.07, 9.47 [17:23:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:28:31] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.90, 2.82, 3.32 [17:32:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [17:35:26] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.85, 10.95, 10.04 [17:36:27] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.13, 3.44, 3.32 [17:37:22] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.97, 11.29, 10.28 [17:37:40] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [17:37:42] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/26148b99710a...ead9209056f2 [17:37:43] [02miraheze/ImportDump] 07Universal-Omega 03ead9209 - Update SpecialRequestImportDump.php [17:38:26] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.89, 3.52, 3.36 [17:40:26] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 3.26, 3.32, 3.30 [17:40:44] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [17:40:45] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/ead9209056f2...6741bc9b6863 [17:40:47] [02miraheze/ImportDump] 07Universal-Omega 036741bc9 - Update importdump_request_comments.sql [17:40:50] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.62, 10.56, 9.82 [17:41:14] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [17:41:16] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/6741bc9b6863...02e20e9e81e7 [17:41:17] [02miraheze/ImportDump] 07Universal-Omega 0302e20e9 - Update importdump_requests.sql [17:41:33] miraheze/ImportDump - Universal-Omega the build has errored. [17:42:45] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.06, 10.84, 10.02 [17:43:02] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [17:44:36] miraheze/ImportDump - Universal-Omega the build has errored. [17:45:07] CosmicAlpha: 😢this is getting really tiring [17:45:15] How do we do this now? [17:47:22] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.04, 3.93, 3.54 [17:48:29] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.93, 10.18, 10.02 [17:49:03] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.77, 9.98, 10.14 [17:49:12] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.34, 10.10, 9.54 [17:51:08] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 8.03, 9.54, 9.42 [17:51:20] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.03, 3.46, 3.45 [17:51:27] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.93, 10.55, 9.72 [17:53:19] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 6.02, 4.09, 3.66 [17:53:26] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.25, 10.17, 9.69 [17:54:50] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.38, 10.20, 10.23 [17:55:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.93, 3.93, 3.65 [17:56:46] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 10.00, 9.96, 10.12 [17:59:09] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [17:59:11] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/02e20e9e81e7...5d30befc6553 [17:59:12] [02miraheze/ImportDump] 07Universal-Omega 035d30bef - Update ImportDumpRequestQueuePager.php [17:59:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.96, 4.06, 3.73 [17:59:55] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/5d30befc6553...6afb844cdb67 [17:59:56] [02miraheze/ImportDump] 07Universal-Omega 036afb844 - Update extension.json [17:59:58] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [18:00:23] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/6afb844cdb67...d60d0090f526 [18:00:24] [02miraheze/ImportDump] 07Universal-Omega 03d60d009 - Update SpecialRequestImportDump.php [18:00:26] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [18:00:38] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.79, 10.40, 10.28 [18:01:17] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.50, 3.66, 3.61 [18:02:18] [02miraheze/ImportDump] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/d60d0090f526...a0656411a275 [18:02:20] [02miraheze/ImportDump] 07github-actions 03a065641 - CI: lint code to MediaWiki standards [18:02:21] [02ImportDump] 07github-actions[bot] synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [18:03:06] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/a0656411a275...b74f2a364f92 [18:03:07] [02miraheze/ImportDump] 07Universal-Omega 03b74f2a3 - Update SpecialImportDumpRequestQueue.php [18:03:09] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [18:03:10] miraheze/ImportDump - Universal-Omega the build has errored. [18:03:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.32, 10.47, 10.15 [18:04:14] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 14.00, 11.24, 10.24 [18:04:33] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.02, 10.02, 10.19 [18:05:49] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.02, 10.09, 10.06 [18:06:13] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.65, 10.75, 10.18 [18:07:01] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.49, 10.22, 9.83 [18:07:06] miraheze/ImportDump - Universal-Omega the build passed. [18:07:15] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.45, 3.01, 3.35 [18:07:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.96, 10.49, 9.86 [18:08:11] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.58, 11.41, 10.49 [18:09:38] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.50, 11.10, 10.49 [18:10:09] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.53, 10.93, 10.42 [18:10:59] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.66, 11.29, 10.35 [18:11:12] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.78, 3.42, 3.42 [18:11:30] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.70, 9.97, 9.83 [18:12:58] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.16, 11.31, 10.48 [18:13:12] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.02, 3.01, 3.28 [18:13:28] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.10, 9.55, 9.98 [18:14:40] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [18:15:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 8.93, 10.38, 10.09 [18:16:23] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.11, 11.12, 10.19 [18:17:09] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.24, 4.58, 3.88 [18:17:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/4b25a418a2b1 [18:17:19] [02miraheze/puppet] 07paladox 034b25a41 - varnish: do gziping in varnish rather then varnish [18:17:21] [02puppet] 07paladox created branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [18:17:22] [02puppet] 07paladox opened pull request 03#2511: varnish: do gziping in varnish rather then varnish - 13https://github.com/miraheze/puppet/pull/2511 [18:18:03] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.56, 9.23, 9.90 [18:19:49] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.32, 1.79, 1.17 [18:20:15] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.64, 10.15, 10.07 [18:20:34] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.42, 3.96, 3.28 [18:20:53] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.01, 9.64, 10.09 [18:21:40] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4b25a418a2b1...5186d3bd91bf [18:21:42] [02miraheze/puppet] 07paladox 035186d3b - Update default.vcl [18:21:43] [02puppet] 07paladox synchronize pull request 03#2511: varnish: do gziping in varnish rather then varnish - 13https://github.com/miraheze/puppet/pull/2511 [18:21:49] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.54, 1.69, 1.21 [18:21:52] [02puppet] 07paladox edited pull request 03#2511: varnish: do gziping in varnish rather than varnish - 13https://github.com/miraheze/puppet/pull/2511 [18:22:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.42, 3.57, 3.20 [18:22:31] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.57, 3.22 [18:22:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.56, 1.97, 1.32 [18:23:24] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.70, 11.50, 10.68 [18:24:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.81, 11.63, 10.70 [18:24:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.54, 3.18, 3.11 [18:24:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.36, 3.09, 3.06 [18:24:45] PROBLEM - cp31 Current Load on cp31 is WARNING: WARNING - load average: 1.04, 1.72, 1.33 [18:24:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.49, 1.22 [18:25:07] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.86, 3.81, 3.91 [18:25:23] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.94, 11.18, 10.64 [18:25:48] PROBLEM - puppet111 Current Load on puppet111 is WARNING: WARNING - load average: 5.79, 4.18, 2.86 [18:25:55] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.14, 10.56, 10.14 [18:26:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.21, 11.15, 10.65 [18:26:43] RECOVERY - cp31 Current Load on cp31 is OK: OK - load average: 0.64, 1.41, 1.27 [18:26:48] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.85, 10.84, 10.43 [18:27:48] RECOVERY - puppet111 Current Load on puppet111 is OK: OK - load average: 2.60, 3.46, 2.75 [18:27:53] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.84, 10.45, 10.14 [18:28:47] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 13.13, 11.61, 10.76 [18:28:47] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.09, 10.34, 9.84 [18:29:20] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.02, 11.55, 10.91 [18:30:01] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.27, 9.97, 9.87 [18:30:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.59, 11.83, 11.03 [18:30:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.31, 9.72, 9.68 [18:30:45] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.41, 10.74, 10.55 [18:31:04] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.31, 3.74, 3.79 [18:31:19] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.47, 10.72, 10.69 [18:31:50] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.85, 9.60, 9.87 [18:31:57] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.45, 9.28, 9.62 [18:33:04] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.75, 3.76, 3.79 [18:34:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.92, 11.09, 10.95 [18:35:12] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 4.07, 6.33, 3.50 [18:39:01] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 7.74, 4.37, 3.92 [18:39:14] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.38, 9.47, 10.15 [18:41:01] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.47, 3.45, 3.63 [18:42:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.45, 11.05, 10.88 [18:42:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.92, 10.51, 9.87 [18:42:34] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.48, 10.62, 10.08 [18:42:38] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.48, 11.13, 10.67 [18:43:10] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.24, 10.48, 10.42 [18:44:20] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [18:44:59] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.45, 3.70, 3.66 [18:46:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.29, 9.60, 9.68 [18:46:35] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.99, 11.84, 11.09 [18:46:58] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.03, 3.46, 3.58 [18:48:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.51, 11.74, 11.27 [18:48:34] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.34, 12.02, 11.24 [18:48:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.01, 10.76, 10.35 [18:50:28] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.66, 2.09, 1.69 [18:50:32] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.86, 11.62, 11.20 [18:50:33] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.96, 10.14, 10.16 [18:51:12] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 0.52, 1.23, 1.99 [18:52:23] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.61, 1.88, 1.66 [18:52:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.34, 11.42, 10.67 [18:52:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.85, 3.12, 3.40 [18:53:12] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.07, 1.56, 2.02 [18:54:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.49, 11.23, 10.68 [18:56:12] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.36, 1.65, 1.61 [18:56:54] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.65, 3.85, 3.64 [19:00:59] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.01, 10.96, 10.89 [19:01:29] PROBLEM - heavyironmodding.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'heavyironmodding.org' expires in 15 day(s) (Thu 05 May 2022 18:39:43 GMT +0000). [19:02:28] [02puppet] 07paladox synchronize pull request 03#2511: varnish: do gziping in varnish rather than varnish - 13https://github.com/miraheze/puppet/pull/2511 [19:02:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5186d3bd91bf...a726d55f00c6 [19:02:31] [02miraheze/puppet] 07paladox 03a726d55 - Update default.vcl [19:02:44] [02puppet] 07paladox synchronize pull request 03#2511: varnish: do gziping in varnish rather than varnish - 13https://github.com/miraheze/puppet/pull/2511 [19:02:46] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a726d55f00c6...0cdb06bc1e3b [19:02:48] [02miraheze/puppet] 07paladox 030cdb06b - Update default.vcl [19:02:58] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.53, 11.10, 10.95 [19:03:54] [02puppet] 07paladox closed pull request 03#2511: varnish: do gziping in varnish rather than varnish - 13https://github.com/miraheze/puppet/pull/2511 [19:03:56] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/450e477c4ea5...e26741541df5 [19:03:58] [02miraheze/puppet] 07paladox 03e267415 - varnish: do gziping in varnish rather than varnish (#2511) [19:03:59] [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [19:04:00] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9 [19:04:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.30, 10.19, 10.00 [19:04:38] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7ea243c18bbf...695336c72156 [19:04:40] [02miraheze/ssl] 07MirahezeSSLBot 03695336c - Bot: Update SSL cert for heavyironmodding.org [19:05:12] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.46, 1.70, 1.99 [19:05:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e26741541df5...0b336ae4de8b [19:05:30] [02miraheze/puppet] 07paladox 030b336ae - varnish: Set proxy_http_version to 1.1 for /check [19:06:33] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 10.04, 10.01, 9.95 [19:06:50] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.17, 2.93, 3.34 [19:07:12] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.05, 1.88, 2.02 [19:10:23] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.02, 10.71, 10.52 [19:10:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.75, 10.51, 9.84 [19:10:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.14, 10.92, 10.30 [19:11:12] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.19, 1.78, 1.96 [19:11:47] PROBLEM - cp21 Current Load on cp21 is CRITICAL: CRITICAL - load average: 1.10, 2.13, 1.74 [19:12:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.44, 11.65, 11.16 [19:12:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.60, 11.00, 10.40 [19:13:12] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.04, 1.96, 2.01 [19:13:43] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 0.93, 1.79, 1.66 [19:13:56] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ImportDump/compare/b74f2a364f92...d1c7409994a2 [19:13:57] [02miraheze/ImportDump] 07Universal-Omega 03d1c7409 - Delete dependencies [19:13:59] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [19:14:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.69, 11.41, 10.89 [19:14:38] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/d1c7409994a2...ec955e7670eb [19:14:40] [02miraheze/ImportDump] 07Universal-Omega 03ec955e7 - Update config.php [19:14:41] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [19:15:39] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-1/±0] 13https://github.com/miraheze/ImportDump/compare/ec955e7670eb...d2d4d6ff0d89 [19:15:39] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 1.41, 1.64, 1.62 [19:15:41] [02miraheze/ImportDump] 07Universal-Omega 03d2d4d6f - Delete .gitattributes [19:15:42] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [19:16:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 8.32, 10.98, 11.08 [19:16:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.50, 9.76, 9.82 [19:16:46] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.63, 3.42, 3.25 [19:19:37] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:19:49] PROBLEM - test101 Current Load on test101 is CRITICAL: CRITICAL - load average: 2.29, 1.87, 1.66 [19:20:31] miraheze/ImportDump - Universal-Omega the build passed. [19:20:33] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.67, 9.79, 10.15 [19:20:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.90, 9.44, 10.18 [19:21:10] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:21:11] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:21:35] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [19:21:49] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.94, 1.91, 1.70 [19:21:58] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:23:08] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [19:23:10] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [19:23:11] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [19:23:12] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/d2d4d6ff0d89...85eab7d2c5ad [19:23:13] [02miraheze/ImportDump] 07Universal-Omega 0385eab7d - Remove source from pager [19:23:22] [02mw-config] 07Reception123 closed pull request 03#4609: rcenhancedfilters & usenewrc settings for squareenixwiki and metroid … - 13https://github.com/miraheze/mw-config/pull/4609 [19:23:23] [02mw-config] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config [19:23:25] [02miraheze/mw-config] 07Reception123 deleted branch 03Reception123-patch-1 [19:23:26] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/54959e7268d2...c2aaf15c237b [19:23:28] [02miraheze/mw-config] 07Reception123 03c2aaf15 - rcenhancedfilters & usenewrc settings for squareenixwiki and metroid (T9100) (#4609) [19:23:56] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:24:33] miraheze/mw-config - Reception123 the build passed. [19:24:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.92, 10.09, 10.31 [19:24:59] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [19:25:00] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/85eab7d2c5ad...e02ff5d7cda8 [19:25:02] [02miraheze/ImportDump] 07Universal-Omega 03e02ff5d - Remove source from pager [19:25:46] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.55, 3.95, 3.31 [19:25:48] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 3.71, 4.07, 3.31 [19:26:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.35, 8.97, 9.97 [19:27:45] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.20, 4.00, 3.40 [19:28:10] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.33, 2.17, 1.60 [19:28:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.36, 9.96, 9.75 [19:29:43] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.45, 3.77, 3.38 [19:29:58] miraheze/ImportDump - Universal-Omega the build passed. [19:30:05] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.49, 1.94, 1.58 [19:30:23] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.78, 10.77, 10.45 [19:30:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.83, 10.10, 9.83 [19:30:40] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.45, 9.45, 10.12 [19:30:58] RECOVERY - heavyironmodding.org - LetsEncrypt on sslhost is OK: OK - Certificate 'heavyironmodding.org' will expire on Mon 18 Jul 2022 18:04:31 GMT +0000. [19:31:42] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 6.00, 4.42, 3.65 [19:32:00] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.69, 2.15, 1.70 [19:32:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.04, 11.06, 10.88 [19:32:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.01, 10.90, 10.54 [19:33:55] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.33, 1.85, 1.64 [19:34:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.56, 11.08, 10.68 [19:35:20] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.53, 10.81, 10.34 [19:35:49] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.61, 1.66, 1.70 [19:35:50] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.97, 1.48, 1.53 [19:36:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.36, 11.06, 10.96 [19:36:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 8.85, 10.46, 10.52 [19:37:01] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:37:17] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.07, 11.37, 10.59 [19:38:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.59, 9.83, 10.16 [19:39:33] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.33, 10.04, 10.00 [19:41:09] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:41:10] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.36, 11.10, 10.70 [19:41:12] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:41:20] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.94, 3.89 [19:41:36] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.26, 3.80, 3.91 [19:41:38] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:41:50] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:42:01] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [19:42:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.77, 10.38, 10.29 [19:44:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.20, 9.97, 10.15 [19:45:12] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.12, 1.72, 2.00 [19:45:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.29, 10.15, 10.13 [19:46:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 10.15, 9.69, 10.12 [19:47:00] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.57, 11.34, 10.84 [19:48:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.79, 9.76, 10.19 [19:48:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.21, 10.74, 10.43 [19:48:41] !log [@test101] starting deploy of {'config': True} to all [19:48:42] !log [@test101] finished deploy of {'config': True} to all - SUCCESS in 0s [19:48:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:57] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.97, 10.78, 10.70 [19:49:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:49:12] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.30, 2.02, 2.06 [19:49:25] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.19, 10.57, 10.30 [19:49:31] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.05, 3.69, 3.78 [19:51:02] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.19, 2.95, 3.40 [19:51:30] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.24, 3.51, 3.71 [19:52:03] !log [@mwtask111] starting deploy of {'config': True} to all [19:52:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:52:28] !log [@mwtask111] finished deploy of {'config': True} to all - SUCCESS in 25s [19:52:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:53:10] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 39 seconds ago with 0 failures [19:55:12] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.37, 1.89, 1.99 [19:55:27] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.08, 3.73, 3.75 [19:56:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.73, 9.83, 10.16 [19:56:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.20, 10.27, 10.20 [19:57:26] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.76, 3.75, 3.76 [19:57:36] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:57:53] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.00, 4.07, 3.68 [19:58:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.56, 9.20, 9.99 [19:58:57] PROBLEM - cp21 Current Load on cp21 is WARNING: WARNING - load average: 1.96, 1.76, 1.42 [19:59:19] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.21, 11.50, 10.79 [20:00:04] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/e02ff5d7cda8...28219d319943 [20:00:06] [02miraheze/ImportDump] 07Universal-Omega 0328219d3 - Use status for onSubmit; support rate limiting [20:00:07] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [20:00:56] RECOVERY - cp21 Current Load on cp21 is OK: OK - load average: 0.70, 1.36, 1.32 [20:01:09] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.15, 11.49, 10.64 [20:01:18] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.35, 11.28, 10.78 [20:01:21] [02miraheze/ImportDump] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/28219d319943...30cd9ede9487 [20:01:23] [02miraheze/ImportDump] 07github-actions 0330cd9ed - CI: lint code to MediaWiki standards [20:01:24] [02ImportDump] 07github-actions[bot] synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [20:01:45] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.06, 3.69, 3.61 [20:03:05] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.23, 11.33, 10.68 [20:03:12] PROBLEM - cp20 Current Load on cp20 is CRITICAL: CRITICAL - load average: 2.46, 2.02, 1.94 [20:03:42] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 3.83, 3.66 [20:03:48] miraheze/ImportDump - Universal-Omega the build has errored. [20:03:59] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [20:04:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.50, 10.28, 10.08 [20:05:12] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.94, 1.87, 1.89 [20:05:21] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.19, 2.81, 3.34 [20:05:38] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 1.91, 3.11, 3.42 [20:06:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.45, 10.58, 10.21 [20:09:31] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.73, 3.59, 3.52 [20:09:57] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [20:10:05] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.81, 11.20, 10.35 [20:10:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.13, 11.17, 10.52 [20:11:27] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.38, 3.13, 3.36 [20:12:00] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.56, 11.25, 10.47 [20:12:01] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:12:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.43, 10.82, 10.41 [20:12:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.02, 9.63, 10.09 [20:13:10] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.47, 9.45, 10.15 [20:13:12] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.36, 1.49, 1.70 [20:14:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.25, 9.86, 10.11 [20:15:49] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 6.84, 9.27, 9.86 [20:15:56] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 30 seconds ago with 0 failures [20:16:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.92, 11.19, 10.82 [20:16:37] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.38, 9.69, 10.18 [20:17:03] alerting : [FIRING:1] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:17:10] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - dnd.bellinrattin.it All nameservers failed to answer the query. [20:17:12] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.41, 1.61, 1.71 [20:18:23] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.17, 11.12, 10.54 [20:19:12] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 0.77, 1.35, 1.61 [20:20:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.41, 10.83, 10.51 [20:20:31] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.24, 10.25, 10.33 [20:21:04] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.98, 10.28, 10.26 [20:21:08] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [20:22:11] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 6.00, 4.67, 3.89 [20:22:49] PROBLEM - cp30 Current Load on cp30 is CRITICAL: CRITICAL - load average: 2.59, 1.82, 1.35 [20:23:08] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.66, 3.88, 3.44 [20:24:23] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.08, 9.89, 10.18 [20:24:49] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 1.14, 1.59, 1.33 [20:27:06] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.18, 3.78, 3.48 [20:28:00] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.06, 3.84, 3.78 [20:28:17] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.09, 10.89, 10.53 [20:29:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.01, 3.91, 3.81 [20:30:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.38, 10.61, 10.44 [20:30:19] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.85, 3.51, 3.94 [20:31:03] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.88, 3.46, 3.43 [20:31:53] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.97, 3.54, 3.68 [20:32:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.90, 11.43, 10.77 [20:32:57] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.37, 11.37, 10.84 [20:33:02] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.30, 3.09, 3.30 [20:34:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.22, 10.33, 10.44 [20:34:17] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.33, 3.59, 3.86 [20:34:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.01, 10.66, 10.61 [20:34:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.75, 11.38, 10.45 [20:34:55] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.41, 11.30, 10.88 [20:36:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.64, 10.51, 10.56 [20:36:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.39, 11.56, 10.64 [20:37:30] PROBLEM - cp30 Current Load on cp30 is WARNING: WARNING - load average: 1.52, 1.97, 1.62 [20:37:43] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.13, 3.57, 3.60 [20:37:52] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.38, 10.56, 10.09 [20:38:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.94, 9.48, 10.10 [20:38:57] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.98, 4.12, 3.63 [20:39:47] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.39, 9.76, 9.86 [20:40:07] PROBLEM - cp20 Current Load on cp20 is WARNING: WARNING - load average: 1.87, 1.57, 1.46 [20:41:20] RECOVERY - cp30 Current Load on cp30 is OK: OK - load average: 0.85, 1.46, 1.49 [20:42:01] alerting : [FIRING:2] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [20:42:06] RECOVERY - cp20 Current Load on cp20 is OK: OK - load average: 1.08, 1.36, 1.39 [20:42:23] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.23, 11.19, 10.72 [20:43:32] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.08, 3.93, 3.80 [20:44:00] PROBLEM - cp30 Stunnel HTTP for mw112 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:44:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.26, 10.13, 10.40 [20:45:28] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.69, 4.12, 3.87 [20:45:57] RECOVERY - cp30 Stunnel HTTP for mw112 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 0.906 second response time [20:46:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.28, 10.84, 10.67 [20:46:50] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [20:46:50] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [20:47:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.58, 3.77, 3.77 [20:48:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.48, 10.24, 10.12 [20:48:30] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.37, 10.66, 10.12 [20:48:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 13.06, 11.43, 10.74 [20:48:47] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.78, 11.45, 10.99 [20:49:21] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 5.75, 4.49, 4.03 [20:50:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.16, 10.30, 10.05 [20:50:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.37, 11.47, 11.01 [20:50:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.76, 11.14, 10.73 [20:50:46] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.98, 11.53, 11.07 [20:51:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/846796a3d285 [20:51:55] [02miraheze/puppet] 07paladox 03846796a - varnish: set nginx backlog config to 4096 matching sysctl [20:51:57] [02puppet] 07paladox created branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [20:51:58] [02puppet] 07paladox opened pull request 03#2512: varnish: set nginx backlog config to 4096 matching sysctl - 13https://github.com/miraheze/puppet/pull/2512 [20:52:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.74, 9.69, 9.86 [20:52:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/846796a3d285...377fdb9a9b66 [20:52:41] [02miraheze/puppet] 07paladox 03377fdb9 - Update mediawiki.conf [20:52:42] [02puppet] 07paladox synchronize pull request 03#2512: varnish: set nginx backlog config to 4096 matching sysctl - 13https://github.com/miraheze/puppet/pull/2512 [20:54:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 9.67, 10.10, 10.11 [20:56:03] PROBLEM - cp31 Stunnel HTTP for mw111 on cp31 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:56:06] PROBLEM - mw111 MediaWiki Rendering on mw111 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:58:22] PROBLEM - cp20 Stunnel HTTP for mw111 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:00:10] RECOVERY - cp31 Stunnel HTTP for mw111 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14626 bytes in 6.926 second response time [21:00:14] RECOVERY - mw111 MediaWiki Rendering on mw111 is OK: HTTP OK: HTTP/1.1 200 OK - 22336 bytes in 6.068 second response time [21:00:22] RECOVERY - cp20 Stunnel HTTP for mw111 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14640 bytes in 5.531 second response time [21:00:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.41, 9.45, 10.03 [21:00:26] Reception123: okay.. [21:00:39] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.63, 9.45, 10.17 [21:02:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.68, 10.74, 10.33 [21:02:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.15, 10.80, 10.67 [21:04:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.70, 10.98, 10.75 [21:04:35] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.20, 10.76, 10.53 [21:06:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 14.81, 12.37, 11.04 [21:06:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.56, 10.91, 10.63 [21:06:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.80, 9.58, 10.13 [21:07:13] PROBLEM - cp20 Stunnel HTTP for mw101 on cp20 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:07:27] PROBLEM - cp30 Stunnel HTTP for mw101 on cp30 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [21:08:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.45, 11.88, 11.02 [21:09:12] RECOVERY - cp20 Stunnel HTTP for mw101 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 2.073 second response time [21:09:28] RECOVERY - cp30 Stunnel HTTP for mw101 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14634 bytes in 1.945 second response time [21:10:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.76, 12.53, 11.36 [21:10:32] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.23, 9.66, 10.16 [21:11:19] PROBLEM - cloud12 Puppet on cloud12 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [21:12:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.88, 11.27, 10.67 [21:12:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.89, 10.27, 10.21 [21:14:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.00, 9.68, 9.99 [21:16:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.71, 11.94, 11.51 [21:16:29] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp21.miraheze.org - CNAME OK [21:20:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 11.71, 10.49, 10.26 [21:23:30] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.38, 10.41, 10.24 [21:25:28] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.03, 9.75, 10.01 [21:26:23] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.22, 10.92, 10.81 [21:28:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.47, 10.57, 10.71 [21:28:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.51, 10.01, 9.83 [21:29:23] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 11.03, 10.52, 10.28 [21:30:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.21, 11.41, 11.11 [21:32:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.86, 10.18, 9.97 [21:32:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.62, 9.73, 10.06 [21:34:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 13.95, 11.72, 10.97 [21:36:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.05, 11.81, 11.46 [21:36:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.36, 11.25, 10.88 [21:38:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.84, 10.38, 10.22 [21:39:15] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.64, 9.81, 10.07 [21:39:18] RECOVERY - cloud12 Puppet on cloud12 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [21:40:23] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.02, 10.62, 10.54 [21:40:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.92, 9.88, 10.06 [21:42:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.02, 10.85, 10.27 [21:44:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 8.17, 10.06, 10.38 [21:44:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 7.60, 9.88, 10.00 [21:46:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.24, 11.22, 11.13 [21:46:33] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.62, 10.89, 10.62 [21:48:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.72, 11.34, 11.18 [21:48:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.09, 10.77, 10.60 [21:50:25] PROBLEM - mw121 Current Load on mw121 is CRITICAL: CRITICAL - load average: 12.42, 10.81, 10.29 [21:50:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.60, 11.80, 10.72 [21:52:24] [02puppet] 07paladox closed pull request 03#2512: varnish: set nginx backlog config to 4096 matching sysctl - 13https://github.com/miraheze/puppet/pull/2512 [21:52:25] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 9.47, 10.46, 10.24 [21:52:26] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0b336ae4de8b...86c59fc27a9c [21:52:27] [02miraheze/puppet] 07paladox 0386c59fc - varnish: set nginx backlog config to 4096 matching sysctl (#2512) [21:52:29] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9 [21:52:30] [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [21:53:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-9 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/e750da326c8a [21:54:00] [02miraheze/puppet] 07paladox 03e750da3 - mediawiki: set nginx backlog config to 4096 matching sysctl [21:54:02] [02puppet] 07paladox created branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [21:54:03] [02puppet] 07paladox opened pull request 03#2513: mediawiki: set nginx backlog config to 4096 matching sysctl - 13https://github.com/miraheze/puppet/pull/2513 [21:54:21] [02puppet] 07paladox closed pull request 03#2513: mediawiki: set nginx backlog config to 4096 matching sysctl - 13https://github.com/miraheze/puppet/pull/2513 [21:54:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/86c59fc27a9c...f4b1309ac404 [21:54:23] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.44, 10.69, 10.47 [21:54:24] [02miraheze/puppet] 07paladox 03f4b1309 - mediawiki: set nginx backlog config to 4096 matching sysctl (#2513) [21:54:25] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.95, 10.15, 10.14 [21:54:26] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-9 [21:54:27] [02puppet] 07paladox deleted branch 03paladox-patch-9 - 13https://github.com/miraheze/puppet [21:54:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 9.24, 11.00, 10.69 [21:55:00] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.20, 10.31, 9.92 [21:56:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.74, 10.92, 10.59 [21:56:59] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 9.50, 9.69, 9.72 [21:58:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 13.37, 11.29, 10.78 [22:00:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.00, 9.73, 10.16 [22:00:41] PROBLEM - graylog121 Puppet on graylog121 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [22:01:23] PROBLEM - mon111 Puppet on mon111 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [22:01:37] PROBLEM - cp21 Puppet on cp21 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [22:01:47] PROBLEM - cp20 Puppet on cp20 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [22:02:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.29, 11.31, 10.98 [22:03:10] PROBLEM - cp31 Puppet on cp31 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [22:03:11] PROBLEM - cp30 Puppet on cp30 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [22:03:35] RECOVERY - cp21 Puppet on cp21 is OK: OK: Puppet is currently enabled, last run 35 seconds ago with 0 failures [22:03:46] RECOVERY - cp20 Puppet on cp20 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [22:05:08] RECOVERY - cp31 Puppet on cp31 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:05:10] RECOVERY - cp30 Puppet on cp30 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:05:29] PROBLEM - mw101 Puppet on mw101 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [22:08:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.36, 9.01, 10.06 [22:10:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 11.62, 10.59, 10.22 [22:12:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.22, 9.59, 9.90 [22:12:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.05, 9.21, 10.01 [22:16:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.75, 9.88, 9.93 [22:16:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.88, 3.74, 4.00 [22:16:33] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.17, 9.72, 10.19 [22:18:14] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 13.95, 11.43, 10.50 [22:18:43] PROBLEM - dnd.bellinrattin.it - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 155, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 136, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 73, in check_records nameserversans = dns_resolver.resolve(root_domain, 'NS') File "/usr/l [22:18:43] n3/dist-packages/dns/resolver.py", line 1040, in resolve (nameserver, port, tcp, backoff) = resolution.next_nameserver() File "/usr/lib/python3/dist-packages/dns/resolver.py", line 598, in next_nameserver raise NoNameservers(request=self.request, errors=self.errors)dns.resolver.NoNameservers: All nameservers failed to answer the query bellinrattin.it. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [22:20:14] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.29, 11.35, 10.59 [22:20:29] PROBLEM - mw112 Current Load on mw112 is CRITICAL: CRITICAL - load average: 12.12, 11.56, 10.61 [22:20:35] PROBLEM - mw111 Current Load on mw111 is CRITICAL: CRITICAL - load average: 12.83, 11.09, 9.94 [22:22:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.47, 9.90, 10.15 [22:22:29] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.97, 11.01, 10.51 [22:24:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.28, 10.18, 10.10 [22:24:35] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 9.97, 11.14, 10.27 [22:24:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.63, 3.39, 3.91 [22:26:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.92, 3.63, 3.71 [22:26:35] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 7.34, 10.06, 10.00 [22:28:23] PROBLEM - mw102 Current Load on mw102 is CRITICAL: CRITICAL - load average: 12.50, 10.53, 10.01 [22:28:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.87, 3.39, 3.62 [22:28:40] RECOVERY - graylog121 Puppet on graylog121 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:29:20] RECOVERY - mon111 Puppet on mon111 is OK: OK: Puppet is currently enabled, last run 36 seconds ago with 0 failures [22:30:23] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.77, 10.58, 10.10 [22:30:33] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.82, 9.47, 9.91 [22:31:04] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.31, 10.86, 10.41 [22:31:43] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.43, 10.19, 10.00 [22:31:44] RECOVERY - mw101 Puppet on mw101 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [22:32:23] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.47, 10.19, 10.02 [22:32:29] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 8.37, 9.31, 9.99 [22:33:01] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 9.61, 10.38, 10.29 [22:33:38] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 9.15, 10.03, 9.97 [22:34:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.57, 2.75, 3.32 [22:34:57] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 6.99, 9.31, 9.92 [22:36:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 3.23, 3.17, 3.40 [22:36:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.25, 10.17, 10.14 [22:38:33] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 9.98, 10.18, 10.16 [22:38:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.73, 4.03, 3.74 [22:40:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.19, 3.86, 3.64 [22:40:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.45, 3.77, 3.68 [22:42:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.83, 3.52, 3.54 [22:44:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.18, 3.92, 3.68 [22:46:57] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.66, 4.23, 3.88 [22:48:23] RECOVERY - dnd.bellinrattin.it - reverse DNS on sslhost is OK: SSL OK - dnd.bellinrattin.it reverse DNS resolves to cp20.miraheze.org - CNAME OK [22:50:29] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 10.84, 10.24, 9.92 [22:52:25] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.95, 9.47, 9.69 [22:52:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.15, 3.64, 3.67 [22:52:57] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.51, 3.96, 3.90 [22:57:01] ok : [RESOLVED] (mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:00:02] !log [@test101] starting deploy of {'l10nupdate': True} to all [23:00:03] !log [@mwtask111] starting deploy of {'l10nupdate': True} to all [23:00:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:00:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:02:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.35, 2.86, 3.37 [23:05:51] PROBLEM - mw122 Current Load on mw122 is CRITICAL: CRITICAL - load average: 12.36, 10.09, 9.56 [23:06:31] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:08:29] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.16, 4.12, 3.76 [23:09:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.29, 3.22, 3.92 [23:09:44] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 9.76, 10.46, 9.89 [23:09:50] PROBLEM - phab121 Current Load on phab121 is CRITICAL: CRITICAL - load average: 2.57, 1.71, 1.32 [23:10:29] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.01, 3.60, 3.60 [23:11:41] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.12, 9.40, 9.57 [23:11:50] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 1.66, 1.57, 1.31 [23:13:13] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.82, 10.34, 9.63 [23:14:29] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.30, 3.00, 3.37 [23:17:10] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.57, 10.15, 9.75 [23:17:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.38, 3.75, 3.82 [23:19:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.68, 3.58, 3.74 [23:19:36] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 3.73, 3.52, 3.43 [23:19:49] PROBLEM - test101 Current Load on test101 is WARNING: WARNING - load average: 1.77, 1.52, 0.99 [23:21:33] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.39, 3.92, 3.59 [23:21:49] RECOVERY - test101 Current Load on test101 is OK: OK - load average: 1.57, 1.62, 1.10 [23:22:43] !log [@test101] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1361s [23:22:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:23:24] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 3.47, 3.52, 3.47 [23:23:36] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.50, 3.65, 3.71 [23:25:23] !log [@mwtask111] finished deploy of {'l10nupdate': True} to all - SUCCESS in 1520s [23:25:25] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.96, 3.75, 3.62 [23:25:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:25:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.40, 3.56, 3.68 [23:27:22] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.15, 4.09, 3.68 [23:27:22] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.66, 4.03, 3.73 [23:27:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.34, 3.66, 3.68 [23:28:27] PROBLEM - mw101 Current Load on mw101 is CRITICAL: CRITICAL - load average: 12.51, 10.86, 10.06 [23:30:23] PROBLEM - mw101 Current Load on mw101 is WARNING: WARNING - load average: 11.44, 11.16, 10.27 [23:31:15] PROBLEM - gluster111 Current Load on gluster111 is WARNING: WARNING - load average: 2.78, 3.70, 3.68 [23:31:19] PROBLEM - gluster121 Current Load on gluster121 is WARNING: WARNING - load average: 2.98, 3.57, 3.56 [23:33:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.97, 3.36, 3.54 [23:34:59] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 10.77, 10.05, 9.48 [23:35:36] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.86, 2.88, 3.34 [23:37:16] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.67, 3.07, 3.35 [23:40:49] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 8.83, 9.85, 9.63 [23:41:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.33, 3.43, 3.45 [23:42:51] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 9.84, 10.30, 9.83 [23:42:57] RECOVERY - gluster111 Current Load on gluster111 is OK: OK - load average: 2.41, 3.01, 3.35 [23:43:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.47, 3.66, 3.53 [23:43:49] PROBLEM - phab121 Current Load on phab121 is WARNING: WARNING - load average: 1.76, 1.62, 1.38 [23:43:53] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 10.58, 10.15, 9.44 [23:44:14] RECOVERY - mw101 Current Load on mw101 is OK: OK - load average: 7.26, 9.61, 10.04 [23:45:47] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 8.05, 9.34, 9.23 [23:45:49] RECOVERY - phab121 Current Load on phab121 is OK: OK - load average: 1.03, 1.39, 1.33 [23:46:12] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://github.com/miraheze/ImportDump/compare/30cd9ede9487...f14ea2f53abd [23:46:13] [02miraheze/ImportDump] 07Universal-Omega 03f14ea2f - Create ImportDumpRequestViewer.php [23:46:15] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [23:46:31] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:46:49] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 8.61, 9.93, 9.83 [23:47:19] [02miraheze/ImportDump] 07github-actions[bot] pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/f14ea2f53abd...1ace53b5ba16 [23:47:21] [02miraheze/ImportDump] 07github-actions 031ace53b - CI: lint code to MediaWiki standards [23:47:22] [02ImportDump] 07github-actions[bot] synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [23:47:36] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.46, 3.63, 3.60 [23:48:19] miraheze/ImportDump - Universal-Omega the build has errored. [23:49:35] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.98, 3.05, 3.38 [23:52:01] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/1ace53b5ba16...da65d66fad9d [23:52:02] [02miraheze/ImportDump] 07Universal-Omega 03da65d66 - Update SpecialImportDumpRequestQueue.php [23:52:04] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [23:52:44] PROBLEM - mw102 Current Load on mw102 is WARNING: WARNING - load average: 10.73, 10.29, 9.97 [23:52:52] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/da65d66fad9d...9afae7fbd510 [23:52:54] [02miraheze/ImportDump] 07Universal-Omega 039afae7f - Update extension.json [23:52:55] [02ImportDump] 07Universal-Omega synchronize pull request 03#1: Create extension - 13https://github.com/miraheze/ImportDump/pull/1 [23:53:01] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [23:53:35] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.26, 3.78, 3.67 [23:54:34] PROBLEM - mw112 Current Load on mw112 is WARNING: WARNING - load average: 10.38, 10.21, 9.54 [23:54:42] RECOVERY - mw102 Current Load on mw102 is OK: OK - load average: 9.01, 9.90, 9.87 [23:55:11] miraheze/ImportDump - Universal-Omega the build has errored. [23:56:33] RECOVERY - mw112 Current Load on mw112 is OK: OK - load average: 9.42, 10.00, 9.55 [23:56:33] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.12, 10.33, 9.93 [23:57:36] PROBLEM - mw111 Current Load on mw111 is WARNING: WARNING - load average: 10.21, 9.93, 9.21 [23:58:59] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 5.52, 4.54, 3.78 [23:59:34] RECOVERY - mw111 Current Load on mw111 is OK: OK - load average: 8.55, 9.45, 9.12 [23:59:35] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.35, 4.36, 3.90 [23:59:40] PROBLEM - gluster111 Current Load on gluster111 is CRITICAL: CRITICAL - load average: 4.12, 4.00, 3.57