[00:05:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.39, 19.32, 19.00 [00:17:25] PROBLEM - cloud17 Puppet on cloud17 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [00:31:34] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.91, 19.95, 18.53 [00:35:28] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.02, 19.35, 18.66 [00:45:25] RECOVERY - cloud17 Puppet on cloud17 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:50:07] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.55, 21.85, 19.92 [00:52:07] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [00:57:52] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.37, 22.36, 20.68 [00:59:49] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.80, 21.62, 20.60 [01:03:42] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.58, 22.81, 21.21 [01:07:35] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.78, 22.45, 21.50 [01:13:25] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 12.12, 16.67, 19.37 [01:23:10] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.68, 21.27, 20.24 [01:24:08] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [01:25:07] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.89, 20.04, 19.91 [01:44:44] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.65, 22.66, 21.08 [01:50:34] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.55, 22.58, 21.39 [01:52:31] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.40, 21.53, 21.22 [01:54:05] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [01:58:20] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.72, 19.55, 20.37 [03:01:03] RECOVERY - db171 Backups SQL on db171 is OK: FILE_AGE OK: /var/log/sql-backup.log is 62 seconds old and 0 bytes [03:29:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.60, 20.95, 18.24 [03:31:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.09, 19.44, 18.01 [04:03:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.15, 21.94, 17.91 [04:07:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.83, 23.07, 19.29 [04:11:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.01, 23.43, 20.29 [04:21:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.89, 22.66, 21.92 [04:45:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.53, 22.16, 21.36 [04:47:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.81, 22.72, 21.70 [04:55:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.60, 18.77, 20.25 [05:20:38] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.69, 18.82, 18.67 [05:22:34] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.42, 18.19, 18.49 [05:23:07] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.38, 20.73, 16.77 [05:25:08] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.63, 21.95, 17.73 [05:26:30] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.37, 21.33, 19.70 [05:28:26] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.84, 21.60, 20.00 [05:29:07] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.48, 18.71, 17.48 [05:35:37] [02CreateWiki] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/6793c1073caa [05:35:38] [02CreateWiki] 07The-Voidwalker 036793c10 - remove top from oo-ui-menuSelectWidget [05:35:41] [02CreateWiki] 07The-Voidwalker created branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/CreateWiki [05:35:43] [02CreateWiki] 07The-Voidwalker opened pull request 03#541: remove top from oo-ui-menuSelectWidget - 13https://github.com/miraheze/CreateWiki/pull/541 [05:35:45] [02CreateWiki] 07coderabbitai[bot] commented on pull request 03#541: remove top from oo-ui-menuSelectWidget - 13https://github.com/miraheze/CreateWiki/pull/541#issuecomment-2208152368 [05:36:13] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.18, 22.59, 20.92 [05:38:10] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.07, 21.34, 20.64 [05:44:51] miraheze/CreateWiki - The-Voidwalker the build passed. [05:49:49] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.34, 23.79, 22.00 [05:51:25] [02CreateWiki] 07The-Voidwalker closed pull request 03#541: remove top from oo-ui-menuSelectWidget - 13https://github.com/miraheze/CreateWiki/pull/541 [05:51:28] [02CreateWiki] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/7207b0b97564...ef21a385aa25 [05:51:29] [02CreateWiki] 07The-Voidwalker 03ef21a38 - remove top from oo-ui-menuSelectWidget (#541) [05:51:31] [02CreateWiki] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 [05:51:32] [02CreateWiki] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/CreateWiki [05:54:58] !log [void@test151] starting deploy of {'versions': ['1.41', '1.42'], 'upgrade_extensions': 'CreateWiki'} to test151 [05:55:01] !log [void@test151] finished deploy of {'versions': ['1.41', '1.42'], 'upgrade_extensions': 'CreateWiki'} to test151 - SUCCESS in 2s [05:55:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:55:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:56:17] !log [void@mwtask181] starting deploy of {'versions': ['1.41', '1.42'], 'upgrade_extensions': 'CreateWiki'} to all [05:56:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:56:43] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [05:56:50] !log [void@mwtask181] finished deploy of {'versions': ['1.41', '1.42'], 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 32s [05:56:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:00:27] miraheze/CreateWiki - The-Voidwalker the build passed. [06:55:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.57, 22.53, 23.79 [06:57:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.54, 24.75, 24.44 [07:11:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.27, 21.65, 23.57 [07:15:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.63, 23.30, 23.68 [07:19:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.33, 23.43, 23.67 [07:23:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.39, 25.08, 24.27 [07:49:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.58, 20.69, 23.64 [07:57:52] PROBLEM - elephants.ggreig.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'elephants.ggreig.com' expires in 15 day(s) (Sat 20 Jul 2024 07:38:29 AM GMT +0000). [07:58:04] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a1e41c0b0ab5...8c690367c57b [07:58:05] [02ssl] 07WikiTideSSLBot 038c69036 - Bot: Update SSL cert for elephants.ggreig.com [07:59:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.95, 22.60, 22.74 [08:01:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.14, 22.11, 22.56 [08:11:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.33, 24.50, 23.45 [08:12:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:15:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.85, 23.16, 23.35 [08:21:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.79, 22.04, 22.54 [08:25:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.67, 23.40, 23.05 [08:27:52] RECOVERY - elephants.ggreig.com - LetsEncrypt on sslhost is OK: OK - Certificate 'elephants.ggreig.com' will expire on Wed 02 Oct 2024 06:57:58 AM GMT +0000. [08:29:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.57, 23.87, 23.15 [08:31:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.37, 21.95, 22.51 [08:43:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.66, 22.68, 22.35 [08:45:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.86, 22.78, 22.47 [08:51:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.69, 22.18, 22.16 [08:53:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.00, 22.14, 22.16 [09:15:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.04, 23.33, 22.16 [09:19:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.84, 22.92, 22.43 [09:37:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.93, 19.08, 20.16 [10:19:47] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.80, 21.94, 20.05 [10:21:12] PROBLEM - wiki.dragonsmp.pro - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.dragonsmp.pro' expires in 15 day(s) (Sat 20 Jul 2024 09:52:16 AM GMT +0000). [10:21:27] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8c690367c57b...e24cb9e52a74 [10:21:30] [02ssl] 07WikiTideSSLBot 03e24cb9e - Bot: Update SSL cert for wiki.dragonsmp.pro [10:23:40] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.26, 23.60, 21.25 [10:27:34] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.80, 24.78, 22.15 [10:31:27] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.59, 21.80, 21.67 [10:31:33] !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'VoteNY'} to test151 [10:31:34] !log [alex@test151] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'VoteNY'} to test151 - SUCCESS in 1s [10:31:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:31:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:35:20] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.55, 22.85, 22.01 [10:37:17] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.41, 22.53, 22.04 [10:43:06] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.24, 18.57, 20.23 [10:50:54] RECOVERY - wiki.dragonsmp.pro - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.dragonsmp.pro' will expire on Wed 02 Oct 2024 09:21:20 AM GMT +0000. [10:57:55] [02puppet] 07redbluegreenhat pushed 031 commit to 03mwdeploy-skip-certain-extensions [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/ec362a0c3fe5 [10:57:57] [02puppet] 07redbluegreenhat 03ec362a0 - mediawiki: create a file on /var/local/mwdeploy for extensions configured to checkout a specific commit [10:57:58] [02puppet] 07redbluegreenhat created branch 03mwdeploy-skip-certain-extensions - 13https://github.com/miraheze/puppet [11:01:05] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.61, 21.84, 19.73 [11:03:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.38, 22.12, 20.09 [11:04:43] [02puppet] 07redbluegreenhat opened pull request 03#3876: mediawiki: prevent mwdeploy from upgrading extensions configured to checkout a specific commit - 13https://github.com/miraheze/puppet/pull/3876 [11:04:47] [02puppet] 07coderabbitai[bot] commented on pull request 03#3876: mediawiki: prevent mwdeploy from upgrading extensions configured to checkout a specific commit - 13https://github.com/miraheze/puppet/pull/3876#issuecomment-2208700937 [11:05:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.93, 23.91, 20.99 [11:05:40] miraheze/puppet - redbluegreenhat the build has errored. [11:07:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.03, 21.95, 20.62 [11:11:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.62, 23.26, 21.34 [11:13:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.61, 22.63, 21.32 [11:17:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.98, 23.68, 22.00 [11:22:56] [02puppet] 07redbluegreenhat pushed 031 commit to 03pywikibot-auto-update [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/0c110d67c0e5 [11:22:58] [02puppet] 07redbluegreenhat 030c110d6 - pywikibot: auto-update and pull from the stable branch [11:23:01] [02puppet] 07redbluegreenhat created branch 03pywikibot-auto-update - 13https://github.com/miraheze/puppet [11:23:07] [02puppet] 07redbluegreenhat opened pull request 03#3877: pywikibot: auto-update and pull from the stable branch - 13https://github.com/miraheze/puppet/pull/3877 [11:23:12] [02puppet] 07coderabbitai[bot] commented on pull request 03#3877: pywikibot: auto-update and pull from the stable branch - 13https://github.com/miraheze/puppet/pull/3877#issuecomment-2208729675 [11:24:54] [02puppet] 07redbluegreenhat closed pull request 03#3877: pywikibot: auto-update and pull from the stable branch - 13https://github.com/miraheze/puppet/pull/3877 [11:24:56] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d5cb972cd5fc...661f34ee66e8 [11:24:57] [02puppet] 07redbluegreenhat 03661f34e - pywikibot: auto-update and pull from the stable branch (#3877) [11:24:59] [02puppet] 07redbluegreenhat deleted branch 03pywikibot-auto-update [11:25:00] [02puppet] 07redbluegreenhat deleted branch 03pywikibot-auto-update - 13https://github.com/miraheze/puppet [11:31:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 14.85, 21.26, 23.13 [11:45:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.36, 17.42, 19.99 [11:55:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.77, 21.51, 20.53 [11:56:52] PROBLEM - bots171 Puppet on bots171 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:59:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.43, 22.36, 21.32 [12:05:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.14, 20.70, 20.70 [12:07:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.97, 19.46, 20.24 [12:12:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:12:43] [02CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/CreateWiki/compare/ef21a385aa25...184b5cd3760d [12:12:44] [02CreateWiki] 07translatewiki 03184b5cd - Localisation updates from https://translatewiki.net. [12:12:45] [02ErrorPages] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/ErrorPages/compare/f4ed225d67fc...9de2e8d1ece2 [12:12:47] [02ErrorPages] 07translatewiki 039de2e8d - Localisation updates from https://translatewiki.net. [12:12:50] [02ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/505e85f05678...b734bfc6f5fb [12:12:53] [02ImportDump] 07translatewiki 03b734bfc - Localisation updates from https://translatewiki.net. [12:12:56] [02IncidentReporting] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/53975bcb9b22...d634317fbc30 [12:12:57] [02IncidentReporting] 07translatewiki 03d634317 - Localisation updates from https://translatewiki.net. [12:12:59] [02landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/5d54c9c18c55...8d195a9c5131 [12:13:00] [02landing] 07translatewiki 038d195a9 - Localisation updates from https://translatewiki.net. [12:13:01] [02DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/DataDump/compare/38687d9f633a...7a788c957193 [12:13:02] [02DataDump] 07translatewiki 037a788c9 - Localisation updates from https://translatewiki.net. [12:13:04] [02RottenLinks] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/b2468bb65bce...256a33bde18d [12:13:07] [02RottenLinks] 07translatewiki 03256a33b - Localisation updates from https://translatewiki.net. [12:13:09] [02MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±8] 13https://github.com/miraheze/MirahezeMagic/compare/7ae49e67b886...7869e47bfbfd [12:13:10] [02MirahezeMagic] 07translatewiki 037869e47 - Localisation updates from https://translatewiki.net. [12:13:45] miraheze/landing - translatewiki the build passed. [12:14:31] miraheze/ErrorPages - translatewiki the build passed. [12:16:32] miraheze/MirahezeMagic - translatewiki the build has errored. [12:17:25] miraheze/DataDump - translatewiki the build passed. [12:17:39] miraheze/IncidentReporting - translatewiki the build passed. [12:17:42] miraheze/RottenLinks - translatewiki the build passed. [12:21:39] miraheze/CreateWiki - translatewiki the build passed. [12:22:09] miraheze/ImportDump - translatewiki the build passed. [12:24:10] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [12:24:11] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:22] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [12:24:23] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:36] !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 [12:24:37] !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:33] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:33:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:44] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 10s [12:33:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:56] !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:34:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:34:06] !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 10s [12:34:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:31] !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:37:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:40] !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s [12:37:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:51] !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:37:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:00] !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 9s [12:38:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:53:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.84, 21.07, 18.67 [12:55:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.69, 20.16, 18.64 [12:59:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.67, 23.78, 20.39 [13:01:07] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.41, 22.36, 20.34 [13:03:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.45, 23.15, 20.86 [13:05:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.95, 22.67, 20.98 [13:07:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.83, 23.27, 21.36 [13:11:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.44, 23.89, 22.08 [13:13:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 23.54, 24.32, 22.49 [13:15:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.15, 22.64, 22.10 [13:29:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.19, 22.89, 21.99 [13:31:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.89, 22.40, 21.96 [13:36:41] [02MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/7869e47bfbfd...42a080ced2ed [13:36:43] [02MirahezeMagic] 07translatewiki 0342a080c - Localisation updates from https://translatewiki.net. [13:37:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.29, 22.27, 21.63 [13:38:04] !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [13:38:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:38:14] !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 10s [13:38:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:38:25] !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [13:38:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:38:34] !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 9s [13:38:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:39:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.28, 21.61, 21.49 [13:40:30] miraheze/MirahezeMagic - translatewiki the build has errored. [13:49:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.38, 17.89, 20.01 [13:54:06] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [13:54:07] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [13:54:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:54:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:54:18] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [13:54:19] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [13:54:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:54:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:54:31] !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 [13:54:32] !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [13:54:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:54:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:55:55] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [13:57:57] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/661f34ee66e8...9626916a9e60 [13:57:58] [02puppet] 07redbluegreenhat 039626916 - pywikibot: fix [13:57:59] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.50, 22.64, 21.39 [13:59:55] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.63, 20.92, 20.93 [14:01:52] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.96, 23.28, 21.79 [14:02:27] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [14:02:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:02:37] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 10s [14:02:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:02:48] !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [14:02:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:02:59] !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 11s [14:03:00] RECOVERY - bots171 Puppet on bots171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [14:03:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:03:48] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.64, 21.73, 21.42 [14:05:45] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.54, 23.09, 21.91 [14:07:42] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.49, 21.35, 21.45 [14:13:31] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.00, 19.04, 20.34 [14:23:17] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.15, 21.46, 20.41 [14:25:13] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.00, 20.04, 20.04 [14:29:09] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.62, 23.36, 21.44 [14:31:06] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.38, 20.94, 20.80 [14:33:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.43, 19.04, 20.17 [14:37:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.67, 20.40, 20.45 [14:39:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.01, 19.20, 20.00 [14:53:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.82, 20.09, 19.47 [14:55:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.02, 19.39, 19.27 [15:15:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.92, 21.53, 20.06 [15:16:45] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:21:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.67, 22.18, 20.67 [15:25:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 17.84, 21.56, 20.93 [15:33:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 16.56, 18.95, 20.01 [15:39:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.95, 20.92, 20.39 [15:43:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.10, 24.52, 21.90 [15:47:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 18.31, 22.57, 21.87 [15:51:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 13.73, 17.98, 20.15 [15:55:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.01, 20.77, 20.90 [15:59:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.84, 18.65, 20.08 [16:09:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.96, 19.80, 19.54 [16:11:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.06, 17.98, 18.92 [16:12:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:14:41] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:40:23] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.35, 20.91, 20.05 [16:48:09] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.67, 23.35, 21.42 [17:23:07] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.17, 22.66, 23.71 [17:29:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 26.69, 23.86, 23.84 [17:33:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.70, 23.54, 23.84 [17:39:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.11, 23.21, 23.56 [17:41:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.81, 23.12, 23.50 [17:45:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.62, 25.20, 24.12 [18:37:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.40, 21.10, 23.73 [18:41:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.55, 23.98, 24.18 [18:47:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [19:15:42] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [19:16:27] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:17:36] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.002810060978 secs [19:22:50] PROBLEM - ping6 on cp27 is WARNING: PING WARNING - Packet loss = 0%, RTA = 232.31 ms [19:27:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.17, 22.83, 23.85 [19:28:56] RECOVERY - ping6 on cp27 is OK: PING OK - Packet loss = 0%, RTA = 137.00 ms [19:29:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 28.25, 25.10, 24.58 [19:39:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 19.17, 21.65, 23.35 [19:41:39] PROBLEM - wiki.teessidehackspace.org.uk - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [19:43:03] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 29.07, 24.35, 23.91 [19:44:40] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [19:45:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.77, 23.48, 23.62 [19:46:30] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o