[00:02:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:52:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:07:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:17:04] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [03:18:59] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0008189082146 secs [03:23:24] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [03:25:25] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001539289951 secs [03:46:58] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [04:13:40] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 17 seconds ago with 0 failures [04:28:20] PROBLEM - wiki.3point0.science - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.3point0.science' expires in 15 day(s) (Wed 10 Jul 2024 04:22:18 AM GMT +0000). [05:46:43] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns] [06:44:53] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 19 seconds ago with 0 failures [06:49:05] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [06:50:59] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.001957297325 secs [07:04:13] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [07:06:08] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0001924037933 secs [07:53:23] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [07:55:18] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.002586513758 secs [08:39:14] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [08:41:09] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.002924352884 secs [09:47:35] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [09:50:41] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [09:52:42] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0007600188255 secs [09:57:02] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:59:07] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [10:01:02] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.0006218850613 secs [10:14:55] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [10:17:18] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:19:13] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.002040803432 secs [10:46:12] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:15:05] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [11:32:55] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:34:50] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.001645386219 secs [11:45:57] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [11:48:54] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [11:50:49] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.001619279385 secs [12:00:59] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [12:02:56] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 9.676814079e-05 secs [12:44:48] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [13:27:48] [02ManageWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/5c3202a3eeb1...90cd37c2510e [13:27:51] [02ManageWiki] 07BlankEclair 0390cd37c - T12237: Fix additional settings being potentially ignored (#471) [13:27:53] [02ManageWiki] 07redbluegreenhat closed pull request 03#471: T12237: Fix additional settings being potentially ignored - 13https://github.com/miraheze/ManageWiki/pull/471 [13:29:02] !log [alex@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'ManageWiki'} to all [13:29:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:29:14] !log [alex@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'ManageWiki'} to all - SUCCESS in 11s [13:29:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:31:46] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/resetWikiCaches.php --wiki=metawiki (END - exit=0) [13:31:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:31:55] miraheze/ManageWiki - redbluegreenhat the build has errored. [14:06:02] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:08:02] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.001553505659 secs [14:12:22] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [14:18:27] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0008450746536 secs [14:22:33] [02ManageWiki] 07redbluegreenhat pushed 031 commit to 03nsalias-spaces-to-underscores [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/commit/69a8ed8e7710 [14:22:34] [02ManageWiki] 07redbluegreenhat 0369a8ed8 - make sure namespace aliases do not have spaces [14:22:37] [02ManageWiki] 07redbluegreenhat created branch 03nsalias-spaces-to-underscores - 13https://github.com/miraheze/ManageWiki [14:22:46] [02ManageWiki] 07redbluegreenhat opened pull request 03#472: make sure namespace aliases do not have spaces - 13https://github.com/miraheze/ManageWiki/pull/472 [14:22:52] [02ManageWiki] 07coderabbitai[bot] commented on pull request 03#472: make sure namespace aliases do not have spaces - 13https://github.com/miraheze/ManageWiki/pull/472#issuecomment-2186704766 [14:24:23] [02ManageWiki] 07github-actions[bot] pushed 031 commit to 03nsalias-spaces-to-underscores [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/69a8ed8e7710...a9791bba4f53 [14:24:24] [02ManageWiki] 07github-actions 03a9791bb - CI: lint code to MediaWiki standards [14:24:27] [02ManageWiki] 07github-actions[bot] synchronize pull request 03#472: make sure namespace aliases do not have spaces - 13https://github.com/miraheze/ManageWiki/pull/472 [14:27:30] miraheze/ManageWiki - redbluegreenhat the build passed. [14:27:46] [02ManageWiki] 07redbluegreenhat closed pull request 03#472: make sure namespace aliases do not have spaces - 13https://github.com/miraheze/ManageWiki/pull/472 [14:27:47] [02ManageWiki] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/90cd37c2510e...f6b759726d5c [14:27:49] [02ManageWiki] 07redbluegreenhat 03f6b7597 - make sure namespace aliases do not have spaces (#472) [14:28:01] !log [alex@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'ManageWiki'} to all [14:28:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:28:13] !log [alex@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'ManageWiki'} to all - SUCCESS in 11s [14:28:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:28:26] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/resetWikiCaches.php --wiki=metawiki (END - exit=0) [14:28:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:32:11] miraheze/ManageWiki - redbluegreenhat the build passed. [14:32:43] !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'ManageWiki'} to test151 [14:32:44] !log [alex@test151] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'ManageWiki'} to test151 - SUCCESS in 1s [14:32:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:32:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:34:09] !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php extensions/MirahezeMagic/resetWikiCaches --wiki=metawikibeta (END - exit=256) [14:34:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:34:23] !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/MirahezeMagic/maintenance/resetWikiCaches.php --wiki=metawikibeta (END - exit=0) [14:34:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:36:30] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [14:41:23] PROBLEM - wiki.myehs.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.myehs.eu' expires in 15 day(s) (Wed 10 Jul 2024 02:13:06 PM GMT +0000). [14:41:34] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bc2ac3c09802...83a660c9efa5 [14:41:36] [02ssl] 07WikiTideSSLBot 0383a660c - Bot: Update SSL cert for wiki.myehs.eu [14:42:34] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.001419633627 secs [14:46:14] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:49:05] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [14:53:05] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.001455038786 secs [15:10:15] RECOVERY - wiki.myehs.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.myehs.eu' will expire on Sun 22 Sep 2024 01:41:27 PM GMT +0000. [15:12:39] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [15:14:34] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.001199841499 secs [15:15:27] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:18:55] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:20:54] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0004213154316 secs [15:47:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:54:55] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:56:59] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [16:01:09] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.001208096743 secs [16:48:12] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:50:13] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.001440435648 secs [17:25:21] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:27:15] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.000110834837 secs [17:36:48] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/83a660c9efa5...dfcd3efc444a [17:36:49] [02ssl] 07WikiTideSSLBot 03dfcd3ef - Bot: Add SSL cert for wiki.betaonline.org [17:39:32] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/dfcd3efc444a...a8a20d30b461 [17:39:34] [02ssl] 07WikiTideSSLBot 03a8a20d3 - Bot: Add SSL cert for wiki.amplicity.fr [17:40:26] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/a8a20d30b461...1d1b34b0e1db [17:40:29] [02ssl] 07WikiTideSSLBot 031d1b34b - Bot: Add SSL cert for wiki.giantesselsie.net [17:41:33] [02dns] 07redbluegreenhat pushed 031 commit to 03dccomicswiki.com [+1/-0/±0] 13https://github.com/miraheze/dns/commit/d46ffe2a2b67 [17:41:34] [02dns] 07redbluegreenhat 03d46ffe2 - add dccomicswiki.com zone [17:41:35] [02dns] 07redbluegreenhat created branch 03dccomicswiki.com - 13https://github.com/miraheze/dns [17:41:43] [02dns] 07redbluegreenhat opened pull request 03#530: add dccomicswiki.com zone - 13https://github.com/miraheze/dns/pull/530 [17:42:25] [02dns] 07redbluegreenhat closed pull request 03#530: add dccomicswiki.com zone - 13https://github.com/miraheze/dns/pull/530 [17:42:28] [02dns] 07redbluegreenhat deleted branch 03dccomicswiki.com [17:42:30] [02dns] 07redbluegreenhat pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/32acc1615ba1...5706d4ce14b5 [17:42:33] [02dns] 07redbluegreenhat 035706d4c - add dccomicswiki.com zone (#530) [17:42:34] [02dns] 07redbluegreenhat deleted branch 03dccomicswiki.com - 13https://github.com/miraheze/dns [17:44:37] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/1d1b34b0e1db...a299f22ae4e9 [17:44:39] [02ssl] 07WikiTideSSLBot 03a299f22 - Bot: Add SSL cert for maxcapacity.wiki [17:45:34] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:49:05] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:50:36] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/a299f22ae4e9...d51bf1d64270 [17:50:39] [02ssl] 07WikiTideSSLBot 03d51bf1d - Bot: Add SSL cert for www.dccomicswiki.com [17:51:13] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:53:21] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [17:57:37] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset 0.001634478569 secs [18:04:12] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:06:11] PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:06:14] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [18:06:50] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:08:29] PROBLEM - cp27 Puppet on cp27 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:08:55] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:09:49] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:09:58] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:10:34] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:16:21] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:16:50] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:17:43] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:18:08] [02mw-config] 07songnguxyz synchronize pull request 03#5584: Update footer icon settings for lhmnwiki - 13https://github.com/miraheze/mw-config/pull/5584 [18:19:11] miraheze/mw-config - songnguxyz the build passed. [18:19:33] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:22:09] PROBLEM - cp36 Puppet on cp36 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:24:57] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:25:24] RECOVERY - cp27 Puppet on cp27 is OK: OK: Puppet is currently enabled, last run 46 seconds ago with 0 failures [18:26:01] RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:26:05] running puppet on cp* and mw* via salt rn [18:26:36] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:26:50] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:27:05] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [18:27:28] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 26 seconds ago with 0 failures [18:27:29] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 27 seconds ago with 0 failures [18:27:36] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 34 seconds ago with 0 failures [18:27:49] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 22 seconds ago with 0 failures [18:28:28] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:28:33] PROBLEM - cp37 Puppet on cp37 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:28:37] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:30:32] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:30:33] RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:32:27] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [18:33:30] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/resetWikiCaches.php --wiki=dccomicswiki (END - exit=0) [18:33:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:42:11] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o ] [18:46:29] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:48:29] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0005723237991 secs [18:56:14] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:13:32] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [19:47:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:17:27] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_dns] [20:46:00] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [22:21:13] PROBLEM - cloud15 IPMI Sensors on cloud15 is WARNING: IPMI Status: Warning [1 system event log (SEL) entry present] [22:38:39] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw161 [22:40:01] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw161 [22:40:39] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [22:41:57] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [23:04:35] PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [23:16:17] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [23:30:35] RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 53 seconds ago with 0 failures [23:47:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1