[01:02:22] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:43:20] [02RequestSSL] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.28.2 [+0/-0/±2] 13https://github.com/miraheze/RequestSSL/commit/f5520681f272 [01:43:22] [02RequestSSL] 07dependabot[bot] 03f552068 - Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 [01:43:25] [02RequestSSL] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.28.2 - 13https://github.com/miraheze/RequestSSL [01:43:28] [02RequestSSL] 07dependabot[bot] opened pull request 03#52: Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 - 13https://github.com/miraheze/RequestSSL/pull/52 [01:43:29] [02RequestSSL] 07dependabot[bot] labeled pull request 03#52: Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 - 13https://github.com/miraheze/RequestSSL/pull/52 [01:43:32] [02RequestSSL] 07dependabot[bot] labeled pull request 03#52: Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 - 13https://github.com/miraheze/RequestSSL/pull/52 [01:43:35] [02RequestSSL] 07dependabot[bot] closed pull request 03#50: Bump eslint-config-wikimedia from 0.27.0 to 0.28.1 - 13https://github.com/miraheze/RequestSSL/pull/50 [01:43:38] [02RequestSSL] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.28.1 [01:43:39] [02RequestSSL] 07dependabot[bot] commented on pull request 03#50: Bump eslint-config-wikimedia from 0.27.0 to 0.28.1 - 13https://github.com/miraheze/RequestSSL/pull/50#issuecomment-2164195124 [01:43:42] [02RequestSSL] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.28.1 - 13https://github.com/miraheze/RequestSSL [01:44:44] [02RequestSSL] 07coderabbitai[bot] commented on pull request 03#52: Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 - 13https://github.com/miraheze/RequestSSL/pull/52#issuecomment-2164196216 [01:51:53] miraheze/RequestSSL - dependabot[bot] the build passed. [02:00:12] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:31:10] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [03:58:04] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [05:19:35] [1/2] https://issue-tracker.miraheze.org/T12186 [05:19:35] [2/2] Can anyone handle this request? Approved by OA, but needs to be added by someone with puppet access. [05:57:05] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [06:40:26] [02ssl] 07redbluegreenhat closed pull request 03#782: Remove pastport.org - 13https://github.com/miraheze/ssl/pull/782 [06:40:29] [02ssl] 07redbluegreenhat pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/42693466d152...47b5cca140a1 [06:40:31] [02ssl] 07redbluegreenhat 0347b5cca - Remove pastport.org (#782) [06:40:32] [02ssl] 07redbluegreenhat deleted branch 03remove-pastport [06:40:34] [02ssl] 07redbluegreenhat deleted branch 03remove-pastport - 13https://github.com/miraheze/ssl [06:40:51] [02dns] 07redbluegreenhat closed pull request 03#525: Remove pastport.org - 13https://github.com/miraheze/dns/pull/525 [06:40:54] [02dns] 07redbluegreenhat deleted branch 03remove-pastport [06:40:55] [02dns] 07redbluegreenhat pushed 031 commit to 03master [+0/-1/±0] 13https://github.com/miraheze/dns/compare/680592302fc2...b43479ff30f9 [06:40:56] [02dns] 07redbluegreenhat 03b43479f - Remove pastport.org (#525) [06:40:59] [02dns] 07redbluegreenhat deleted branch 03remove-pastport - 13https://github.com/miraheze/dns [06:41:15] [02ssl] 07redbluegreenhat closed pull request 03#780: drop pokemonwiki.info - 13https://github.com/miraheze/ssl/pull/780 [06:41:18] [02ssl] 07redbluegreenhat deleted branch 03drop-pokemonwiki.info [06:41:21] [02ssl] 07redbluegreenhat pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/47b5cca140a1...1c6d132bb321 [06:41:23] [02ssl] 07redbluegreenhat 031c6d132 - drop pokemonwiki.info (#780) [06:41:25] [02ssl] 07redbluegreenhat deleted branch 03drop-pokemonwiki.info - 13https://github.com/miraheze/ssl [06:53:28] [02ssl] 07redbluegreenhat pushed 031 commit to 03remove-osdevwiki [+0/-1/±1] 13https://github.com/miraheze/ssl/commit/ec6c92ad0698 [06:53:31] [02ssl] 07redbluegreenhat 03ec6c92a - remove osdev.wiki [06:53:34] [02ssl] 07redbluegreenhat created branch 03remove-osdevwiki - 13https://github.com/miraheze/ssl [06:53:36] [02ssl] 07redbluegreenhat opened pull request 03#783: remove osdev.wiki - 13https://github.com/miraheze/ssl/pull/783 [06:53:43] [02ssl] 07coderabbitai[bot] commented on pull request 03#783: remove osdev.wiki - 13https://github.com/miraheze/ssl/pull/783#issuecomment-2164689003 [06:56:49] [02ssl] 07redbluegreenhat closed pull request 03#783: remove osdev.wiki - 13https://github.com/miraheze/ssl/pull/783 [06:56:50] [02ssl] 07redbluegreenhat deleted branch 03remove-osdevwiki [06:56:51] [02ssl] 07redbluegreenhat pushed 031 commit to 03master [+0/-1/±1] 13https://github.com/miraheze/ssl/compare/1c6d132bb321...c239b269dffe [06:56:54] [02ssl] 07redbluegreenhat 03c239b26 - remove osdev.wiki (#783) [06:56:57] [02ssl] 07redbluegreenhat deleted branch 03remove-osdevwiki - 13https://github.com/miraheze/ssl [06:57:23] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/c239b269dffe...97e5cadb9636 [06:57:24] [02ssl] 07WikiTideSSLBot 0397e5cad - Bot: Add SSL cert for osdev.wiki [06:57:50] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#783: remove osdev.wiki - 13https://github.com/miraheze/ssl/pull/783#issuecomment-2164689003 [07:04:35] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:06:08] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:06:18] PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:08:18] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:09:29] PROBLEM - cp27 Puppet on cp27 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:11:08] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:11:15] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:11:29] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:18:21] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:19:46] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:21:31] PROBLEM - cp36 Puppet on cp36 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:24:03] PROBLEM - cp26 Puppet on cp26 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:25:45] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:27:00] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:28:15] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:30:10] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:30:27] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 40 seconds ago with 0 failures [07:32:33] PROBLEM - cp37 Puppet on cp37 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [07:34:29] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:35:12] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 125.18 ms [07:35:57] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:36:13] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:37:13] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 123.91 ms [07:37:39] RECOVERY - cp27 Puppet on cp27 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:38:59] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:39:24] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:40:12] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:46:30] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:47:57] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:50:50] RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:53:27] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [07:53:31] RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:55:11] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 57 seconds ago with 0 failures [07:56:09] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [07:57:52] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 9 seconds ago with 0 failures [08:00:34] RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:21:29] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [09:21:04] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [09:32:25] PROBLEM - privacywiki.xyz - LetsEncrypt on sslhost is CRITICAL: No address associated with hostnameHTTP CRITICAL - Unable to open TCP socket [09:49:51] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [10:07:15] PROBLEM - privacywiki.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for privacywiki.xyz could not be found [10:20:44] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [10:36:20] PROBLEM - privacywiki.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - privacywiki.xyz All nameservers failed to answer the query. [11:05:24] PROBLEM - privacywiki.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for privacywiki.xyz could not be found [11:18:10] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [12:17:38] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [13:15:05] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [13:54:26] [02ImportDump] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.28.2 [+0/-0/±2] 13https://github.com/miraheze/ImportDump/commit/9f490885146a [13:54:28] [02ImportDump] 07dependabot[bot] 039f49088 - Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 [13:54:31] [02ImportDump] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.28.2 - 13https://github.com/miraheze/ImportDump [13:54:34] [02ImportDump] 07dependabot[bot] labeled pull request 03#115: Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 - 13https://github.com/miraheze/ImportDump/pull/115 [13:54:38] [02ImportDump] 07dependabot[bot] labeled pull request 03#115: Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 - 13https://github.com/miraheze/ImportDump/pull/115 [13:54:38] [02ImportDump] 07dependabot[bot] opened pull request 03#115: Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 - 13https://github.com/miraheze/ImportDump/pull/115 [13:54:39] [02ImportDump] 07dependabot[bot] commented on pull request 03#114: Bump eslint-config-wikimedia from 0.27.0 to 0.28.1 - 13https://github.com/miraheze/ImportDump/pull/114#issuecomment-2165746861 [13:54:43] [02ImportDump] 07dependabot[bot] closed pull request 03#114: Bump eslint-config-wikimedia from 0.27.0 to 0.28.1 - 13https://github.com/miraheze/ImportDump/pull/114 [13:54:45] [02ImportDump] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.28.1 - 13https://github.com/miraheze/ImportDump [13:54:48] [02ImportDump] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.28.1 [13:54:51] [02ImportDump] 07coderabbitai[bot] commented on pull request 03#115: Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 - 13https://github.com/miraheze/ImportDump/pull/115#issuecomment-2165747095 [14:03:15] miraheze/ImportDump - dependabot[bot] the build passed. [15:12:04] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [15:44:59] !log actually whitelist bast1[68]1 IPs in cf console [15:45:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:47:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:07:16] [02MatomoAnalytics] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.28.2 [+0/-0/±2] 13https://github.com/miraheze/MatomoAnalytics/commit/82e76e5b50f6 [16:07:17] [02MatomoAnalytics] 07dependabot[bot] 0382e76e5 - Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 [16:07:20] [02MatomoAnalytics] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.28.2 - 13https://github.com/miraheze/MatomoAnalytics [16:07:21] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#134: Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 - 13https://github.com/miraheze/MatomoAnalytics/pull/134 [16:07:23] [02MatomoAnalytics] 07dependabot[bot] opened pull request 03#134: Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 - 13https://github.com/miraheze/MatomoAnalytics/pull/134 [16:07:25] [02MatomoAnalytics] 07dependabot[bot] labeled pull request 03#134: Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 - 13https://github.com/miraheze/MatomoAnalytics/pull/134 [16:07:26] [02MatomoAnalytics] 07dependabot[bot] commented on pull request 03#132: Bump eslint-config-wikimedia from 0.27.0 to 0.28.1 - 13https://github.com/miraheze/MatomoAnalytics/pull/132#issuecomment-2166114116 [16:07:28] [02MatomoAnalytics] 07dependabot[bot] closed pull request 03#132: Bump eslint-config-wikimedia from 0.27.0 to 0.28.1 - 13https://github.com/miraheze/MatomoAnalytics/pull/132 [16:07:31] [02MatomoAnalytics] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.28.1 [16:07:33] [02MatomoAnalytics] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.28.1 - 13https://github.com/miraheze/MatomoAnalytics [16:07:34] [02MatomoAnalytics] 07coderabbitai[bot] commented on pull request 03#134: Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 - 13https://github.com/miraheze/MatomoAnalytics/pull/134#issuecomment-2166114283 [16:09:33] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [16:11:54] miraheze/MatomoAnalytics - dependabot[bot] the build passed. [16:19:12] [02mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/c9897b65afd2...f038816ce1f8 [16:19:13] [02mw-config] 07MacFan4000 03f038816 - enable MirahezeSurvey on staffwiki [16:20:14] miraheze/mw-config - MacFan4000 the build passed. [16:21:31] !log [macfan@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [16:21:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:21:42] !log [macfan@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 11s [16:21:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:23:07] !log [@test151] starting deploy of {'config': True} to test151 [16:23:08] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [16:23:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:23:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:30:35] PROBLEM - privacywiki.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - privacywiki.xyz All nameservers failed to answer the query. [16:37:46] !log [@mwtask171] starting deploy of {'config': True} to all [16:37:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:55] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 9s [16:38:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:00:35] PROBLEM - privacywiki.xyz - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for privacywiki.xyz could not be found [17:05:47] [02ManageWiki] 07dependabot[bot] pushed 031 commit to 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.28.2 [+0/-0/±2] 13https://github.com/miraheze/ManageWiki/commit/523183ae570d [17:05:49] [02ManageWiki] 07dependabot[bot] 03523183a - Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 [17:05:50] [02ManageWiki] 07dependabot[bot] created branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.28.2 - 13https://github.com/miraheze/ManageWiki [17:05:51] [02ManageWiki] 07dependabot[bot] opened pull request 03#468: Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 - 13https://github.com/miraheze/ManageWiki/pull/468 [17:05:52] [02ManageWiki] 07dependabot[bot] labeled pull request 03#468: Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 - 13https://github.com/miraheze/ManageWiki/pull/468 [17:05:54] [02ManageWiki] 07dependabot[bot] labeled pull request 03#468: Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 - 13https://github.com/miraheze/ManageWiki/pull/468 [17:05:56] [02ManageWiki] 07dependabot[bot] closed pull request 03#467: Bump eslint-config-wikimedia from 0.27.0 to 0.28.1 - 13https://github.com/miraheze/ManageWiki/pull/467 [17:05:57] [02ManageWiki] 07dependabot[bot] commented on pull request 03#467: Bump eslint-config-wikimedia from 0.27.0 to 0.28.1 - 13https://github.com/miraheze/ManageWiki/pull/467#issuecomment-2166313623 [17:05:58] [02ManageWiki] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.28.1 [17:06:00] [02ManageWiki] 07dependabot[bot] deleted branch 03dependabot/npm_and_yarn/eslint-config-wikimedia-0.28.1 - 13https://github.com/miraheze/ManageWiki [17:06:01] [02ManageWiki] 07coderabbitai[bot] commented on pull request 03#468: Bump eslint-config-wikimedia from 0.27.0 to 0.28.2 - 13https://github.com/miraheze/ManageWiki/pull/468#issuecomment-2166313926 [17:10:47] miraheze/ManageWiki - dependabot[bot] the build passed. [17:53:53] PROBLEM - wiki.strangereons.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.strangereons.com' expires in 15 day(s) (Sat 29 Jun 2024 05:36:45 PM GMT +0000). [17:54:05] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/97e5cadb9636...a7a071e2dece [17:54:07] [02ssl] 07WikiTideSSLBot 03a7a071e - Bot: Update SSL cert for wiki.strangereons.com [17:58:55] [02mw-config] 07Reception123 closed pull request 03#5585: Change a couple of wiki names - 13https://github.com/miraheze/mw-config/pull/5585 [17:58:58] [02mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f038816ce1f8...ee55e89a9cd2 [17:58:59] [02mw-config] 07robkam 03ee55e89 - Change a couple of wiki names (#5585) [17:59:55] miraheze/mw-config - Reception123 the build passed. [18:02:00] !log [@mwtask181] starting deploy of {'config': True} to all [18:02:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:02:10] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s [18:02:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:02:28] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:04:16] PROBLEM - polcompball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'polcompball.wiki' expires in 15 day(s) (Sat 29 Jun 2024 06:01:42 PM GMT +0000). [18:04:31] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a7a071e2dece...a5f84813f39f [18:04:34] [02ssl] 07WikiTideSSLBot 03a5f8481 - Bot: Update SSL cert for polcompball.wiki [18:05:19] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:06:05] PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:07:04] !log [@mwtask171] starting deploy of {'config': True} to all [18:07:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:07:13] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 9s [18:07:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:07:56] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:09:35] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [18:09:35] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:10:11] PROBLEM - cp27 Puppet on cp27 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:10:37] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:11:57] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:17:51] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:19:58] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:22:50] PROBLEM - cp36 Puppet on cp36 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:23:05] PROBLEM - cities.simulz.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'cities.simulz.kr' expires in 15 day(s) (Sat 29 Jun 2024 06:04:01 PM GMT +0000). [18:23:23] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a5f84813f39f...693ebe876d6a [18:23:25] [02ssl] 07WikiTideSSLBot 03693ebe8 - Bot: Update SSL cert for cities.simulz.kr [18:24:04] !log [@test151] starting deploy of {'config': True} to test151 [18:24:05] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [18:24:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:24:12] PROBLEM - cp26 Puppet on cp26 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:24:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:41] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:27:00] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:27:19] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:31:07] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:32:37] PROBLEM - cp37 Puppet on cp37 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [18:48:50] PROBLEM - farthestfrontier.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'farthestfrontier.wiki' expires in 15 day(s) (Sat 29 Jun 2024 06:37:25 PM GMT +0000). [18:49:07] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/693ebe876d6a...677728b3dca8 [18:49:09] [02ssl] 07WikiTideSSLBot 03677728b - Bot: Update SSL cert for farthestfrontier.wiki [19:00:38] RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:04:47] PROBLEM - alpha.sagan4.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'alpha.sagan4.org' expires in 15 day(s) (Sat 29 Jun 2024 06:39:18 PM GMT +0000). [19:05:01] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/677728b3dca8...91d2046d0354 [19:05:02] [02ssl] 07WikiTideSSLBot 0391d2046 - Bot: Update SSL cert for alpha.sagan4.org [19:32:38] PROBLEM - cp37 Puppet on cp37 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [19:36:12] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [19:46:19] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:47:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:16:42] RECOVERY - mw172 Puppet on mw172 is OK: OK: Puppet is currently enabled, last run 42 seconds ago with 0 failures [20:18:30] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [20:20:50] RECOVERY - cp36 Puppet on cp36 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:24:10] RECOVERY - cp26 Puppet on cp26 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:24:41] RECOVERY - mw152 Puppet on mw152 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:25:00] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:26:06] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:29:05] RECOVERY - mw182 Puppet on mw182 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:30:27] RECOVERY - cp51 Puppet on cp51 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [20:30:39] RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:34:01] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:35:56] RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:36:05] RECOVERY - mw151 Puppet on mw151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:37:05] RECOVERY - cp27 Puppet on cp27 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [20:37:35] RECOVERY - mw162 Puppet on mw162 is OK: OK: Puppet is currently enabled, last run 23 seconds ago with 0 failures [20:37:41] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [20:39:03] RECOVERY - mw161 Puppet on mw161 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [20:46:39] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures [21:47:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [23:08:54] PROBLEM - richterian.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'richterian.com' expires in 15 day(s) (Sat 29 Jun 2024 10:54:46 PM GMT +0000). [23:09:06] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/91d2046d0354...e228b85f3bb1 [23:09:07] [02ssl] 07WikiTideSSLBot 03e228b85 - Bot: Update SSL cert for richterian.com [23:18:25] PROBLEM - www.clinitheque.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.clinitheque.fr' expires in 15 day(s) (Sat 29 Jun 2024 10:46:52 PM GMT +0000). [23:18:37] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e228b85f3bb1...41fad4cb696e [23:18:38] [02ssl] 07WikiTideSSLBot 0341fad4c - Bot: Update SSL cert for www.clinitheque.fr [23:24:12] PROBLEM - cp26 Puppet on cp26 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:26:41] PROBLEM - mw152 Puppet on mw152 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:27:00] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:27:18] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:31:05] PROBLEM - mw182 Puppet on mw182 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:31:26] PROBLEM - coffeewiki.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'coffeewiki.net' expires in 15 day(s) (Sat 29 Jun 2024 11:08:56 PM GMT +0000). [23:31:37] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/41fad4cb696e...53bd8499aaad [23:31:38] [02ssl] 07WikiTideSSLBot 0353bd849 - Bot: Update SSL cert for coffeewiki.net [23:32:39] PROBLEM - cp37 Puppet on cp37 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:34:28] PROBLEM - cp51 Puppet on cp51 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:36:50] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:37:56] PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:38:05] PROBLEM - mw151 Puppet on mw151 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:39:35] PROBLEM - mw162 Puppet on mw162 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:40:21] PROBLEM - cp27 Puppet on cp27 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 4 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:40:59] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:42:22] PROBLEM - mw161 Puppet on mw161 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:47:37] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:50:05] PROBLEM - mw172 Puppet on mw172 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] [23:52:50] PROBLEM - cp36 Puppet on cp36 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx]