[00:00:38] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:00:42] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:02:07] !log [@mwtask181] starting deploy of {'config': True} to all [00:02:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:02:17] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [00:02:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:02:38] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:02:42] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 50% [00:04:08] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 20390MiB (23% inode=98%); [00:05:14] RECOVERY - cp41 Disk Space on cp41 is OK: DISK OK - free space: / 19783MiB (21% inode=98%); [00:05:38] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 20974MiB (23% inode=98%); [00:10:42] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 63% [05:31:31] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [05:47:06] PROBLEM - ping on changeprop151 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 369.36 ms [05:49:06] RECOVERY - ping on changeprop151 is OK: PING OK - Packet loss = 0%, RTA = 0.23 ms [05:57:17] PROBLEM - ping on changeprop151 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 216.38 ms [05:58:20] [02miraheze/CreateWiki] 07Reception123 pushed 032 commits to 03Reception123-patch-1 [+0/-0/±3] 13https://github.com/miraheze/CreateWiki/compare/da4bbf1152b1...277473f3f2a5 [05:58:23] [02miraheze/CreateWiki] 07Reception123 030b358dd - add extra [05:58:26] [02miraheze/CreateWiki] 07Reception123 03277473f - Merge branch 'Reception123-patch-1' of https://github.com/miraheze/createwiki into Reception123-patch-1 [05:58:29] [02CreateWiki] 07Reception123 synchronize pull request 03#500: add more details needed status - 13https://github.com/miraheze/CreateWiki/pull/500 [05:59:17] RECOVERY - ping on changeprop151 is OK: PING OK - Packet loss = 0%, RTA = 0.23 ms [06:00:25] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [06:06:27] miraheze/CreateWiki - Reception123 the build passed. [06:11:23] PROBLEM - ping on changeprop151 is WARNING: PING WARNING - Packet loss = 0%, RTA = 111.73 ms [06:13:23] RECOVERY - ping on changeprop151 is OK: PING OK - Packet loss = 0%, RTA = 0.24 ms [06:16:21] [02miraheze/ManageWiki] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/commit/11e9d3b5a378 [06:16:24] [02miraheze/ManageWiki] 07The-Voidwalker 0311e9d3b - Improve handling of overridedefault for MWN [06:16:26] [02ManageWiki] 07The-Voidwalker created branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/ManageWiki [06:16:28] [02ManageWiki] 07The-Voidwalker opened pull request 03#457: Improve handling of overridedefault for MWN - 13https://github.com/miraheze/ManageWiki/pull/457 [06:18:14] [02miraheze/ManageWiki] 07github-actions[bot] pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/11e9d3b5a378...fd7180d28440 [06:18:17] [02miraheze/ManageWiki] 07github-actions 03fd7180d - CI: lint code to MediaWiki standards [06:18:20] [02ManageWiki] 07github-actions[bot] synchronize pull request 03#457: Improve handling of overridedefault for MWN - 13https://github.com/miraheze/ManageWiki/pull/457 [06:19:43] miraheze/ManageWiki - The-Voidwalker the build has errored. [06:24:17] [02miraheze/ManageWiki] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/fd7180d28440...c764fc561997 [06:24:19] [02miraheze/ManageWiki] 07The-Voidwalker 03c764fc5 - fix function signature [06:24:21] [02ManageWiki] 07The-Voidwalker synchronize pull request 03#457: Improve handling of overridedefault for MWN - 13https://github.com/miraheze/ManageWiki/pull/457 [06:26:46] miraheze/ManageWiki - The-Voidwalker the build has errored. [06:28:29] [02miraheze/ManageWiki] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/c764fc561997...30ac6ca70901 [06:28:31] [02miraheze/ManageWiki] 07The-Voidwalker 0330ac6ca - Update Hooks.php [06:28:33] [02ManageWiki] 07The-Voidwalker synchronize pull request 03#457: Improve handling of overridedefault for MWN - 13https://github.com/miraheze/ManageWiki/pull/457 [06:31:38] miraheze/ManageWiki - The-Voidwalker the build has errored. [06:32:51] [02miraheze/ManageWiki] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/30ac6ca70901...dd34fbdd2337 [06:32:54] [02miraheze/ManageWiki] 07The-Voidwalker 03dd34fbd - Update Hooks.php [06:32:55] [02ManageWiki] 07The-Voidwalker synchronize pull request 03#457: Improve handling of overridedefault for MWN - 13https://github.com/miraheze/ManageWiki/pull/457 [06:36:40] miraheze/ManageWiki - The-Voidwalker the build has errored. [06:37:46] [02miraheze/ManageWiki] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/dd34fbdd2337...5cc626dd91fb [06:37:49] [02miraheze/ManageWiki] 07The-Voidwalker 035cc626d - fix linter rule on comment [06:37:51] [02ManageWiki] 07The-Voidwalker synchronize pull request 03#457: Improve handling of overridedefault for MWN - 13https://github.com/miraheze/ManageWiki/pull/457 [06:42:05] miraheze/ManageWiki - The-Voidwalker the build passed. [09:00:27] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.79 ms [09:08:38] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.38 ms [09:54:12] PROBLEM - wikigenius.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wikigenius.org could not be found [10:47:42] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libboost-regex1.74.0-icu72] [11:15:40] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 33 seconds ago with 0 failures [11:54:11] PROBLEM - cp26 APT on cp26 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [11:54:12] PROBLEM - cp51 APT on cp51 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [11:54:54] PROBLEM - bots171 APT on bots171 is CRITICAL: APT CRITICAL: 51 packages available for upgrade (14 critical updates). [11:55:04] PROBLEM - ns1 APT on ns1 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [11:55:41] PROBLEM - cp37 APT on cp37 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [11:55:45] PROBLEM - puppet181 APT on puppet181 is CRITICAL: APT CRITICAL: 47 packages available for upgrade (14 critical updates). [11:56:11] PROBLEM - mw181 APT on mw181 is CRITICAL: APT CRITICAL: 45 packages available for upgrade (17 critical updates). [11:56:11] PROBLEM - swiftproxy161 APT on swiftproxy161 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (14 critical updates). [11:56:12] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 45 packages available for upgrade (17 critical updates). [11:56:23] PROBLEM - reports171 APT on reports171 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (14 critical updates). [11:56:35] PROBLEM - os162 APT on os162 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (14 critical updates). [11:56:53] PROBLEM - mw152 APT on mw152 is CRITICAL: APT CRITICAL: 45 packages available for upgrade (17 critical updates). [11:57:06] PROBLEM - matomo151 APT on matomo151 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [11:57:12] PROBLEM - cloud16 APT on cloud16 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (13 critical updates). [11:57:16] PROBLEM - db161 APT on db161 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (14 critical updates). [11:57:16] PROBLEM - swiftobject161 APT on swiftobject161 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [11:57:35] PROBLEM - mem151 APT on mem151 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [11:57:48] PROBLEM - os161 APT on os161 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (14 critical updates). [11:58:03] PROBLEM - swiftac171 APT on swiftac171 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (14 critical updates). [11:58:45] PROBLEM - cp27 APT on cp27 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [11:58:50] PROBLEM - graphite151 APT on graphite151 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (14 critical updates). [11:59:02] PROBLEM - db171 APT on db171 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (14 critical updates). [11:59:18] PROBLEM - bast181 APT on bast181 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [11:59:20] PROBLEM - mw182 APT on mw182 is CRITICAL: APT CRITICAL: 45 packages available for upgrade (17 critical updates). [11:59:25] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [11:59:43] PROBLEM - cloud18 APT on cloud18 is CRITICAL: APT CRITICAL: 71 packages available for upgrade (13 critical updates). [11:59:53] PROBLEM - swiftobject181 APT on swiftobject181 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [12:00:48] PROBLEM - rdb151 APT on rdb151 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [12:01:07] PROBLEM - bast161 APT on bast161 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [12:01:11] PROBLEM - db182 APT on db182 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (14 critical updates). [12:01:17] PROBLEM - ldap171 APT on ldap171 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [12:01:31] PROBLEM - swiftobject151 APT on swiftobject151 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [12:01:46] PROBLEM - mem161 APT on mem161 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [12:02:00] PROBLEM - graylog161 APT on graylog161 is CRITICAL: APT CRITICAL: 45 packages available for upgrade (14 critical updates). [12:02:17] PROBLEM - eventgate181 APT on eventgate181 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (14 critical updates). [12:02:37] PROBLEM - changeprop151 APT on changeprop151 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (14 critical updates). [12:02:41] PROBLEM - db181 APT on db181 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (14 critical updates). [12:02:42] PROBLEM - cp36 APT on cp36 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [12:03:29] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 17 packages available for upgrade (17 critical updates). [12:03:53] PROBLEM - os151 APT on os151 is CRITICAL: APT CRITICAL: 43 packages available for upgrade (14 critical updates). [12:04:04] PROBLEM - db151 APT on db151 is CRITICAL: APT CRITICAL: 48 packages available for upgrade (14 critical updates). [12:04:11] PROBLEM - phorge171 APT on phorge171 is CRITICAL: APT CRITICAL: 44 packages available for upgrade (14 critical updates). [12:05:14] PROBLEM - kafka181 APT on kafka181 is CRITICAL: APT CRITICAL: 14 packages available for upgrade (14 critical updates). [12:05:16] PROBLEM - cloud17 APT on cloud17 is CRITICAL: APT CRITICAL: 69 packages available for upgrade (13 critical updates). [12:05:18] PROBLEM - cp41 APT on cp41 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [12:05:50] PROBLEM - swiftproxy171 APT on swiftproxy171 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [12:05:54] PROBLEM - swiftobject171 APT on swiftobject171 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (14 critical updates). [12:06:00] PROBLEM - mw162 APT on mw162 is CRITICAL: APT CRITICAL: 45 packages available for upgrade (17 critical updates). [12:06:10] PROBLEM - mon181 APT on mon181 is CRITICAL: APT CRITICAL: 46 packages available for upgrade (14 critical updates). [12:06:11] PROBLEM - cloud15 APT on cloud15 is CRITICAL: APT CRITICAL: 72 packages available for upgrade (13 critical updates). [12:06:16] PROBLEM - mw161 APT on mw161 is CRITICAL: APT CRITICAL: 45 packages available for upgrade (17 critical updates). [12:06:56] PROBLEM - mw171 APT on mw171 is CRITICAL: APT CRITICAL: 45 packages available for upgrade (17 critical updates). [12:06:58] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 18 packages available for upgrade (17 critical updates). [12:07:35] PROBLEM - mw172 APT on mw172 is CRITICAL: APT CRITICAL: 45 packages available for upgrade (17 critical updates). [12:07:41] PROBLEM - mw151 APT on mw151 is CRITICAL: APT CRITICAL: 45 packages available for upgrade (17 critical updates). [12:08:26] PROBLEM - ns2 APT on ns2 is CRITICAL: APT CRITICAL: 42 packages available for upgrade (13 critical updates). [13:29:09] PROBLEM - franchise.franchising.org.ua - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - franchise.franchising.org.ua All nameservers failed to answer the query. [13:58:49] RECOVERY - franchise.franchising.org.ua - reverse DNS on sslhost is OK: SSL OK - franchise.franchising.org.ua reverse DNS resolves to cp36.wikitide.net - CNAME OK [14:33:58] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.91 ms [14:38:03] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 207.28 ms [14:48:31] PROBLEM - www.pomologia.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'www.pomologia.org' expires in 7 day(s) (Mon 08 Apr 2024 02:38:30 PM GMT +0000). [15:45:32] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.69 ms [15:47:34] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 174.67 ms [15:49:05] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query line.pm. IN NS: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:12:11] PROBLEM - en.clockup.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'en.clockup.wiki' expires in 7 day(s) (Mon 08 Apr 2024 03:58:44 PM GMT +0000). [16:27:14] PROBLEM - cp41 Disk Space on cp41 is WARNING: DISK WARNING - free space: / 10234MiB (10% inode=98%); [16:47:55] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.32 ms [16:49:57] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 184.66 ms [16:59:02] PROBLEM - ping on changeprop151 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 340.02 ms [17:01:02] RECOVERY - ping on changeprop151 is OK: PING OK - Packet loss = 0%, RTA = 0.26 ms [17:40:17] [02CreateWiki] 07redbluegreenhat reviewed pull request 03#500 commit - 13https://github.com/miraheze/CreateWiki/pull/500#discussion_r1545750404 [17:42:39] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d4af3fd7394c...3c1f74a18b85 [17:42:41] [02miraheze/puppet] 07Universal-Omega 033c1f74a - graylog: use opensearch.wikitide.net again [18:04:05] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3c1f74a18b85...b7881c2fd381 [18:04:07] [02miraheze/puppet] 07Universal-Omega 03b7881c2 - Update [18:16:59] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [18:20:52] [02dns] 07Universal-Omega created branch 03remove-aliases - 13https://github.com/miraheze/dns [18:20:54] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03remove-aliases [+0/-0/±1] 13https://github.com/miraheze/dns/commit/8b10e4bed3d9 [18:20:55] [02miraheze/dns] 07Universal-Omega 038b10e4b - miraheze.org: remove unused server aliases [18:20:57] [02dns] 07Universal-Omega opened pull request 03#505: miraheze.org: remove unused server aliases - 13https://github.com/miraheze/dns/pull/505 [18:24:50] [02dns] 07Universal-Omega closed pull request 03#505: miraheze.org: remove unused server aliases - 13https://github.com/miraheze/dns/pull/505 [18:24:51] [02miraheze/dns] 07Universal-Omega deleted branch 03remove-aliases [18:24:52] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/114b20bbe7f5...11c88960a2ce [18:24:54] [02miraheze/dns] 07Universal-Omega 0311c8896 - miraheze.org: remove unused server aliases (#505) [18:24:57] [02dns] 07Universal-Omega deleted branch 03remove-aliases - 13https://github.com/miraheze/dns [18:26:54] [02phorge-extensions] 07Universal-Omega closed pull request 03#47: Update php-parallel-lint/php-parallel-lint requirement from 1.3.1 to 1.4.0 - 13https://github.com/miraheze/phorge-extensions/pull/47 [18:26:56] [02miraheze/phorge-extensions] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/phorge-extensions/compare/1ba9a3621c6c...c65d9043c531 [18:26:58] [02miraheze/phorge-extensions] 07dependabot[bot] 03c65d904 - Update php-parallel-lint/php-parallel-lint requirement from 1.3.1 to 1.4.0 (#47) [18:26:59] [02miraheze/phorge-extensions] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [18:27:02] [02phorge-extensions] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/phorge-extensions [18:27:40] miraheze/phorge-extensions - Universal-Omega the build passed. [18:33:35] [02dns] 07MacFan4000 opened pull request 03#506: Switch farthestfrontier.wiki to Miraheze cps - 13https://github.com/miraheze/dns/pull/506 [18:36:35] PROBLEM - wiki.strangereons.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.strangereons.com' expires in 15 day(s) (Tue 16 Apr 2024 06:15:07 PM GMT +0000). [18:36:53] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/978c021dfa59...9e8e1e488110 [18:36:55] [02miraheze/ssl] 07WikiTideSSLBot 039e8e1e4 - Bot: Update SSL cert for wiki.strangereons.com [18:40:02] [02dns] 07Universal-Omega commented on pull request 03#506: Switch farthestfrontier.wiki to Miraheze cps - 13https://github.com/miraheze/dns/pull/506#issuecomment-2028864855 [18:42:56] [02dns] 07MacFan4000 synchronize pull request 03#506: Switch farthestfrontier.wiki to Miraheze cps - 13https://github.com/miraheze/dns/pull/506 [18:43:31] [02dns] 07Universal-Omega closed pull request 03#506: Switch farthestfrontier.wiki to Miraheze cps - 13https://github.com/miraheze/dns/pull/506 [18:43:33] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/11c88960a2ce...853327304514 [18:43:34] [02miraheze/dns] 07MacFan4000 038533273 - Switch farthestfrontier.wiki to Miraheze cps (#506) [18:50:57] [02CreateWiki] 07Reception123 synchronize pull request 03#500: add more details needed status - 13https://github.com/miraheze/CreateWiki/pull/500 [18:50:58] [02miraheze/CreateWiki] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/277473f3f2a5...2f0e65f89b0c [18:50:59] [02miraheze/CreateWiki] 07Reception123 032f0e65f - Update includes/RequestWiki/WikiRequest.php [18:58:37] miraheze/CreateWiki - Reception123 the build passed. [19:01:36] PROBLEM - polcompball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'polcompball.wiki' expires in 15 day(s) (Tue 16 Apr 2024 06:40:00 PM GMT +0000). [19:01:48] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9e8e1e488110...ff730ab9c167 [19:01:51] [02miraheze/ssl] 07WikiTideSSLBot 03ff730ab - Bot: Update SSL cert for polcompball.wiki [19:03:52] PROBLEM - cities.simulz.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'cities.simulz.kr' expires in 15 day(s) (Tue 16 Apr 2024 06:33:38 PM GMT +0000). [19:04:08] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ff730ab9c167...0d9d2c25e672 [19:04:09] [02miraheze/ssl] 07WikiTideSSLBot 030d9d2c2 - Bot: Update SSL cert for cities.simulz.kr [19:05:10] RECOVERY - wiki.strangereons.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.strangereons.com' will expire on Sat 29 Jun 2024 05:36:45 PM GMT +0000. [19:05:18] [02CreateWiki] 07Reception123 synchronize pull request 03#500: add more details needed status - 13https://github.com/miraheze/CreateWiki/pull/500 [19:05:20] [02miraheze/CreateWiki] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/2f0e65f89b0c...e37033e67321 [19:05:22] [02miraheze/CreateWiki] 07Reception123 03e37033e - Update WikiRequest.php [19:10:47] [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/70f268a3ad10...227853d0ca16 [19:10:49] [02miraheze/mw-config] 07Reception123 03227853d - new CreateWikiStateDays per modified DP [19:11:47] miraheze/mw-config - Reception123 the build passed. [19:12:17] [02miraheze/CreateWiki] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/e37033e67321...3f4f004bc9b7 [19:12:20] [02miraheze/CreateWiki] 07Reception123 033f4f004 - no moredetails icon, change [19:12:22] [02CreateWiki] 07Reception123 synchronize pull request 03#500: add more details needed status - 13https://github.com/miraheze/CreateWiki/pull/500 [19:13:23] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/1a35dfed49b6 [19:13:26] [02miraheze/puppet] 07Reception123 031a35dfe - re-enable manageInactiveWikis cron [19:13:28] [02puppet] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/puppet [19:13:29] [02puppet] 07Reception123 opened pull request 03#3813: re-enable manageInactiveWikis cron - 13https://github.com/miraheze/puppet/pull/3813 [19:20:03] miraheze/CreateWiki - Reception123 the build passed. [19:23:24] [02miraheze/CreateWiki] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/8f8442ed5299...09af630a0441 [19:23:25] [02miraheze/CreateWiki] 07Reception123 0309af630 - disable deletion for now [19:23:37] !log [@test151] starting deploy of {'config': True} to test151 [19:23:38] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [19:23:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:23:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:26:28] miraheze/CreateWiki - Reception123 the build has errored. [19:31:13] RECOVERY - polcompball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'polcompball.wiki' will expire on Sat 29 Jun 2024 06:01:42 PM GMT +0000. [19:32:42] !log [@mwtask181] starting deploy of {'config': True} to all [19:32:45] RECOVERY - cities.simulz.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'cities.simulz.kr' will expire on Sat 29 Jun 2024 06:04:01 PM GMT +0000. [19:32:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:32:52] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 9s [19:33:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:36:49] !log [@mwtask171] starting deploy of {'config': True} to all [19:36:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:36:57] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [19:37:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:37:32] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/0d9d2c25e672...658a02b2a589 [19:37:34] [02miraheze/ssl] 07WikiTideSSLBot 03658a02b - Bot: Add SSL cert for farthestfrontier.wiki [19:39:25] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/658a02b2a589...e1e39b0cbe50 [19:39:26] [02miraheze/ssl] 07WikiTideSSLBot 03e1e39b0 - Bot: Add SSL cert for alpha.sagan4.org [19:54:09] !log [salt-user@mwtask181] Renamed vividstasiswikiwiki to vividstasiswiki using renamewiki.py [19:54:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:55:33] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/bff89d0c0d2b [19:55:35] [02miraheze/puppet] 07Reception123 03bff89d0 - update rename wiki script with "good version" [19:55:36] [02puppet] 07Reception123 created branch 03Reception123-patch-2 - 13https://github.com/miraheze/puppet [19:55:37] [02puppet] 07Reception123 opened pull request 03#3814: update rename wiki script with "good version" - 13https://github.com/miraheze/puppet/pull/3814 [19:56:57] miraheze/puppet - Reception123 the build passed. [20:06:04] !log restarted jobchron171 (promox) [20:06:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:06:50] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b7881c2fd381...3557f81b4e51 [20:06:51] [02miraheze/puppet] 07Universal-Omega 033557f81 - mwtask181: disable cpjobqueue for now [20:06:52] !log [reception@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all [20:07:00] RECOVERY - Host jobchron171 is UP: PING OK - Packet loss = 0%, RTA = 0.26 ms [20:07:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:07:02] [02puppet] 07Reception123 closed pull request 03#3813: re-enable manageInactiveWikis cron - 13https://github.com/miraheze/puppet/pull/3813 [20:07:03] !log [reception@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'CreateWiki'} to all - SUCCESS in 10s [20:07:04] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3557f81b4e51...039d4e4e5a1e [20:07:06] [02miraheze/puppet] 07Reception123 03039d4e4 - re-enable manageInactiveWikis cron (#3813) [20:07:09] [02miraheze/puppet] 07Reception123 deleted branch 03Reception123-patch-1 [20:07:09] RECOVERY - jobchron171 conntrack_table_size on jobchron171 is OK: OK: nf_conntrack is 0 % full [20:07:09] RECOVERY - jobchron171 ferm_active on jobchron171 is OK: OK ferm input default policy is set [20:07:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:07:11] RECOVERY - jobchron171 Disk Space on jobchron171 is OK: DISK OK - free space: / 5763MiB (65% inode=86%); [20:07:12] [02puppet] 07Reception123 deleted branch 03Reception123-patch-1 - 13https://github.com/miraheze/puppet [20:07:17] RECOVERY - jobchron171 JobChron Service on jobchron171 is OK: PROCS OK: 1 process with args 'redisJobChronService' [20:07:23] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/853327304514...f125270935bb [20:07:26] [02miraheze/dns] 07Universal-Omega 03f125270 - Remove mwtask181 from jobrunner pool [20:08:11] RECOVERY - jobchron171 Redis Process on jobchron171 is OK: PROCS OK: 1 process with args 'redis-server' [20:08:51] RECOVERY - jobchron171 SSH on jobchron171 is OK: SSH OK - OpenSSH_9.2p1 Debian-2+deb12u2 (protocol 2.0) [20:08:56] RECOVERY - jobchron171 Puppet on jobchron171 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:11:54] !log [reception@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [20:12:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:12:04] !log [reception@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [20:12:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:16:21] PROBLEM - chobots.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'chobots.wiki' expires in 15 day(s) (Tue 16 Apr 2024 07:51:15 PM GMT +0000). [20:20:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03jobs [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/90083f0ed11e [20:20:02] [02miraheze/mw-config] 07Universal-Omega 0390083f0 - Switch some job types back to redis temporarily [20:20:04] [02mw-config] 07Universal-Omega created branch 03jobs - 13https://github.com/miraheze/mw-config [20:20:05] [02mw-config] 07Universal-Omega opened pull request 03#5523: Switch some job types back to redis temporarily - 13https://github.com/miraheze/mw-config/pull/5523 [20:20:58] miraheze/mw-config - Universal-Omega the build passed. [20:41:42] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/refreshLinks.php --wiki=sagan4alphawiki (START) [20:41:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:42:45] [02landing] 07colleirose opened pull request 03#141: Improve code formatting and add subresource integrity - 13https://github.com/miraheze/landing/pull/141 [20:42:56] [02landing] 07colleirose edited pull request 03#141: Improve code formatting and add subresource integrity - 13https://github.com/miraheze/landing/pull/141 [20:48:15] miraheze/landing - colleirose the build passed. [20:57:27] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/039d4e4e5a1e...eb2e0ab5d6c8 [20:57:28] [02miraheze/puppet] 07Universal-Omega 03eb2e0ab - changeprop: reduce workers to 16 and use debug logging [21:04:23] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9690MiB (10% inode=98%); [21:08:29] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/eb2e0ab5d6c8...d053c70bc9c8 [21:08:30] [02miraheze/puppet] 07Universal-Omega 03d053c70 - changeprop: try direct connection to jobrunner [21:09:21] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d053c70bc9c8...53355dba43b7 [21:09:23] [02miraheze/puppet] 07Universal-Omega 0353355db - changeprop: reduce workers to 12 [21:10:20] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9705MiB (10% inode=98%); [21:18:51] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.66 ms [21:20:54] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 165.77 ms [21:33:33] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/53355dba43b7...85b4e55e7849 [21:33:36] [02miraheze/puppet] 07Universal-Omega 0385b4e55 - php: increase max requests to 30000 [21:38:07] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/85b4e55e7849...75a237c70ed1 [21:38:10] [02miraheze/puppet] 07Universal-Omega 0375a237c - mwtask171: increase fpm_workers_multiplier to 1.5 [21:47:20] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw172 [21:47:37] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw172 [21:49:20] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [21:49:36] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [22:58:07] [02landing] 07Universal-Omega closed pull request 03#141: Improve code formatting and add subresource integrity - 13https://github.com/miraheze/landing/pull/141 [22:58:09] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/cb494bab439c...0a228d0ab883 [22:58:11] [02miraheze/landing] 07colleirose 030a228d0 - Improve code formatting and add subresource integrity (#141) [22:58:53] miraheze/landing - Universal-Omega the build passed. [23:00:15] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 39% [23:02:54] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/75a237c70ed1...d99eabce2857 [23:02:55] [02miraheze/puppet] 07Universal-Omega 03d99eabc - mwtask181: increase fpm_workers_multiplier to 1.5 [23:06:41] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d99eabce2857...cada389d5496 [23:06:42] [02miraheze/puppet] 07Universal-Omega 03cada389 - nginx: rename log_format for access_log to match hostnames [23:17:48] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03nginx [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/90a1ff4a2461 [23:17:51] [02miraheze/puppet] 07Universal-Omega 0390a1ff4 - nginx: replace http2_max_field_size with large_client_header_buffers [23:17:52] [02puppet] 07Universal-Omega created branch 03nginx - 13https://github.com/miraheze/puppet [23:17:53] [02puppet] 07Universal-Omega opened pull request 03#3815: nginx: replace http2_max_field_size with large_client_header_buffers - 13https://github.com/miraheze/puppet/pull/3815 [23:18:22] [02puppet] 07Universal-Omega closed pull request 03#3815: nginx: replace http2_max_field_size with large_client_header_buffers - 13https://github.com/miraheze/puppet/pull/3815 [23:18:23] [02miraheze/puppet] 07Universal-Omega deleted branch 03nginx [23:18:24] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/cada389d5496...15b9281a07e7 [23:18:25] [02miraheze/puppet] 07Universal-Omega 0315b9281 - nginx: replace http2_max_field_size with large_client_header_buffers (#3815) [23:18:26] [02puppet] 07Universal-Omega deleted branch 03nginx - 13https://github.com/miraheze/puppet [23:27:32] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/15b9281a07e7...414492bedb64 [23:27:35] [02miraheze/puppet] 07Universal-Omega 03414492b - nginx: remove application/font-woff from mime.types [23:29:38] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03mime.types [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/08733b74a862 [23:29:41] [02miraheze/puppet] 07Universal-Omega 0308733b7 - nginx: update and cleanup mime.types [23:29:42] [02puppet] 07Universal-Omega created branch 03mime.types - 13https://github.com/miraheze/puppet [23:29:44] [02puppet] 07Universal-Omega opened pull request 03#3816: nginx: update and cleanup mime.types - 13https://github.com/miraheze/puppet/pull/3816 [23:33:18] [02puppet] 07Universal-Omega closed pull request 03#3816: nginx: update and cleanup mime.types - 13https://github.com/miraheze/puppet/pull/3816 [23:33:21] [02miraheze/puppet] 07Universal-Omega deleted branch 03mime.types [23:33:23] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/414492bedb64...85bfcff623c7 [23:33:25] [02miraheze/puppet] 07Universal-Omega 0385bfcff - nginx: update and cleanup mime.types (#3816) [23:33:27] [02puppet] 07Universal-Omega deleted branch 03mime.types - 13https://github.com/miraheze/puppet [23:46:46] PROBLEM - www.clinitheque.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.clinitheque.fr' expires in 15 day(s) (Tue 16 Apr 2024 11:41:10 PM GMT +0000). [23:46:57] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e1e39b0cbe50...d5d6b3e401ef [23:46:58] [02miraheze/ssl] 07WikiTideSSLBot 03d5d6b3e - Bot: Update SSL cert for www.clinitheque.fr [23:54:40] PROBLEM - richterian.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'richterian.com' expires in 15 day(s) (Tue 16 Apr 2024 11:28:06 PM GMT +0000). [23:54:51] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d5d6b3e401ef...5bb9902f7576 [23:54:54] [02miraheze/ssl] 07WikiTideSSLBot 035bb9902 - Bot: Update SSL cert for richterian.com