[00:00:08] RECOVERY - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is OK: OK - NGINX Error Rate is 0% [00:02:03] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 25% [00:05:11] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 20487MiB (23% inode=98%); [00:05:51] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 20601MiB (23% inode=98%); [00:06:07] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 43% [00:08:57] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 40% [01:10:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [01:56:29] PROBLEM - mw181 Puppet on mw181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[libboost-regex1.74.0-icu72] [02:24:28] RECOVERY - mw181 Puppet on mw181 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [03:11:39] PROBLEM - cp28 Puppet on cp28 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [03:35:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [03:39:48] RECOVERY - cp28 Puppet on cp28 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [04:25:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [05:52:11] !log restart ircrcbot on bots171 [05:52:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:55:35] [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [06:21:35] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=fallenlondonpersonalwiki --username-prefix=fallenlondon /home/reception/Fallen+London+Wiki-20240209044052.xml (START) [06:21:36] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=fallenlondonpersonalwiki --username-prefix=fallenlondon /home/reception/Fallen+London+Wiki-20240209044052.xml (END - exit=256) [06:21:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:21:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:25:41] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=fallenlondonpersonalwiki --username-prefix=fallenlondon /home/reception/Fallen_London_Wiki-20240209044052.xml (START) [06:25:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:28:27] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/40cba47147d3...a83e60eddd8b [06:28:30] [02miraheze/ssl] 07MirahezeSSLBot 03a83e60e - Bot: Add SSL cert for patrykstan.veles-crimea.ru [06:32:09] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/a83e60eddd8b...7eaec75cb7c7 [06:32:12] [02miraheze/ssl] 07MirahezeSSLBot 037eaec75 - Bot: Add SSL cert for tfuk.app [06:36:34] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/7eaec75cb7c7...c43171bb0aa7 [06:36:36] [02miraheze/ssl] 07MirahezeSSLBot 03c43171b - Bot: Add SSL cert for wiki.kkstudio.eu.org [06:47:35] PROBLEM - patrykstan.veles-crimea.ru - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for patrykstan.veles-crimea.ru could not be found [09:59:09] PROBLEM - ie.unigon.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ie.unigon.net' expires in 15 day(s) (Thu 29 Feb 2024 09:33:00 AM GMT +0000). [09:59:18] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c43171bb0aa7...b8a50366649b [09:59:19] [02miraheze/ssl] 07MirahezeSSLBot 03b8a5036 - Bot: Update SSL cert for ie.unigon.net [10:27:41] RECOVERY - ie.unigon.net - LetsEncrypt on sslhost is OK: OK - Certificate 'ie.unigon.net' will expire on Mon 13 May 2024 08:59:12 AM GMT +0000. [12:44:21] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 6.30, 3.08, 1.30 [12:48:21] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 0.39, 1.52, 1.06 [14:27:18] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [16:05:39] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9694MiB (10% inode=98%); [16:07:47] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 39% [16:21:25] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9697MiB (10% inode=98%); [16:27:09] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [17:12:06] [02mediawiki-repos] 07R4356th opened pull request 03#11: Add UnlinkedWikibase - 13https://github.com/miraheze/mediawiki-repos/pull/11 [17:57:44] [02mediawiki-repos] 07Universal-Omega closed pull request 03#11: Add UnlinkedWikibase - 13https://github.com/miraheze/mediawiki-repos/pull/11 [17:57:45] [02miraheze/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/7c83ae33598e...5ff99ad6f2b4 [17:57:46] [02miraheze/mediawiki-repos] 07R4356th 035ff99ad - Add UnlinkedWikibase (#11) [18:05:15] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 138.39 ms [18:07:16] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 139.02 ms [18:16:11] I did put alot of translated strings to the MirahezeMagic extension and I haven't see it in action yet- [18:20:47] I'll run rebuildLC in a bit [18:22:40] great to know. [18:23:07] I'm out for the night a:b_abt2sleep: [18:59:19] [02miraheze/RequestSSL] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RequestSSL/compare/692249d07a8d...ba7e6b4db7a6 [18:59:22] [02miraheze/RequestSSL] 07Universal-Omega 03ba7e6b4 - Fix RequestSSLQueue SpecialPage title [19:02:01] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.402 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [19:06:47] miraheze/RequestSSL - Universal-Omega the build passed. [19:11:41] !log [universalomega@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_extensions': 'RequestSSL'} to all [19:11:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:11:51] !log [universalomega@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_extensions': 'RequestSSL'} to all - SUCCESS in 10s [19:11:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:12:18] [02mw-config] 07pixDeVl opened pull request 03#5480: Add Snap! Blocks to ManageWikiExtensions.php - 13https://github.com/miraheze/mw-config/pull/5480 [19:13:10] [02mediawiki-repos] 07pixDeVl opened pull request 03#12: Add Snap! Blocks to mediawiki-repos.yaml - 13https://github.com/miraheze/mediawiki-repos/pull/12 [19:14:34] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/cf33f21b8554...a3ca4a4c796f [19:14:36] [02miraheze/mw-config] 07Universal-Omega 03a3ca4a4 - Add myself to wgRequestSSLUsersNotifiedOnAllRequests [19:14:44] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [19:14:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:14:53] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [19:14:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:15:27] miraheze/mw-config - Universal-Omega the build passed. [19:31:57] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query. [19:41:47] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 40% [19:55:45] [02miraheze/RequestSSL] 07Universal-Omega pushed 031 commit to 03fix-notifs [+0/-0/±1] 13https://github.com/miraheze/RequestSSL/commit/51d8cf602d17 [19:55:46] [02miraheze/RequestSSL] 07Universal-Omega 0351d8cf6 - Allow any comment to send notifications [19:55:49] [02RequestSSL] 07Universal-Omega created branch 03fix-notifs - 13https://github.com/miraheze/RequestSSL [19:55:51] [02mediawiki-repos] 07Reception123 closed pull request 03#12: Add Snap! Blocks to mediawiki-repos.yaml - 13https://github.com/miraheze/mediawiki-repos/pull/12 [19:55:52] [02RequestSSL] 07Universal-Omega opened pull request 03#32: Allow any comment to send notifications - 13https://github.com/miraheze/RequestSSL/pull/32 [19:55:54] [02miraheze/mediawiki-repos] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/5ff99ad6f2b4...6127e75ef174 [19:55:55] [02miraheze/mediawiki-repos] 07pixDeVl 036127e75 - Add Snap! Blocks to mediawiki-repos.yaml (#12) [20:03:03] miraheze/RequestSSL - Universal-Omega the build passed. [20:07:35] [02miraheze/RequestSSL] 07Universal-Omega pushed 031 commit to 03fix [+0/-0/±1] 13https://github.com/miraheze/RequestSSL/commit/91b1814c4117 [20:07:36] [02miraheze/RequestSSL] 07Universal-Omega 0391b1814 - Fix comment notifications [20:07:37] [02RequestSSL] 07Universal-Omega created branch 03fix - 13https://github.com/miraheze/RequestSSL [20:07:39] [02RequestSSL] 07Universal-Omega opened pull request 03#33: Fix comment notifications - 13https://github.com/miraheze/RequestSSL/pull/33 [20:14:10] miraheze/RequestSSL - Universal-Omega the build passed. [20:20:16] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03fix [+0/-0/±1] 13https://github.com/miraheze/ImportDump/commit/63c18050af0a [20:20:17] [02miraheze/ImportDump] 07Universal-Omega 0363c1805 - Fix comment notifications [20:20:18] [02ImportDump] 07Universal-Omega created branch 03fix - 13https://github.com/miraheze/ImportDump [20:20:21] [02ImportDump] 07Universal-Omega opened pull request 03#64: Fix comment notifications - 13https://github.com/miraheze/ImportDump/pull/64 [20:21:30] [02miraheze/dns] 07Reception123 pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/dns/compare/3b660bfc8f29...00acbd81aee0 [20:21:32] [02miraheze/dns] 07Reception123 0300acbd8 - add lads.wiki zone [20:23:53] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=welcomewiki --no-updates --username-prefix=wikia:mt6580 /home/reception/ImportDump/welcomewiki-20240213202156.xml (START) [20:23:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:24:42] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=welcomewiki --no-updates --username-prefix=wikia:mt6580 /home/reception/ImportDump/welcomewiki-20240213202156.xml (END - exit=0) [20:24:43] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=welcomewiki (START) [20:24:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:24:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:24:52] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=welcomewiki (END - exit=0) [20:24:53] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=welcomewiki --active --update (END - exit=0) [20:24:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:24:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [20:26:55] miraheze/ImportDump - Universal-Omega the build passed. [20:31:51] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.403 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [20:34:19] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/b8a50366649b...677dd2213cc5 [20:34:22] [02miraheze/ssl] 07MirahezeSSLBot 03677dd22 - Bot: Add SSL cert for lads.wiki [20:36:15] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is CRITICAL: CRITICAL - NGINX Error Rate is 85% [20:42:15] PROBLEM - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is WARNING: WARNING - NGINX Error Rate is 54% [20:43:02] [02RequestSSL] 07Universal-Omega closed pull request 03#33: Fix comment notifications - 13https://github.com/miraheze/RequestSSL/pull/33 [20:43:05] [02miraheze/RequestSSL] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RequestSSL/compare/ba7e6b4db7a6...7d58b792c553 [20:43:08] [02miraheze/RequestSSL] 07Universal-Omega 037d58b79 - Fix comment notifications (#33) [20:43:09] [02miraheze/RequestSSL] 07Universal-Omega deleted branch 03fix [20:43:12] [02RequestSSL] 07Universal-Omega deleted branch 03fix - 13https://github.com/miraheze/RequestSSL [20:43:14] [02ImportDump] 07Universal-Omega closed pull request 03#64: Fix comment notifications - 13https://github.com/miraheze/ImportDump/pull/64 [20:43:16] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/5cb64d95ee15...2ba171b0df31 [20:43:17] [02miraheze/ImportDump] 07Universal-Omega 032ba171b - Fix comment notifications (#64) [20:43:20] [02miraheze/ImportDump] 07Universal-Omega deleted branch 03fix [20:43:23] [02ImportDump] 07Universal-Omega deleted branch 03fix - 13https://github.com/miraheze/ImportDump [20:46:15] RECOVERY - cp41 HTTP 4xx/5xx ERROR Rate on cp41 is OK: OK - NGINX Error Rate is 28% [20:50:14] miraheze/ImportDump - Universal-Omega the build passed. [20:50:22] miraheze/RequestSSL - Universal-Omega the build passed. [21:31:35] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [22:11:28] PROBLEM - cp36 Disk Space on cp36 is CRITICAL: DISK CRITICAL - free space: / 5310MiB (5% inode=98%); [22:33:49] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 5302MiB (5% inode=98%);