[00:00:15] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:00:51] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:02:15] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:02:51] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:04:02] PROBLEM - cp37 Disk Space on cp37 is CRITICAL: DISK CRITICAL - free space: / 5154MiB (5% inode=98%); [00:04:43] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 22468MiB (25% inode=98%); [00:06:02] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 20851MiB (23% inode=98%); [00:08:51] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is WARNING: WARNING - NGINX Error Rate is 40% [00:11:56] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b366b819dc3b...17c92f676407 [00:11:59] [02miraheze/mw-config] 07Universal-Omega 0317c92f6 - Use Phorge [00:12:56] miraheze/mw-config - Universal-Omega the build passed. [00:13:30] [02miraheze/phorge-extensions] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/phorge-extensions/compare/515fa5e0d730...a6fd5921fbd8 [00:13:33] [02miraheze/phorge-extensions] 07Universal-Omega 03a6fd592 - Use Phorge [00:14:16] miraheze/phorge-extensions - Universal-Omega the build passed. [00:14:34] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/3b73f6b124a8...d9c743d4b182 [00:14:36] [02miraheze/ManageWiki] 07Universal-Omega 03d9c743d - Use issue-tracker [00:15:25] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/4d4f6159abf8...7def37634f67 [00:15:27] [02miraheze/CreateWiki] 07Universal-Omega 037def376 - Use Phorge [00:16:13] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/871b78ccda4d...04c85125c528 [00:16:15] [02miraheze/IncidentReporting] 07Universal-Omega 0304c8512 - Use issue-tracker [00:17:06] [02miraheze/phorge-extensions] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/phorge-extensions/compare/a6fd5921fbd8...0dc218d43332 [00:17:07] [02miraheze/phorge-extensions] 07Universal-Omega 030dc218d - Use Phorge [00:17:46] [02miraheze/RequestSSL] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RequestSSL/compare/d7319476ef50...b643e44bb527 [00:17:48] [02miraheze/RequestSSL] 07Universal-Omega 03b643e44 - Use issue-tracker [00:17:51] miraheze/phorge-extensions - Universal-Omega the build passed. [00:18:40] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/69a725081940...30d134a302e9 [00:18:42] [02miraheze/RemovePII] 07Universal-Omega 0330d134a - Use Phorge [00:18:56] miraheze/ManageWiki - Universal-Omega the build passed. [00:19:25] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/1779c865cc69...2d1b5e4560c0 [00:19:26] [02miraheze/GlobalNewFiles] 07Universal-Omega 032d1b5e4 - Use issue-tracker [00:20:22] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 61% [00:20:52] miraheze/RequestSSL - Universal-Omega the build has errored. [00:20:52] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/98b3b8742fa7...cecd55849e55 [00:20:53] [02miraheze/landing] 07Universal-Omega 03cecd558 - Use issue tracker [00:21:01] miraheze/IncidentReporting - Universal-Omega the build passed. [00:21:28] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/cecd55849e55...057b84dde40e [00:21:30] [02miraheze/landing] 07Universal-Omega 03057b84d - Fix typo [00:22:11] miraheze/landing - Universal-Omega the build passed. [00:22:17] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03redirects [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b6e18b55abc5...b06c67b1f063 [00:22:18] [02miraheze/ssl] 07Universal-Omega 03b06c67b - Use issue-tracker for redirect [00:22:20] [02ssl] 07Universal-Omega synchronize pull request 03#776: Add redirects - 13https://github.com/miraheze/ssl/pull/776 [00:22:35] miraheze/landing - Universal-Omega the build passed. [00:23:21] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/9c61515f4229...5e93ab788279 [00:23:23] [02miraheze/ErrorPages] 07Universal-Omega 035e93ab7 - Use Phorge [00:23:33] miraheze/GlobalNewFiles - Universal-Omega the build passed. [00:23:45] miraheze/CreateWiki - Universal-Omega the build passed. [00:24:18] miraheze/ErrorPages - Universal-Omega the build passed. [00:24:22] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/d9c743d4b182...6f287837d6f4 [00:24:24] [02miraheze/ManageWiki] 07Universal-Omega 036f28783 - Use Phorge [00:25:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/17c92f676407...b707188f3338 [00:25:23] [02miraheze/mw-config] 07Universal-Omega 03b707188 - Remove unneeded config [00:26:00] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/a2a251630b29...6484bf50af3c [00:26:02] [02miraheze/MatomoAnalytics] 07Universal-Omega 036484bf5 - Use issue-tracker [00:26:28] miraheze/mw-config - Universal-Omega the build passed. [00:26:42] miraheze/RemovePII - Universal-Omega the build passed. [00:27:07] [02miraheze/RequestSSL] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RequestSSL/compare/b643e44bb527...e36b1bec5a9f [00:27:10] [02miraheze/RequestSSL] 07Universal-Omega 03e36b1be - Use issue-tracker [00:27:47] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/df07d2e60a5a...b1fe460bec71 [00:27:48] [02miraheze/DataDump] 07Universal-Omega 03b1fe460 - Use issue-tracker [00:28:31] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/cfa59df51cd6...2fcce742d416 [00:28:33] [02miraheze/ImportDump] 07Universal-Omega 032fcce74 - Use issue-tracker [00:29:10] miraheze/ManageWiki - Universal-Omega the build passed. [00:29:53] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/ff436bb023cf...437798eb4d4c [00:29:55] [02miraheze/MirahezeMagic] 07Universal-Omega 03437798e - Use Phorge [00:30:05] miraheze/RequestSSL - Universal-Omega the build has errored. [00:30:30] miraheze/MatomoAnalytics - Universal-Omega the build passed. [00:31:20] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/6f287837d6f4...fb998bb024af [00:31:23] [02miraheze/ManageWiki] 07Universal-Omega 03fb998bb - Use Phorge [00:32:18] miraheze/DataDump - Universal-Omega the build passed. [00:32:47] !log [@mwtask181] starting deploy of {'config': True} to all [00:32:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:32:55] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 8s [00:32:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:33:20] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±22] 13https://github.com/miraheze/ManageWiki/compare/fb998bb024af...68c60d2e180d [00:33:21] [02miraheze/ManageWiki] 07Universal-Omega 0368c60d2 - Use Phorge [00:33:21] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [00:33:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:33:30] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s [00:33:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:36:10] miraheze/ImportDump - Universal-Omega the build passed. [00:36:24] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±32] 13https://github.com/miraheze/CreateWiki/compare/7def37634f67...c2ef506f5425 [00:36:27] [02miraheze/CreateWiki] 07Universal-Omega 03c2ef506 - Use Phorge [00:38:33] miraheze/ManageWiki - Universal-Omega the build passed. [00:39:17] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±21] 13https://github.com/miraheze/MirahezeMagic/compare/437798eb4d4c...fcd8ce3053ba [00:39:19] [02miraheze/MirahezeMagic] 07Universal-Omega 03fcd8ce3 - Use Phorge [00:44:09] miraheze/CreateWiki - Universal-Omega the build passed. [00:52:53] miraheze/MirahezeMagic - Universal-Omega the build passed. [01:02:28] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [01:02:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:02:37] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s [01:02:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:04:07] [02miraheze/landing] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/057b84dde40e...34ab495b1b50 [01:04:08] [02miraheze/landing] 07Universal-Omega 0334ab495 - Use Phorge [01:05:01] miraheze/landing - Universal-Omega the build passed. [01:05:08] [02miraheze/TSPortal] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/TSPortal/compare/f3d8c4890e55...db6723960740 [01:05:09] [02miraheze/TSPortal] 07Universal-Omega 03db67239 - Use Phorge [01:08:48] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/b707188f3338...a2de9cdbd526 [01:08:50] [02miraheze/mw-config] 07Universal-Omega 03a2de9cd - Use Phorge [01:09:45] miraheze/mw-config - Universal-Omega the build passed. [01:10:11] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/4f2065b7728a...92cc45208ab3 [01:10:14] [02miraheze/RottenLinks] 07Universal-Omega 0392cc452 - Use issue-tracker [01:10:57] [02miraheze/RottenLinks] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RottenLinks/compare/92cc45208ab3...c60af8833c17 [01:11:00] [02miraheze/RottenLinks] 07Universal-Omega 03c60af88 - Use Phorge [01:11:45] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/04c85125c528...443243294292 [01:11:47] [02miraheze/IncidentReporting] 07Universal-Omega 034432432 - Use issue-tracker [01:12:14] [02miraheze/IncidentReporting] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/IncidentReporting/compare/443243294292...30a3de4693b9 [01:12:15] [02miraheze/IncidentReporting] 07Universal-Omega 0330a3de4 - Use Phorge [01:13:00] [02miraheze/MatomoAnalytics] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/6484bf50af3c...34074693004c [01:13:02] [02miraheze/MatomoAnalytics] 07Universal-Omega 033407469 - Use Phorge [01:13:33] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/30d134a302e9...dbf50b1f79d9 [01:13:34] [02miraheze/RemovePII] 07Universal-Omega 03dbf50b1 - Use issue-tracker [01:14:22] [02miraheze/ReopenWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ReopenWiki/compare/9253d1a7582b...21503fef7def [01:14:24] [02miraheze/ReopenWiki] 07Universal-Omega 0321503fe - Use issue-tracker [01:15:22] miraheze/RottenLinks - Universal-Omega the build passed. [01:16:22] miraheze/IncidentReporting - Universal-Omega the build passed. [01:16:29] [02miraheze/DataDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/b1fe460bec71...84b89a11ea4e [01:16:30] [02miraheze/DataDump] 07Universal-Omega 0384b89a1 - Use Phorge [01:17:24] miraheze/MatomoAnalytics - Universal-Omega the build passed. [01:20:37] miraheze/DataDump - Universal-Omega the build passed. [01:20:52] miraheze/RemovePII - Universal-Omega the build passed. [01:21:11] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/71768ba99314...640449f69c35 [01:21:14] [02miraheze/WikiDiscover] 07Universal-Omega 03640449f - Use Phorge [01:22:11] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/19a33d491ba2...0a1af551e9af [01:22:14] [02miraheze/ssl] 07Universal-Omega 030a1af55 - Use issue-tracker for redirect [01:23:43] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/2fcce742d416...9e25e0ccc037 [01:23:45] [02miraheze/ImportDump] 07Universal-Omega 039e25e0c - Use issue-tracker [01:24:35] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/WikiDiscover/compare/640449f69c35...ff660fe607d7 [01:24:38] [02miraheze/WikiDiscover] 07Universal-Omega 03ff660fe - Use issue-tracker [01:25:21] miraheze/WikiDiscover - Universal-Omega the build passed. [01:25:26] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/GlobalNewFiles/compare/2d1b5e4560c0...5d48a3dd5e2a [01:25:28] [02miraheze/GlobalNewFiles] 07Universal-Omega 035d48a3d - Use Phorge [01:28:45] miraheze/WikiDiscover - Universal-Omega the build passed. [01:29:27] miraheze/GlobalNewFiles - Universal-Omega the build passed. [01:31:22] miraheze/ImportDump - Universal-Omega the build passed. [01:32:30] !log [@mwtask181] starting deploy of {'config': True} to all [01:32:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:32:38] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 8s [01:32:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:33:26] PROBLEM - elephants.ggreig.com - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out. [01:40:03] [02miraheze/ssl] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0a1af551e9af...ca5392b77067 [01:40:06] [02miraheze/ssl] 07AgentIsai 03ca5392b - Add www.lgbtqia.wiki redirect [01:44:23] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [01:53:49] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=lgbtawiki (START) [01:53:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:56:22] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=lgbtawiki (END - exit=256) [01:56:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:00:15] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=lgbtawiki --query ALTER TABLE /*_*/searchindex MODIFY si_title LONGTEXT NOT NULL; (END - exit=256) [02:00:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:00:38] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=lgbtawiki --query ALTER TABLE /*_*/searchindex MODIFY si_title LONGTEXT NOT NULL [02:00:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:00:53] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=lgbtawiki (START) [02:00:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:03:05] RECOVERY - elephants.ggreig.com - reverse DNS on sslhost is OK: SSL OK - elephants.ggreig.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:03:13] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=lgbtawiki . --overwrite --search-recursively (START) [02:03:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:11:30] [02miraheze/ManageWiki] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/68c60d2e180d...1c9a9df4c1e9 [02:11:32] [02miraheze/ManageWiki] 07AgentIsai 031c9a9df - Switch expression [02:13:32] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [02:15:51] miraheze/ManageWiki - AgentIsai the build passed. [02:21:05] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=polandballwiki . --search-recursively --overwrite (START) [02:21:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:28:00] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=lgbtawiki (END - exit=0) [02:28:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:47] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=rosettacodewiki rc.xml --no-updates (START) [02:37:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:51] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=rosettacodewiki rc.xml --no-updates (END - exit=2) [02:37:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:54] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=rosettacodewiki rc.xml --no-updates (START) [02:37:57] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=rosettacodewiki rc.xml --no-updates (END - exit=2) [02:37:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:38:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:41:49] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=rosettacodewiki rcf.xml --no-updates (START) [02:41:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:55:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [03:03:32] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=lgbtawiki . --search-recursively (START) [03:03:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:05:15] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/9ae52ba23349...675faf53b534 [03:05:17] [02miraheze/dns] 07AgentIsai 03675faf5 - Update rosettacode.org [03:05:35] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=lgbtawiki . --search-recursively (END - exit=0) [03:05:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:08:30] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/ca5392b77067...3abfb0bfdd6b [03:08:33] [02miraheze/ssl] 07MirahezeSSLBot 033abfb0b - Bot: Add SSL cert for rosettacode.org [03:10:30] [02miraheze/ssl] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3abfb0bfdd6b...32587044a993 [03:10:32] [02miraheze/ssl] 07AgentIsai 033258704 - Add Rosetta Code redirect [03:37:15] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/a2de9cdbd526...a5b354970e06 [03:37:18] [02miraheze/mw-config] 07AgentIsai 03a5b3549 - Restrict wgImplicitGroups and wgRCAge [03:38:15] miraheze/mw-config - AgentIsai the build passed. [03:42:28] [02mw-config] 07Universal-Omega commented on commit 03a5b354970e06e3c39f8fcddb6bdd5711bb8074a9 - 13https://github.com/miraheze/mw-config/commit/a5b354970e06e3c39f8fcddb6bdd5711bb8074a9#commitcomment-139090514 [03:58:06] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=rosettacodewiki rcf.xml --no-updates (END - exit=0) [03:58:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:01:55] !log [@mwtask181] starting deploy of {'config': True} to all [04:01:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:02:04] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 8s [04:02:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:48:31] !log [universalomega@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_pack': 'miraheze'} to all [04:48:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:49:56] !log [universalomega@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_pack': 'miraheze'} to all - SUCCESS in 84s [04:49:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:50:26] !log [universalomega@mwtask181] starting deploy of {'l10n': True, 'versions': '1.41'} to all [04:50:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:55:00] [02miraheze/MessageCachePerformance] 07Universal-Omega pushed 033 commits to 03main [+0/-0/±3] 13https://github.com/miraheze/MessageCachePerformance/compare/67c3e47c3150...c8e0c6d3f26f [04:55:03] [02miraheze/MessageCachePerformance] 07mszabo-wikia 0332df6e2 - Make matcherDataProvider static [04:55:05] [02miraheze/MessageCachePerformance] 07mszabo-wikia 033f78640 - Merge pull request #11 from Wikia/static-data-provider [04:55:08] [02miraheze/MessageCachePerformance] 07Universal-Omega 03c8e0c6d - Merge branch 'Wikia:main' into main [04:55:23] [02miraheze/MessageCachePerformance] 07Universal-Omega created branch 03master 13https://github.com/miraheze/MessageCachePerformance/commit/c8e0c6d3f26fdf6b7de4942c0747871200023e2b [04:55:25] [02MessageCachePerformance] 07Universal-Omega created branch 03master - 13https://github.com/miraheze/MessageCachePerformance [04:55:28] [02miraheze/MessageCachePerformance] 07Universal-Omega deleted branch 03main [04:55:31] [02MessageCachePerformance] 07Universal-Omega deleted branch 03main - 13https://github.com/miraheze/MessageCachePerformance [04:56:31] [02miraheze/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/6127e75ef174...3d1c8cc31e2c [04:56:32] [02miraheze/mediawiki-repos] 07Universal-Omega 033d1c8cc - Use master for MessageCachePerformance [04:57:06] !log [universalomega@mwtask181] finished deploy of {'l10n': True, 'versions': '1.41'} to all - SUCCESS in 400s [04:57:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:59:04] [02miraheze/.github] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/.github/compare/bfc5f2d59714...411221ab9cb6 [04:59:05] [02miraheze/.github] 07Universal-Omega 03411221a - Order extensions alphabetically [04:59:30] miraheze/MessageCachePerformance - Universal-Omega the build passed. [04:59:49] [02miraheze/.github] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/.github/compare/411221ab9cb6...64df55693f4f [04:59:52] [02miraheze/.github] 07Universal-Omega 0364df556 - Add MessageCachePerformance to CI [05:00:11] [02miraheze/MessageCachePerformance] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MessageCachePerformance/compare/c8e0c6d3f26f...d1060a6a1cbe [05:00:13] [02miraheze/MessageCachePerformance] 07Universal-Omega 03d1060a6 - Update workflow: Add MessageCachePerformance to CI [05:03:45] [02ErrorPages] 07Universal-Omega closed pull request 03#40: Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 - 13https://github.com/miraheze/ErrorPages/pull/40 [05:03:48] [02miraheze/ErrorPages] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ErrorPages/compare/5e93ab788279...e20927ac982a [05:03:51] [02miraheze/ErrorPages] 07dependabot[bot] 03e20927a - Bump styfle/cancel-workflow-action from 0.12.0 to 0.12.1 (#40) [05:03:53] [02miraheze/ErrorPages] 07Universal-Omega deleted branch 03dependabot/github_actions/styfle/cancel-workflow-action-0.12.1 [05:03:55] [02ErrorPages] 07Universal-Omega deleted branch 03dependabot/github_actions/styfle/cancel-workflow-action-0.12.1 - 13https://github.com/miraheze/ErrorPages [05:04:11] miraheze/MessageCachePerformance - Universal-Omega the build passed. [05:04:28] miraheze/ErrorPages - Universal-Omega the build passed. [05:06:16] [02mw-config] 07Universal-Omega closed pull request 03#5481: Install UnlinkedWikibase - 13https://github.com/miraheze/mw-config/pull/5481 [05:06:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/a5b354970e06...a3f8cb02fe3d [05:06:20] [02miraheze/mw-config] 07R4356th 03a3f8cb0 - Install UnlinkedWikibase (#5481) [05:06:28] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_41 MessageCachePerformance] [05:06:52] [02mediawiki-repos] 07Universal-Omega closed pull request 03#13: Fix path for Snap! Blocks - 13https://github.com/miraheze/mediawiki-repos/pull/13 [05:06:54] [02miraheze/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/3d1c8cc31e2c...1292f595e859 [05:06:57] [02miraheze/mediawiki-repos] 07R4356th 031292f59 - Fix path for Snap! Blocks (#13) [05:07:09] miraheze/mw-config - Universal-Omega the build passed. [05:12:33] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=metawiki --procs 36 (START) [05:12:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:12:36] !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/runJobs.php --wiki=metawiki --procs 36 (END - exit=0) [05:12:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:13:17] [02miraheze/phorge-extensions] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/phorge-extensions/compare/0dc218d43332...de1ac66e6ff7 [05:13:18] [02miraheze/phorge-extensions] 07Universal-Omega 03de1ac66 - Remove dead link [05:14:06] miraheze/phorge-extensions - Universal-Omega the build passed. [05:17:58] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [05:18:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:18:07] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 9s [05:18:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:30:57] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 179.06 ms [05:32:57] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 182.17 ms [06:32:12] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 178.05 ms [06:36:15] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 187.97 ms [06:40:29] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 175.91 ms [06:42:29] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 186.81 ms [06:55:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 2500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1 [07:24:02] PROBLEM - phorge171 Current Load on phorge171 is CRITICAL: LOAD CRITICAL - total load average: 2.65, 1.97, 1.63 [07:26:01] PROBLEM - phorge171 Current Load on phorge171 is WARNING: LOAD WARNING - total load average: 1.50, 1.80, 1.61 [07:27:59] RECOVERY - phorge171 Current Load on phorge171 is OK: LOAD OK - total load average: 0.99, 1.48, 1.52 [07:38:03] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 195.89 ms [07:48:09] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 179.54 ms [08:06:43] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 184.67 ms [08:08:43] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 191.76 ms [08:12:54] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 200.54 ms [08:16:56] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 187.62 ms [08:21:21] [02mw-config] 07redbluegreenhat closed pull request 03#5490: Use namespaced MirahezeIRCRCFeedFormatter - 13https://github.com/miraheze/mw-config/pull/5490 [08:51:59] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 191.39 ms [08:56:01] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 185.84 ms [09:00:50] PROBLEM - wiki.wikimedia.cat - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.wikimedia.cat. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [09:01:10] [02mw-config] 07Universal-Omega commented on pull request 03#5490: Use namespaced MirahezeIRCRCFeedFormatter - 13https://github.com/miraheze/mw-config/pull/5490#issuecomment-1966076783 [09:02:16] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 197.12 ms [09:04:17] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 192.31 ms [09:12:08] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/32587044a993...3445922e8594 [09:12:09] [02miraheze/ssl] 07MirahezeSSLBot 033445922 - Bot: Add SSL cert for wiki.hwfib.com [09:18:25] [02RequestSSL] 07redbluegreenhat closed pull request 03#26: rename updateManageWiki and split it into two functions - 13https://github.com/miraheze/RequestSSL/pull/26 [09:18:27] [02miraheze/RequestSSL] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/RequestSSL/compare/e36b1bec5a9f...0ec69da1b18e [09:18:28] [02miraheze/RequestSSL] 07redbluegreenhat 030ec69da - rename updateManageWiki and split it into two functions (#26) [09:20:04] !log [alex@test151] starting deploy of {'force': True, 'versions': '1.41', 'upgrade_extensions': 'RequestSSL'} to test151 [09:20:05] !log [alex@test151] finished deploy of {'force': True, 'versions': '1.41', 'upgrade_extensions': 'RequestSSL'} to test151 - SUCCESS in 1s [09:20:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:20:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:22:04] !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/createAndPromote.php --wiki=metawikibeta BlueMoon --custom-groups=wikicreator --force (END - exit=0) [09:22:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:22:07] miraheze/RequestSSL - redbluegreenhat the build has errored. [09:23:09] !log [alex@test151] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/createAndPromote.php --wiki=metawikibeta BlueMoon --custom-groups=wiki-creator --force (END - exit=0) [09:23:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:23:16] There we go [09:24:04] I want wikicreator rights on Meta in order to test RequestSSL after my latest PR, don't want to change Meta Mirabeta's servername [09:27:27] though now how do I go about it? [09:28:01] @cosmicalpha or @reception123: Anything I should keep in mind when it comes to changing the domain name of wikis on Mirabeta? [09:28:06] @bluemoon0332 i can point a domain at mh [09:28:10] you can use [09:28:16] yeh but I don't know if that's allowed [09:28:48] @bluemoon0332 it will need varnish logic i think to actually point at test131 [09:29:07] but you should be able to test does this domain point and set fine [09:29:32] RECOVERY - wiki.wikimedia.cat - reverse DNS on sslhost is OK: SSL OK - wiki.wikimedia.cat reverse DNS resolves to cp37.wikitide.net - CNAME OK [09:29:50] Actually I wanted to test a different thing, I haven't started with that just yet [09:30:25] I split the managewiki log functionality from the function that changes the servername, and added a check to see if ManageWiki is loaded [09:30:30] @bluemoon0332 i set bluemoon.rhinosf1.org to point at mh if you need a test domain [09:30:56] https://cdn.discordapp.com/attachments/808001911868489748/1211968703822823464/Screenshot_2024-02-27_at_09.30.43.png?ex=65f020d0&is=65ddabd0&hm=46af76441825475748a52dd95d603794a1fba7548f629b211fd4458ca86552e7& [09:31:00] nice [09:31:30] well, since I don't actually need to visit the wiki, I think it'll be fine as long as I remove it after I'm done [09:31:35] [1/2] ;; ANSWER SECTION: [09:31:35] [2/2] bluemoon.rhinosf1.com. 300 IN CNAME mw-lb.miraheze.org. [09:31:41] so confirmed set [09:32:36] i'm at uni so around all day @bluemoon0332 [09:33:47] ooof, RequestWiki is broken on mirabeta LMAO [09:37:48] Why are changing domain names? [09:37:57] oh RequestSSL [09:38:09] https://issue-tracker.miraheze.org/T11910 [09:38:15] ye @cosmicalpha [09:38:54] @bluemoon0332 according to what I see you have Graylog access? [09:39:02] wot 😳 [09:39:06] I don't?? [09:39:11] Since I don't have LDAP [09:39:16] It shows you have an account with the reader role [09:39:23] You do have an ldap also. [09:39:57] Not like I need it for this since a stacktrace is shown, no need for Graylog [09:40:11] But I don't know if I should share it, so I just put the ID on the task [09:40:18] @bluemoon0332 yeah I just noticed lol [09:40:37] I set so wgShowExceptionDetails is enabled on test151 [09:40:45] and mwtask181 [09:40:48] So OK to share publicly? [09:41:15] If there isn't any confidential stuff about you personally don't want shown yeah [09:41:17] that is really bad [09:41:29] Why? [09:41:46] @cosmicalpha the stack trace can leak confidential info [09:41:50] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 37%, RTA = 193.90 ms [09:42:11] not this one, I think [09:42:17] It doesn't. I have never once in all my time here seen the stack trace actually show anything confidential [09:42:30] i have seen it happen [09:43:05] Years ago but MW was patched for it, and that's why it isn't shown on public fscing servees and db stack trace is still hidden. [09:43:32] but can't anyone proxy to task? [09:43:36] no [09:43:42] It is behind a code now. [09:43:46] X-Debug ? [09:43:46] I added that also [09:43:50] oh cool [09:43:54] that's fine then [09:43:56] It doesn't work without an access code anymore [09:44:53] as long as that access code is reasonably unguessable then it's fine [09:45:22] Yeah, the access code was meant to be there to prevent like being able to DDoS a specific server and unauthorized proxying for things like this. [09:45:43] It's supposed to be 32 characters long but that reminds me I need to change it. [09:45:53] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 187.89 ms [09:47:04] That's at least the standard I went with but it might not need to be quite that long... [09:48:22] It does save the code in local storage in the MirahezeDebug extension so you don't have to enter each time but that just saves locally in browser... [09:49:19]