[00:01:42] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN [00:03:04] RECOVERY - mwtask181 ferm_active on mwtask181 is OK: OK ferm input default policy is set [00:03:20] RECOVERY - mwtask171 ferm_active on mwtask171 is OK: OK ferm input default policy is set [00:03:33] RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 20048MiB (22% inode=98%); [00:03:42] PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 100% [00:04:01] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 20073MiB (22% inode=98%); [00:05:05] RECOVERY - cp41 Disk Space on cp41 is OK: DISK OK - free space: / 20636MiB (22% inode=98%); [00:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:20:42] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK [00:33:26] [02miraheze/mediawiki-repos] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/5c178c1de3f5...d79c847b93a3 [00:33:28] [02miraheze/mediawiki-repos] 07Universal-Omega 03d79c847 - Add EventStreamConfig [00:38:16] [02mw-config] 07Universal-Omega created branch 03EventStreamConfig - 13https://github.com/miraheze/mw-config [00:38:18] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03EventStreamConfig [+1/-0/±0] 13https://github.com/miraheze/mw-config/commit/21f346ca09d7 [00:38:19] [02miraheze/mw-config] 07Universal-Omega 0321f346c - Add EventStreamConfig [00:38:22] [02mw-config] 07Universal-Omega opened pull request 03#5520: Add EventStreamConfig - 13https://github.com/miraheze/mw-config/pull/5520 [00:39:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03EventStreamConfig [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/21f346ca09d7...6378830b38e2 [00:39:10] [02miraheze/mw-config] 07Universal-Omega 036378830 - Add [00:39:13] [02mw-config] 07Universal-Omega synchronize pull request 03#5520: Add EventStreamConfig - 13https://github.com/miraheze/mw-config/pull/5520 [00:39:20] miraheze/mw-config - Universal-Omega the build passed. [00:40:05] miraheze/mw-config - Universal-Omega the build passed. [00:51:44] PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL [00:52:43] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/56cd33308c61...0e06d7cfdf77 [00:52:45] [02miraheze/puppet] 07Universal-Omega 030e06d7c - eventgate: fix calling uuid [01:08:11] [02miraheze/RequestSSL] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [+0/-0/±1] 13https://github.com/miraheze/RequestSSL/commit/1ad87c2b0682 [01:08:14] [02miraheze/RequestSSL] 07dependabot[bot] 031ad87c2 - Update php-parallel-lint/php-parallel-lint requirement [01:08:17] [02RequestSSL] 07dependabot[bot] created branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/RequestSSL [01:08:18] [02RequestSSL] 07dependabot[bot] labeled pull request 03#42: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/RequestSSL/pull/42 [01:08:19] [02RequestSSL] 07dependabot[bot] labeled pull request 03#42: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/RequestSSL/pull/42 [01:08:20] [02RequestSSL] 07dependabot[bot] opened pull request 03#42: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/RequestSSL/pull/42 [01:16:49] miraheze/RequestSSL - dependabot[bot] the build passed. [01:20:35] RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp36.wikitide.net - CNAME OK [03:03:45] !log [universalomega@mwtask181] starting deploy of {'world': True, 'extension_list': True, 'versions': '1.41'} to all [03:03:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:04:57] !log [universalomega@mwtask181] finished deploy of {'world': True, 'extension_list': True, 'versions': '1.41'} to all - SUCCESS in 72s [03:05:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:05:52] !log [universalomega@mwtask181] starting deploy of {'l10n': True, 'versions': '1.41'} to all [03:05:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:07:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03EventStreamConfig [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6378830b38e2...3ceb8219d3c5 [03:07:40] [02miraheze/mw-config] 07Universal-Omega 033ceb821 - Load file [03:07:42] [02mw-config] 07Universal-Omega synchronize pull request 03#5520: Add EventStreamConfig - 13https://github.com/miraheze/mw-config/pull/5520 [03:08:37] miraheze/mw-config - Universal-Omega the build passed. [03:12:52] !log [universalomega@mwtask181] finished deploy of {'l10n': True, 'versions': '1.41'} to all - SUCCESS in 420s [03:13:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:13:27] [02mw-config] 07Universal-Omega closed pull request 03#5520: Add EventStreamConfig - 13https://github.com/miraheze/mw-config/pull/5520 [03:13:29] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/miraheze/mw-config/compare/34ce07836e06...988286b0528a [03:13:30] !log [universalomega@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [03:13:31] [02miraheze/mw-config] 07Universal-Omega 03988286b - Add EventStreamConfig (#5520) [03:13:32] [02mw-config] 07Universal-Omega deleted branch 03EventStreamConfig - 13https://github.com/miraheze/mw-config [03:13:34] [02miraheze/mw-config] 07Universal-Omega deleted branch 03EventStreamConfig [03:13:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:13:41] !log [universalomega@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s [03:13:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:14:23] miraheze/mw-config - Universal-Omega the build passed. [03:22:02] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0e06d7cfdf77...8e12436fc2f3 [03:22:05] [02miraheze/puppet] 07Universal-Omega 038e12436 - eventgate: use EventStreamConfig API [03:23:09] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8e12436fc2f3...5ba119aabd42 [03:23:12] [02miraheze/puppet] 07Universal-Omega 035ba119a - Update comment [03:23:17] !log [@test151] starting deploy of {'config': True} to test151 [03:23:18] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [03:23:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:23:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:25:51] PROBLEM - eventgate181 eventgate on eventgate181 is CRITICAL: connect to address 10.0.18.147 and port 8192: Connection refused [03:25:56] PROBLEM - test151 MediaWiki Rendering on test151 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [03:26:17] PROBLEM - test151 HTTPS on test151 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 500 [03:27:17] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/5ba119aabd42...f8490377c669 [03:27:20] [02miraheze/puppet] 07Universal-Omega 03f849037 - eventgate: update test_events [03:29:37] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/f8490377c669...6041994cecbe [03:29:38] [02miraheze/puppet] 07Universal-Omega 036041994 - Just use meta.miraheze.org [03:31:51] RECOVERY - eventgate181 eventgate on eventgate181 is OK: TCP OK - 0.000 second response time on 10.0.18.147 port 8192 [03:37:39] !log [@mwtask171] starting deploy of {'config': True} to all [03:37:45] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6041994cecbe...0acec1343ad5 [03:37:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:37:47] [02miraheze/puppet] 07Universal-Omega 030acec13 - jobrunner: allow eventgate access to port 443 [03:37:48] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s [03:37:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:38:40] miraheze/puppet - Universal-Omega the build has errored. [03:39:25] [02mw-config] 07AgentIsai closed pull request 03#5517: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/mw-config/pull/5517 [03:39:28] [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/988286b0528a...f095df46c609 [03:39:30] [02miraheze/mw-config] 07dependabot[bot] 03f095df4 - Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 (#5517) [03:39:32] [02mw-config] 07AgentIsai deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/mw-config [03:39:35] [02miraheze/mw-config] 07AgentIsai deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [03:40:18] miraheze/mw-config - AgentIsai the build passed. [03:40:27] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/0acec1343ad5...dd629aa045e3 [03:40:30] [02miraheze/puppet] 07Universal-Omega 03dd629aa - Use jobrunner.wikitide.net again [03:53:52] !log [@test151] starting deploy of {'config': True} to test151 [03:53:53] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [03:54:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:54:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:02:34] !log [@mwtask181] starting deploy of {'config': True} to all [04:02:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:02:43] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 8s [04:02:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:07:23] !log [@mwtask171] starting deploy of {'config': True} to all [04:07:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:07:31] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 7s [04:07:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:40:07] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dd629aa045e3...ab5c1f89ad9f [04:40:08] [02miraheze/puppet] 07Universal-Omega 03ab5c1f8 - changeprop: add ImportDumpNotifyJob queue [05:35:33] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=shinwikievangelionfrwiki --no-updates --username-prefix=wikia:evangelion /home/universalomega/shinwikievangelionfrwiki-20240327211836.xml (START) [05:35:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:43:06] [02miraheze/RemovePII] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/commit/ea95dc19f5d0 [05:43:08] [02miraheze/RemovePII] 07dependabot[bot] 03ea95dc1 - Update php-parallel-lint/php-parallel-lint requirement [05:43:09] [02RemovePII] 07dependabot[bot] created branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/RemovePII [05:43:10] [02RemovePII] 07dependabot[bot] opened pull request 03#83: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/RemovePII/pull/83 [05:43:13] [02RemovePII] 07dependabot[bot] labeled pull request 03#83: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/RemovePII/pull/83 [05:43:15] [02RemovePII] 07dependabot[bot] labeled pull request 03#83: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/RemovePII/pull/83 [05:47:47] PROBLEM - swiftproxy161 HTTPS on swiftproxy161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/1.1 502 Bad Gateway [05:49:45] RECOVERY - swiftproxy161 HTTPS on swiftproxy161 is OK: HTTP OK: Status line output matched "HTTP/1.1 404" - 352 bytes in 0.023 second response time [05:50:48] miraheze/RemovePII - dependabot[bot] the build passed. [06:01:24] [02RemovePII] 07Universal-Omega closed pull request 03#83: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/RemovePII/pull/83 [06:01:25] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/RemovePII/compare/0eb282fc9635...fc83d870785f [06:01:26] [02miraheze/RemovePII] 07dependabot[bot] 03fc83d87 - Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 (#83) [06:01:27] [02miraheze/RemovePII] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [06:01:29] [02RemovePII] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/RemovePII [06:01:49] [02ImportDump] 07Universal-Omega closed pull request 03#91: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/ImportDump/pull/91 [06:01:51] [02miraheze/ImportDump] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [06:01:54] [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/a99fe914c60c...e761cc86ae92 [06:01:56] [02miraheze/ImportDump] 07dependabot[bot] 03e761cc8 - Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 (#91) [06:01:57] [02ImportDump] 07Universal-Omega deleted branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/ImportDump [06:09:12] miraheze/RemovePII - Universal-Omega the build passed. [06:10:08] miraheze/ImportDump - Universal-Omega the build passed. [06:20:27] [02miraheze/phorge-extensions] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [+0/-0/±1] 13https://github.com/miraheze/phorge-extensions/commit/34bff9343c9e [06:20:29] [02miraheze/phorge-extensions] 07dependabot[bot] 0334bff93 - Update php-parallel-lint/php-parallel-lint requirement [06:20:32] [02phorge-extensions] 07dependabot[bot] created branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/phorge-extensions [06:20:33] [02phorge-extensions] 07dependabot[bot] opened pull request 03#47: Update php-parallel-lint/php-parallel-lint requirement from 1.3.1 to 1.4.0 - 13https://github.com/miraheze/phorge-extensions/pull/47 [06:20:34] [02phorge-extensions] 07dependabot[bot] labeled pull request 03#47: Update php-parallel-lint/php-parallel-lint requirement from 1.3.1 to 1.4.0 - 13https://github.com/miraheze/phorge-extensions/pull/47 [06:20:37] [02phorge-extensions] 07dependabot[bot] labeled pull request 03#47: Update php-parallel-lint/php-parallel-lint requirement from 1.3.1 to 1.4.0 - 13https://github.com/miraheze/phorge-extensions/pull/47 [06:21:17] miraheze/phorge-extensions - dependabot[bot] the build passed. [06:26:49] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=shinwikievangelionfrwiki --no-updates --username-prefix=wikia:evangelion /home/universalomega/shinwikievangelionfrwiki-20240327211836.xml (END - exit=0) [06:26:49] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=shinwikievangelionfrwiki (START) [06:26:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:26:59] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=shinwikievangelionfrwiki (END - exit=256) [06:27:00] !log [universalomega@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/initSiteStats.php --wiki=shinwikievangelionfrwiki --active --update (END - exit=0) [06:27:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:27:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:27:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:36:02] PROBLEM - changeprop151 Current Load on changeprop151 is CRITICAL: LOAD CRITICAL - total load average: 302.84, 112.26, 51.41 [06:41:53] RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 2.22, 37.88, 36.92 [07:45:52] PROBLEM - changeprop151 NTP time on changeprop151 is WARNING: NTP WARNING: Offset -0.3510146737 secs [07:48:06] PROBLEM - changeprop151 NTP time on changeprop151 is CRITICAL: connect to address 10.0.15.148 port 5666: Connection refusedconnect to host 10.0.15.148 port 5666: Connection refused [07:52:16] PROBLEM - changeprop151 Current Load on changeprop151 is CRITICAL: LOAD CRITICAL - total load average: 121.84, 31.72, 10.67 [07:54:18] PROBLEM - changeprop151 NTP time on changeprop151 is WARNING: NTP WARNING: Offset 0.2369841337 secs [07:58:18] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ab5c1f89ad9f...3681777d5d2b [07:58:20] [02miraheze/puppet] 07Universal-Omega 033681777 - changeprop: use 28 workers [08:00:16] RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 0.00, 0.00, 0.00 [08:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:09:52] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3681777d5d2b...05ce17f62861 [08:09:54] [02miraheze/puppet] 07Universal-Omega 0305ce17f - changeprop: raise default concurrency to 50 and workers to 32 [08:12:57] PROBLEM - changeprop151 NTP time on changeprop151 is CRITICAL: NTP CRITICAL: Offset 1.200922579 secs [08:27:51] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/05ce17f62861...4d7f94a73ee7 [08:27:54] [02miraheze/puppet] 07Universal-Omega 034d7f94a - changeprop: set http_proxy [08:28:47] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/puppet/compare/4d7f94a73ee7...4e87ea0be6c3 [08:28:50] [02miraheze/puppet] 07Universal-Omega 034e87ea0 - eventgate: set http_proxy [08:29:42] PROBLEM - swiftac171 Current Load on swiftac171 is CRITICAL: LOAD CRITICAL - total load average: 13.43, 10.49, 5.41 [08:29:59] PROBLEM - swiftac171 Current Load on swiftac171 is WARNING: LOAD WARNING - total load average: 10.81, 10.05, 5.35 [08:30:03] RECOVERY - swiftac171 Current Load on swiftac171 is OK: LOAD OK - total load average: 10.11, 9.92, 5.33 [08:35:17] PROBLEM - changeprop151 Current Load on changeprop151 is CRITICAL: LOAD CRITICAL - total load average: 336.17, 232.10, 111.31 [08:41:57] PROBLEM - changeprop151 Current Load on changeprop151 is WARNING: LOAD WARNING - total load average: 10.86, 82.31, 83.07 [08:42:17] RECOVERY - changeprop151 Current Load on changeprop151 is OK: LOAD OK - total load average: 7.78, 76.98, 81.30 [09:15:58] PROBLEM - wiki.lesunk.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.lesunk.com' expires in 15 day(s) (Sat 13 Apr 2024 08:58:25 AM GMT +0000). [09:22:00] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 172.27 ms [09:24:01] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.41 ms [09:36:19] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 166.92 ms [09:37:39] Hello. I'm in the process of migrating my Fandom Wiki to Miraheze. The transfer of articles has just been done and I would also like to transfer the images. A friend of mine has made a dump of the images I can provide for this. Is it possible to send it to you to complete the migration? [09:38:21] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 170.94 ms [09:38:23] Sorry if my request is unclear or if I make mistakes. English is not my native language. [09:38:54] MrVava: please open a task on https://issue-tracker.miraheze.org [09:45:57] Thanks [10:06:42] PROBLEM - db171 Backups SQL on db171 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864186 seconds old and 116092 bytes [11:24:58] PROBLEM - ping on changeprop151 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 405.90 ms [11:28:54] PROBLEM - ping on changeprop151 is WARNING: PING WARNING - Packet loss = 0%, RTA = 127.09 ms [11:30:51] RECOVERY - ping on changeprop151 is OK: PING OK - Packet loss = 0%, RTA = 27.30 ms [11:54:45] [02miraheze/YouTube] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 [+0/-0/±1] 13https://github.com/miraheze/YouTube/commit/6fa540b8bdd7 [11:54:47] [02miraheze/YouTube] 07dependabot[bot] 036fa540b - Update php-parallel-lint/php-parallel-lint requirement [11:54:49] [02YouTube] 07dependabot[bot] created branch 03dependabot/composer/php-parallel-lint/php-parallel-lint-1.4.0 - 13https://github.com/miraheze/YouTube [11:54:51] [02YouTube] 07dependabot[bot] labeled pull request 03#26: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/YouTube/pull/26 [11:54:52] [02YouTube] 07dependabot[bot] opened pull request 03#26: Update php-parallel-lint/php-parallel-lint requirement from 1.3.2 to 1.4.0 - 13https://github.com/miraheze/YouTube/pull/26 [11:58:54] miraheze/YouTube - dependabot[bot] the build passed. [12:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:13:56] [02miraheze/landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/5d1f7a851e8a...de1eef75827c [12:13:59] [02miraheze/landing] 07translatewiki 03de1eef7 - Localisation updates from https://translatewiki.net. [12:14:02] [02miraheze/ManageWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ManageWiki/compare/fed4e9b7ef60...7a9745456906 [12:14:04] [02miraheze/ManageWiki] 07translatewiki 037a97454 - Localisation updates from https://translatewiki.net. [12:14:06] [02miraheze/MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/MirahezeMagic/compare/02488311ecb0...9a9dd7e7b848 [12:14:08] [02miraheze/MirahezeMagic] 07translatewiki 039a9dd7e - Localisation updates from https://translatewiki.net. [12:15:17] miraheze/landing - translatewiki the build passed. [12:18:13] miraheze/MirahezeMagic - translatewiki the build has errored. [12:18:43] miraheze/ManageWiki - translatewiki the build passed. [12:23:44] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [12:23:45] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [12:23:52] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [12:23:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:23:53] !log [@test151] DEPLOY ABORTED: Canary check failed for meta.mirabeta.org@localhost [12:24:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:28:46] That's real [12:32:15] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:32:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:23] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 8s [12:32:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:05:51] PROBLEM - 100acg.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate '100acg.wiki' expires in 15 day(s) (Sat 13 Apr 2024 12:37:36 PM GMT +0000). [13:29:10] PROBLEM - www.pyramidgames.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pyramidgames.wiki' expires in 15 day(s) (Sat 13 Apr 2024 01:26:13 PM GMT +0000). [13:50:45] PROBLEM - pyramidgames.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pyramidgames.wiki' expires in 15 day(s) (Sat 13 Apr 2024 01:26:13 PM GMT +0000). [13:54:00] PROBLEM - wiki.fediverse.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.fediverse.kr' expires in 15 day(s) (Sat 13 Apr 2024 01:31:16 PM GMT +0000). [14:12:41] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 162.67 ms [14:14:43] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 167.46 ms [14:22:00] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 163.43 ms [14:24:02] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 161.79 ms [14:40:36] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 205.05 ms [14:48:50] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 160.33 ms [14:50:26] PROBLEM - talumn.neptune.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'talumn.neptune.wiki' expires in 15 day(s) (Sat 13 Apr 2024 02:31:46 PM GMT +0000). [14:58:25] PROBLEM - wiki.esnmilanostatale.it - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.esnmilanostatale.it' expires in 15 day(s) (Sat 13 Apr 2024 02:26:38 PM GMT +0000). [15:04:23] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 50%, RTA = 196.79 ms [15:08:29] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 163.82 ms [16:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:31:56] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/4e87ea0be6c3...d2a5f7ce1b49 [16:31:58] [02miraheze/puppet] 07Universal-Omega 03d2a5f7c - Remove mw* from jobrunner HostGroup [16:33:06] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d2a5f7ce1b49...3fd5926875d9 [16:33:07] [02miraheze/puppet] 07Universal-Omega 033fd5926 - Replace elasticsearch HostGroup with opensearch [16:34:53] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3fd5926875d9...9b6edb069af9 [16:34:56] [02miraheze/puppet] 07Universal-Omega 039b6edb0 - Add monitoring HostGroup and replace irc with bots [16:37:12] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/9b6edb069af9...c79e29e3b6a8 [16:37:14] [02miraheze/puppet] 07Universal-Omega 03c79e29e - Remove jobchron HostGroup and add redis [16:42:25] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c79e29e3b6a8...373782b82e35 [16:42:28] [02miraheze/puppet] 07Universal-Omega 03373782b - Add more HostGroup objects [17:00:28] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/373782b82e35...ecb34aab311a [17:00:31] [02miraheze/puppet] 07Universal-Omega 03ecb34aa - Update elasticsearch hostname [17:01:24] miraheze/puppet - Universal-Omega the build has errored. [17:12:48] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.52 ms [17:16:59] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.68 ms [17:24:24] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03sslbot [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/84924d1e2d1d [17:24:25] [02miraheze/puppet] 07Universal-Omega 0384924d1 - ssl-certificate: use new bot account [17:24:27] [02puppet] 07Universal-Omega created branch 03sslbot - 13https://github.com/miraheze/puppet [17:25:14] [02puppet] 07Universal-Omega opened pull request 03#3807: T11987: ssl-certificate: use new bot account - 13https://github.com/miraheze/puppet/pull/3807 [17:26:37] miraheze/puppet - Universal-Omega the build passed. [17:51:22] [02puppet] 07Universal-Omega closed pull request 03#3807: T11987: ssl-certificate: use new bot account - 13https://github.com/miraheze/puppet/pull/3807 [17:51:25] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/ecb34aab311a...b45f20b550a4 [17:51:26] [02miraheze/puppet] 07Universal-Omega 03b45f20b - T11987: ssl-certificate: use new bot account (#3807) [17:51:28] [02miraheze/puppet] 07Universal-Omega deleted branch 03sslbot [17:51:30] [02puppet] 07Universal-Omega deleted branch 03sslbot - 13https://github.com/miraheze/puppet [17:52:42] miraheze/puppet - Universal-Omega the build passed. [17:57:01] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 173.39 ms [17:59:00] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 165.94 ms [18:08:14] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=4tourneywiki 4tourneywiki_xml_de8f478c1b4008b7eafe.xml --no-updates (START) [18:08:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:08:24] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/febd3ed57037...6e768b9ed5be [18:08:27] [02miraheze/ssl] 07WikiTideSSLBot 036e768b9 - Bot: Update SSL cert for wikitide.net [18:09:16] miraheze/ssl - WikiTideSSLBot the build has errored. [18:15:53] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/miraheze/ssl/compare/6e768b9ed5be...e73e54662820 [18:15:54] [02miraheze/ssl] 07WikiTideSSLBot 03e73e546 - Bot: Update SSL cert for apocrypha.wiki [18:16:54] miraheze/ssl - WikiTideSSLBot the build has errored. [18:17:30] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e73e54662820...a72d9bf7e92d [18:17:32] [02miraheze/ssl] 07WikiTideSSLBot 03a72d9bf - Bot: Update SSL cert for zenith.nsmbu.net [18:18:45] miraheze/ssl - WikiTideSSLBot the build has errored. [18:20:48] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=elysiumwiki elysiumwiki_xml_7998eb1b0013a9d5377b.xml --no-updates (START) [18:20:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:21:47] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=elysiumwiki elysiumwiki_xml_7998eb1b0013a9d5377b.xml --no-updates (END - exit=0) [18:21:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:01] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.56 ms [18:25:20] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a72d9bf7e92d...605967eb2a45 [18:25:23] [02miraheze/ssl] 07WikiTideSSLBot 03605967e - Bot: Update SSL cert for mirabeta.org [18:25:33] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=elsiumwiki (START) [18:25:34] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=elsiumwiki (END - exit=65280) [18:25:38] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=elsyiumwiki (START) [18:25:39] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=elsyiumwiki (END - exit=65280) [18:25:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:46] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=elysiumwiki (START) [18:25:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:25:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:00] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/rebuildall.php --wiki=elysiumwiki (END - exit=0) [18:26:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:26:20] miraheze/ssl - WikiTideSSLBot the build has errored. [18:29:00] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 173.04 ms [18:30:10] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/840278a9ff44...9b66f33e6c8e [18:30:13] [02miraheze/dns] 07Universal-Omega 039b66f33 - Remove m.miraheze.org acme challenges [18:32:30] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/605967eb2a45...6c84a9c592b5 [18:32:33] [02miraheze/ssl] 07WikiTideSSLBot 036c84a9c - Bot: Update SSL cert for m.miraheze.org [18:33:25] miraheze/ssl - WikiTideSSLBot the build has errored. [18:34:40] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw182 [18:39:02] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=4tourneywiki 4tourneywiki_xml_de8f478c1b4008b7eafe.xml --no-updates (END - exit=0) [18:39:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:40:07] RECOVERY - wikitide.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wikitide.net' will expire on Wed 26 Jun 2024 05:08:18 PM GMT +0000. [18:40:45] [02miraheze/ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/6c84a9c592b5...ce163d3ed0dc [18:40:48] [02miraheze/ssl] 07WikiTideSSLBot 03ce163d3 - Bot: Add SSL cert for volunteerforukraine.com [18:41:50] miraheze/ssl - WikiTideSSLBot the build has errored. [18:47:37] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=4tourneywiki . --search-recursively (START) [18:47:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:49:58] RECOVERY - m.miraheze.org - LetsEncrypt on sslhost is OK: OK - Certificate 'm.miraheze.org' will expire on Wed 26 Jun 2024 05:32:24 PM GMT +0000. [18:50:40] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=elysiumwiki . --search-recursively (START) [18:50:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:52:49] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/SemanticMediaWiki/maintenance/setupStore.php --wiki=farthestfrontierwiki (END - exit=0) [18:52:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:53:05] !log [alex@mwtask181] starting deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all [18:53:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:53:14] !log [alex@mwtask181] finished deploy of {'files': '../mediawiki/1.41/extensions/SemanticMediaWiki/.smw.json'} to all - SUCCESS in 9s [18:53:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:54:25] !log [agent@mwtask171] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=elysiumwiki . --search-recursively (END - exit=0) [18:54:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:58:24] [02miraheze/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/9b66f33e6c8e...24270e8a940a [18:58:27] [02miraheze/dns] 07AgentIsai 0324270e8 - Add elysium and 4tourney redirects [19:00:54] [02miraheze/ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ce163d3ed0dc...795dbad9ddfc [19:00:55] [02miraheze/ssl] 07MacFan4000 03795dbad - cleanup certs.yaml [19:02:49] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [19:03:58] RECOVERY - mirabeta.org - LetsEncrypt on sslhost is OK: OK - Certificate 'mirabeta.org' will expire on Wed 26 Jun 2024 05:25:14 PM GMT +0000. [19:06:54] [02miraheze/ssl] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/795dbad9ddfc...af0389133a8b [19:06:57] [02miraheze/ssl] 07Universal-Omega 03af03891 - Fix [19:10:32] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=/home/alex/farthestfrontierwiki --search-recursively --comment=Per [[phorge:T12003|request]], WikiTide migration (START) [19:10:32] https://meta.miraheze.org/wiki/phorge:T12003 [19:10:34] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=/home/alex/farthestfrontierwiki --search-recursively --comment=Per [[phorge:T12003|request]], WikiTide migration (END - exit=32512) [19:10:34] https://meta.miraheze.org/wiki/phorge:T12003 [19:10:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:10:40] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/b45f20b550a4...93d6901ff350 [19:10:41] [02miraheze/puppet] 07Universal-Omega 0393d6901 - ssl-certificate: fix [19:10:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:11:18] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=farthestfrontierwiki /home/alex/farthestfrontierwiki --search-recursively --comment=Per [[phorge:T12003|request]], WikiTide migration (START) [19:11:18] https://meta.miraheze.org/wiki/phorge:T12003 [19:11:19] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=farthestfrontierwiki /home/alex/farthestfrontierwiki --search-recursively --comment=Per [[phorge:T12003|request]], WikiTide migration (END - exit=32512) [19:11:19] https://meta.miraheze.org/wiki/phorge:T12003 [19:11:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:11:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:12:30] miraheze/puppet - Universal-Omega the build passed. [19:12:32] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=farthestfrontierwiki /home/alex/farthestfrontierwiki --search-recursively --comment=[[phorge:T12003]] (START) [19:12:32] https://meta.miraheze.org/wiki/phorge:T12003 [19:12:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:13:12] @rhinosf1 mwscript should not remove double quotes IMO [19:14:07] That's a Python thing [19:14:09] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/93d6901ff350...c6112dbc0bcd [19:14:12] [02miraheze/puppet] 07Universal-Omega 03c6112db - varnish: fix Main Page URL [19:14:15] Do '"text"' [19:14:19] That should keep it [19:14:21] too late for that [19:14:22] Or escape it [19:14:26] not like it mattered tbh [19:15:33] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c6112dbc0bcd...e2f4471d1292 [19:15:36] [02miraheze/puppet] 07Universal-Omega 03e2f4471 - varnish: use wikitide.net for ssl and acme [19:23:53] some images are failing to upload due to temporary errors on Swift [19:24:09] so the script will have to be re-run once it finishes to catch those images [19:24:45] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/e2f4471d1292...a034c6304597 [19:24:46] [02miraheze/puppet] 07Universal-Omega 03a034c63 - Remove example [19:25:38] miraheze/puppet - Universal-Omega the build has errored. [19:27:18] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=farthestfrontierwiki --search-recursively /home/reception/farthestfrontier --sumary=Imported from farthestfrontier.wikitide.org (START) [19:27:19] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=farthestfrontierwiki --search-recursively /home/reception/farthestfrontier --sumary=Imported from farthestfrontier.wikitide.org (END - exit=256) [19:27:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:27:43] @reception123 I'm running the import [19:27:54] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=farthestfrontierwiki --search-recursively /home/reception/farthestfrontier --summary=Imported from farthestfrontier.wikitide.org (START) [19:27:55] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=farthestfrontierwiki --search-recursively /home/reception/farthestfrontier --summary=Imported from farthestfrontier.wikitide.org (END - exit=0) [19:28:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:28:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:28:17] oh, thanks! I'll abort then [19:28:23] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/82fb2b853e02...8f85dc964f69 [19:28:24] [02miraheze/puppet] 07Universal-Omega 038f85dc9 - ssl: use wikitide.net [19:28:58] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a034c6304597...82fb2b853e02 [19:29:01] [02miraheze/puppet] 07Universal-Omega 0382fb2b8 - bastion: fix hostnames [19:31:40] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8f85dc964f69...a6f93c06e5db [19:31:43] [02miraheze/puppet] 07Universal-Omega 03a6f93c0 - prometheus: use wikitide.net for resolver [19:33:05] miraheze/puppet - Universal-Omega the build passed. [19:34:14] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/a6f93c06e5db...dbd54de38edb [19:34:15] [02miraheze/puppet] 07Universal-Omega 03dbd54de - Use wikitide.net for check_dns_auth [19:35:08] miraheze/puppet - Universal-Omega the build has errored. [19:35:18] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dbd54de38edb...793df64f82f0 [19:35:21] [02miraheze/puppet] 07Universal-Omega 03793df64 - Use wikitide.net for PowerDNS Recursor monitoring [19:36:23] miraheze/puppet - Universal-Omega the build has errored. [19:45:07] [02miraheze/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/24270e8a940a...974a1f31e738 [19:45:08] [02miraheze/dns] 07Universal-Omega 03974a1f3 - Point wikitide.net to cp [19:52:38] RECOVERY - mon181 IRCEcho on mon181 is OK: PROCS OK: 1 process with args '/usr/local/bin/ircecho' [19:53:10] RECOVERY - mon181 Puppet on mon181 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [20:02:28] RECOVERY - wikitide.net - reverse DNS on sslhost is OK: SSL OK - wikitide.net reverse DNS resolves to cp37.wikitide.net - NS RECORDS OK [20:05:35] [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:08:13] PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9697MiB (10% inode=98%); [20:19:11] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9679MiB (10% inode=98%); [20:35:34] RECOVERY - test151 HTTPS on test151 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 285 bytes in 0.074 second response time [20:36:09] RECOVERY - test151 MediaWiki Rendering on test151 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.846 second response time [20:40:37] PROBLEM - ping on changeprop151 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 1480.85 ms [20:42:35] RECOVERY - ping on changeprop151 is OK: PING OK - Packet loss = 0%, RTA = 0.26 ms [20:43:08] !log [alex@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importImages.php --wiki=farthestfrontierwiki /home/alex/farthestfrontierwiki --search-recursively --comment=[[phorge:T12003]] (END - exit=0) [20:43:08] https://meta.miraheze.org/wiki/phorge:T12003 [20:43:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [21:03:18] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/793df64f82f0...49b1cbf76230 [21:03:21] [02miraheze/puppet] 07Universal-Omega 0349b1cbf - monitoring::wiki: use wikitide.net [21:04:16] miraheze/puppet - Universal-Omega the build has errored. [21:10:55] [1/5] ``` [21:10:55] [2/5] Found: 1214 [21:10:56] [3/5] Added: 510 [21:10:56] [4/5] Failed: 704 [21:10:56]