[00:01:10] [02RemovePII] 07Universal-Omega closed pull request 03#19: CI: Add PHPCS and Phan - 13https://git.io/Juf1L [00:01:12] [02miraheze/RemovePII] 07Universal-Omega pushed 031 commit to 03master [+7/-1/±4] 13https://git.io/Juf9s [00:01:13] [02miraheze/RemovePII] 07Universal-Omega 03b8ff8ad - CI: Add PHPCS and Phan (#19) [00:01:15] [02RemovePII] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/Jsfpj [00:01:16] [02miraheze/RemovePII] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [00:03:25] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.92, 1.54, 1.26 [00:03:30] miraheze/RemovePII - Universal-Omega the build passed. [00:05:25] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.04, 1.27, 1.19 [00:05:25] [02GlobalNewFiles] 07Universal-Omega closed pull request 03#19: CI: Add PHPCS and Phan - 13https://git.io/JEhI8 [00:05:27] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03master [+7/-1/±5] 13https://git.io/Juf97 [00:05:28] [02miraheze/GlobalNewFiles] 07Universal-Omega 0328d6b8b - CI: Add PHPCS and Phan (#19) [00:05:30] [02miraheze/GlobalNewFiles] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [00:05:31] [02GlobalNewFiles] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/JtShO [00:07:27] miraheze/GlobalNewFiles - Universal-Omega the build passed. [00:11:22] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.60, 22.94, 16.59 [00:14:27] [02WikiDiscover] 07Universal-Omega closed pull request 03#57: CI: Add PHPCS and Phan - 13https://git.io/JEpSv [00:14:29] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+7/-1/±9] 13https://git.io/JufQ8 [00:14:30] [02miraheze/WikiDiscover] 07Universal-Omega 03ee24d57 - CI: Add PHPCS and Phan (#57) [00:14:33] [02WikiDiscover] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vhUAp [00:14:35] [02miraheze/WikiDiscover] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [00:16:22] miraheze/WikiDiscover - Universal-Omega the build passed. [00:21:22] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.17, 21.31, 19.41 [00:23:20] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.41, 19.73, 19.09 [00:23:53] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 2.33, 1.83, 1.38 [00:25:49] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.28, 1.64, 1.36 [00:44:18] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.92, 3.86, 3.36 [00:48:18] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.18, 3.50, 3.33 [00:52:19] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.10, 2.96, 3.16 [00:55:41] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JufbS [00:55:43] [02miraheze/puppet] 07paladox 0307a9b60 - mediawiki and varnish: Tweek connections to handle more [00:55:44] [02puppet] 07paladox created branch 03paladox-patch-2 - 13https://git.io/vbiAS [00:55:46] [02puppet] 07paladox opened pull request 03#1933: mediawiki and varnish: Tweek connections to handle more - 13https://git.io/Jufb9 [00:56:23] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.46, 3.32, 3.27 [00:56:35] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jufbb [00:56:37] [02miraheze/puppet] 07paladox 03d3ecfe4 - Update php.pp [00:56:39] [02puppet] 07paladox synchronize pull request 03#1933: mediawiki and varnish: Tweek connections to handle more - 13https://git.io/Jufb9 [00:57:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/Jufbj [00:57:08] [02miraheze/puppet] 07paladox 035a56f50 - Update cp12.yaml [00:57:09] [02puppet] 07paladox synchronize pull request 03#1933: mediawiki and varnish: Tweek connections to handle more - 13https://git.io/Jufb9 [00:57:31] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JufNk [00:57:33] [02miraheze/puppet] 07paladox 033b9fac9 - Update cp13.yaml [00:57:34] [02puppet] 07paladox synchronize pull request 03#1933: mediawiki and varnish: Tweek connections to handle more - 13https://git.io/Jufb9 [00:58:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+1/-0/±0] 13https://git.io/JufNI [00:58:03] [02miraheze/puppet] 07paladox 03900f022 - Create cp14.yaml [00:58:05] [02puppet] 07paladox synchronize pull request 03#1933: mediawiki and varnish: Tweek connections to handle more - 13https://git.io/Jufb9 [00:58:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.40, 3.85, 3.48 [00:58:51] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-2 [+0/-0/±1] 13https://git.io/JufNO [00:58:52] [02miraheze/puppet] 07paladox 035dcd2ba - Update cp3.yaml [00:58:54] [02puppet] 07paladox synchronize pull request 03#1933: mediawiki and varnish: Tweek connections to handle more - 13https://git.io/Jufb9 [01:02:33] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.38, 3.70, 3.51 [01:06:36] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.91, 3.32, 3.39 [01:10:25] RECOVERY - mon2 Puppet on mon2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [01:10:51] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.29, 4.22, 3.73 [01:15:00] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.20, 3.97, 3.77 [01:21:05] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.45, 3.77, 3.69 [01:23:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.64, 3.66 [01:27:11] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.39, 3.65, 3.64 [01:31:14] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 4.00, 3.91, 3.76 [01:47:20] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.54, 3.77, 3.67 [01:49:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.99, 3.55, 3.61 [01:51:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.13, 3.85, 3.71 [01:53:27] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.89, 3.39, 3.56 [01:59:28] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.66, 2.93, 3.32 [02:15:45] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 3.80, 3.43 [02:17:45] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.92, 3.46, 3.35 [02:21:46] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.34, 3.92, 3.56 [02:23:47] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.09, 3.72, 3.54 [02:25:49] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.30, 3.23, 3.38 [03:20:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.70, 3.90, 3.32 [03:22:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.88, 3.39 [03:24:06] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.56, 3.37, 3.26 [03:36:12] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.50, 3.28 [03:40:16] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.91, 3.53, 3.33 [03:42:19] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.84, 3.47 [03:43:45] [02puppet] 07Universal-Omega reviewed pull request 03#1933 commit - 13https://git.io/JuJBR [03:44:20] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.77, 3.70, 3.46 [03:46:22] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 3.39, 3.38 [03:50:36] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 3.86, 3.56 [03:52:37] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.82, 3.93, 3.63 [03:54:37] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.88, 3.64 [03:56:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.82, 3.42, 3.50 [04:02:43] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.91, 3.79, 3.61 [04:04:42] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.50, 3.33, 3.46 [04:10:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.22, 3.20, 3.35 [04:34:14] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JuJV9 [04:34:15] [02miraheze/GlobalNewFiles] 07Universal-Omega 03c706e35 - Update continuousIntegration.yml [04:34:17] [02GlobalNewFiles] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/JtShO [04:34:36] [02GlobalNewFiles] 07Universal-Omega opened pull request 03#20: CI: Use submodule instead of git clone - 13https://git.io/JuJVd [04:34:50] [02GlobalNewFiles] 07Universal-Omega edited pull request 03#20: CI: Use `git submodule` instead of `git clone` - 13https://git.io/JuJVd [04:35:30] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JuJwk [04:35:31] [02miraheze/GlobalNewFiles] 07Universal-Omega 033779130 - Update .gitignore [04:35:33] [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#20: CI: Use `git submodule` instead of `git clone` - 13https://git.io/JuJVd [04:35:52] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JuJwm [04:35:54] [02miraheze/GlobalNewFiles] 07Universal-Omega 03f474472 - Update config.php [04:35:55] [02GlobalNewFiles] 07Universal-Omega synchronize pull request 03#20: CI: Use `git submodule` instead of `git clone` - 13https://git.io/JuJVd [04:36:16] miraheze/GlobalNewFiles - Universal-Omega the build passed. [04:37:10] miraheze/GlobalNewFiles - Universal-Omega the build passed. [04:37:22] miraheze/GlobalNewFiles - Universal-Omega the build passed. [04:37:46] [02GlobalNewFiles] 07Universal-Omega closed pull request 03#20: CI: Use `git submodule` instead of `git clone` - 13https://git.io/JuJVd [04:37:48] [02miraheze/GlobalNewFiles] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JuJwr [04:37:49] [02miraheze/GlobalNewFiles] 07Universal-Omega 0387496a2 - CI: Use `git submodule` instead of `git clone` (#20) [04:37:51] [02miraheze/GlobalNewFiles] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [04:37:52] [02GlobalNewFiles] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/JtShO [04:38:26] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JuJwP [04:38:27] [02miraheze/WikiDiscover] 07Universal-Omega 03cbbcba2 - Update continuousIntegration.yml [04:38:29] [02WikiDiscover] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/vhUAp [04:38:42] [02WikiDiscover] 07Universal-Omega opened pull request 03#58: CI: Use `git submodule` instead of `git clone` - 13https://git.io/JuJw1 [04:39:24] miraheze/GlobalNewFiles - Universal-Omega the build passed. [04:39:28] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JuJwd [04:39:29] [02miraheze/WikiDiscover] 07Universal-Omega 0363f8b52 - Update .gitignore [04:39:31] [02WikiDiscover] 07Universal-Omega synchronize pull request 03#58: CI: Use `git submodule` instead of `git clone` - 13https://git.io/JuJw1 [04:39:51] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/JuJwA [04:39:52] [02miraheze/WikiDiscover] 07Universal-Omega 03aa26a7f - Update config.php [04:39:54] [02WikiDiscover] 07Universal-Omega synchronize pull request 03#58: CI: Use `git submodule` instead of `git clone` - 13https://git.io/JuJw1 [04:40:37] miraheze/WikiDiscover - Universal-Omega the build passed. [04:41:08] miraheze/WikiDiscover - Universal-Omega the build passed. [04:41:27] miraheze/WikiDiscover - Universal-Omega the build passed. [04:41:39] [02WikiDiscover] 07Universal-Omega closed pull request 03#58: CI: Use `git submodule` instead of `git clone` - 13https://git.io/JuJw1 [04:41:41] [02miraheze/WikiDiscover] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JuJr0 [04:41:42] [02miraheze/WikiDiscover] 07Universal-Omega 033751f97 - CI: Use `git submodule` instead of `git clone` (#58) [04:41:44] [02WikiDiscover] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vhUAp [04:41:45] [02miraheze/WikiDiscover] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [04:42:33] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JuJr6 [04:42:35] [02miraheze/ManageWiki] 07Universal-Omega 03480e387 - Update continuousIntegration.yml [04:42:36] [02ManageWiki] 07Universal-Omega synchronize pull request 03#297: Add CI for MediaWiki standards - 13https://git.io/JEbi4 [04:42:49] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JuJr1 [04:42:51] [02miraheze/ManageWiki] 07Universal-Omega 0369c15ca - Update .gitignore [04:42:52] [02ManageWiki] 07Universal-Omega synchronize pull request 03#297: Add CI for MediaWiki standards - 13https://git.io/JEbi4 [04:43:16] [02miraheze/ManageWiki] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-2 [+0/-0/±1] 13https://git.io/JuJr9 [04:43:18] [02miraheze/ManageWiki] 07Universal-Omega 038c68a56 - Update config.php [04:43:18] miraheze/WikiDiscover - Universal-Omega the build passed. [04:43:19] [02ManageWiki] 07Universal-Omega synchronize pull request 03#297: Add CI for MediaWiki standards - 13https://git.io/JEbi4 [04:44:58] miraheze/ManageWiki - Universal-Omega the build passed. [04:45:13] miraheze/ManageWiki - Universal-Omega the build passed. [04:45:44] miraheze/ManageWiki - Universal-Omega the build passed. [05:58:26] !log renamed testdelete1wiki to polandballruwiki (issues with re-import) [05:58:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:05:55] !log [reception@mwtask1] sudo -u www-data php createLocalAccount.php --wiki polandballruwiki Rairakkumee [06:05:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:10:23] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.68, 3.41, 3.06 [06:12:23] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.43, 3.07, 2.98 [06:17:23] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=402611wiki /home/reception/402611wiki --search-recursively (END - exit=256) [06:17:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:22:05] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=402611wiki /home/reception/402611wiki --search-recursively (END - exit=0) [06:22:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:44:37] [02miraheze/ssl] 07Reception123 pushed 033 commits to 03master [+3/-0/±3] 13https://git.io/JuUed [06:44:39] [02miraheze/ssl] 07Reception123 03d1f34d0 - add commons.polandballwiki.com cert [06:44:40] [02miraheze/ssl] 07Reception123 03e7dc979 - add wiki.konjitownmc.cf cert [06:44:42] [02miraheze/ssl] 07Reception123 036d6a65e - add wiki.digitalcandela.com cert [06:51:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.12, 3.46, 3.06 [06:53:55] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.79, 3.18, 3.01 [07:32:35] [02puppet] 07Reception123 closed pull request 03#1884: mwdeploy: add ssh keys - 13https://git.io/J08VJ [07:32:36] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JuUqU [07:32:38] [02miraheze/puppet] 07RhinosF1 03b946fdb - mwdeploy: add ssh keys (#1884) [07:36:07] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:36:37] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:37:29] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:37:46] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:38:46] PROBLEM - mw13 Puppet on mw13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:42:34] PROBLEM - mwtask1 Puppet on mwtask1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:45:33] PROBLEM - jobchron1 Puppet on jobchron1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:48:26] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=tippingpointwiki /home/reception/tippingpoint_pages_full.xml --username-prefix wikia:tipping-point (END - exit=256) [07:48:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:50:40] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JuU3t [07:50:41] [02miraheze/puppet] 07Reception123 03c810056 - fix syntax [07:53:49] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:15:32] RECOVERY - jobchron1 Puppet on jobchron1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:16:20] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JuUcA [08:16:21] [02miraheze/puppet] 07Reception123 03be1a08f - revert "mwdeploy: add ssh keys" [08:33:41] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 7 seconds ago with 0 failures [08:34:04] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [08:35:27] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [08:35:46] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 13 seconds ago with 0 failures [08:36:09] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:36:47] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 43 seconds ago with 0 failures [08:42:10] RECOVERY - mwtask1 Puppet on mwtask1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [09:15:19] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/initSiteStats.php --wiki=polandballruwiki --update (END - exit=0) [09:15:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:16:19] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=countryballswiki /home/reception/rucountryball_pages_full.xml --username-prefix wikia:countryballs --report 1 (END - exit=0) [10:16:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:27:26] PROBLEM - www.stardewmodding.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'stardewmodding.wiki' expires in 15 day(s) (Mon 20 Sep 2021 10:22:22 GMT +0000). [10:28:48] PROBLEM - stardewmodding.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'stardewmodding.wiki' expires in 15 day(s) (Mon 20 Sep 2021 10:22:22 GMT +0000). [10:32:15] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JuUHQ [10:32:17] [02miraheze/ssl] 07MirahezeSSLBot 037fd57ce - Bot: Update SSL cert for stardewmodding.wiki [11:09:14] RECOVERY - www.stardewmodding.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'stardewmodding.wiki' will expire on Fri 03 Dec 2021 09:32:10 GMT +0000. [11:09:42] RECOVERY - stardewmodding.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'stardewmodding.wiki' will expire on Fri 03 Dec 2021 09:32:10 GMT +0000. [11:22:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.58, 3.25, 2.56 [11:24:48] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.20, 2.88, 2.51 [11:54:24] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03Reception123-patch-2 [+0/-0/±1] 13https://git.io/JuTe4 [11:54:25] [02miraheze/puppet] 07Reception123 03141d087 - mwdeploy: add SSH keys back [11:54:27] [02puppet] 07Reception123 created branch 03Reception123-patch-2 - 13https://git.io/vbiAS [11:54:28] [02puppet] 07Reception123 opened pull request 03#1934: mwdeploy: add SSH keys back - 13https://git.io/JuTeR [12:07:44] Reception123: hi [12:07:46] Go [12:08:08] [02puppet] 07RhinosF1 commented on pull request 03#1934: mwdeploy: add SSH keys back - 13https://git.io/JuTf2 [12:09:45] RhinosF1: merging [12:09:58] \o/ [12:11:36] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JuTfp [12:11:37] [02miraheze/puppet] 07Reception123 0309c66f7 - mwdeploy: add SSH keys back (#1934) [12:11:39] [02puppet] 07Reception123 closed pull request 03#1934: mwdeploy: add SSH keys back - 13https://git.io/JuTeR [12:11:40] [02miraheze/puppet] 07Reception123 deleted branch 03Reception123-patch-2 [12:11:42] [02puppet] 07Reception123 deleted branch 03Reception123-patch-2 - 13https://git.io/vbiAS [12:11:54] RhinosF1: I wonder if there's any way to use GH actions somehow to have a Puppet linter [12:12:08] not sure how useful that would be but just a thought [12:12:08] Reception123: probably [12:15:50] RhinosF1: ergh [12:15:50] Failed to apply catalog: Parameter key failed on Ssh_authorize d_key[www-data-deploys]: Key must not contain whitespace: AAAAC3NzaC1lZDI1NTE5AA AAIDktIRXHBi4hDZvb6tBrPZ0Ag6TxLbXoQ7CkisQqOY6V MediaWikiDeploy (file: /etc/puppe [12:15:50] tlabs/puppet/environments/production/modules/mediawiki/manifests/init.pp, line: 31)?[0m [12:16:46] PROBLEM - mw13 Puppet on mw13 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:17:00] It's the comment [12:17:12] PROBLEM - mwtask1 Puppet on mwtask1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:17:40] PROBLEM - mw9 Puppet on mw9 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:17:47] PROBLEM - mw12 Puppet on mw12 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): File[/srv/mediawiki-staging/deploykey.pub],File[/srv/mediawiki-staging/deploykey] [12:18:03] PROBLEM - mw8 Puppet on mw8 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:18:11] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [12:18:52] Reading docs [12:18:58] Yeah I thought so [12:19:07] Why is 12 failing different [12:21:40] Reception123: remove the bit after the space [12:22:18] Ok. Brb in a few mins and will try that [12:23:03] [02puppet] 07RhinosF1 opened pull request 03#1935: mw12: do not set as canary - 13https://git.io/JuTUV [12:23:17] Reception123: that's for mw12 ^ [12:23:37] Ok [12:24:41] [02puppet] 07RhinosF1 opened pull request 03#1936: mwdeploy: remove key comment - 13https://git.io/JuTUM [12:24:51] There both ready [12:33:41] Reception123: ? [12:34:21] merging now [12:34:31] [02puppet] 07Reception123 closed pull request 03#1936: mwdeploy: remove key comment - 13https://git.io/JuTUM [12:34:33] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JuTTA [12:34:35] [02miraheze/puppet] 07RhinosF1 03b35279d - mwdeploy: remove key comment (#1936) [12:34:38] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JuTTx [12:34:40] [02miraheze/puppet] 07RhinosF1 03dccdb27 - mw12: do not set as canary (#1935) [12:34:41] [02puppet] 07Reception123 closed pull request 03#1935: mw12: do not set as canary - 13https://git.io/JuTUV [12:35:07] \o/ [12:35:45] Puppet is good on mw11 so it should work :) [12:36:13] paladox: please be careful when copying puppet heira values to new servers [12:36:34] Reception123: lmk how that looks [12:37:12] let's see [12:40:16] RhinosF1: ?[1;31mError: /Stage[main]/Mediawiki/Ssh_authorized_key[www-data-deploys]: Could not evaluate: Permission denied @ dir_s_mkdir - /var/www/.ssh?[0m [12:40:53] Reception123: can you mkdir /var/www/.ssh [12:41:03] ok [12:41:17] Not sure why setup doesn't do that [12:43:52] Reception123: make sure www-data owns that directory. It should be chmod 700 [12:44:10] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 11 seconds ago with 0 failures [12:44:28] That's a good sign [12:44:28] yup it's all good now as you can see ^ [12:44:49] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures [12:45:11] RECOVERY - mwtask1 Puppet on mwtask1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:45:21] Reception123: let's try ssh www-data@mw10.miraheze.org -i /srv/mediawiki-staging/deploykey from 11 [12:45:53] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:45:54] RECOVERY - mw12 Puppet on mw12 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:46:12] RECOVERY - mw8 Puppet on mw8 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [12:46:38] load pubkey "/srv/mediawiki-staging/deploykey": Permission denied [12:46:38] The authenticity of host 'mw10.miraheze.org ' can't be es tablished. [12:47:56] Reception123: you need to sudo -u www-data [12:48:30] I did use www-data [12:48:50] feel free to try yourself [12:49:11] Reception123: and you still see permission denied? [12:49:25] yup [12:49:49] https://www.irccloud.com/pastebin/Xd6eLLAv/ [12:50:37] Okay permission denied public key [12:50:45] Maybe it's the comment but puppet doesn't like that [12:53:08] yeah that's what I thought but afaik it should work without a comment too [12:54:00] Reception123: cat /srv/mediawiki-staging/deploykey and .pub and see what's there [12:54:07] I just read something on the doc [12:54:56] RhinosF1: private key is there [12:55:26] Reception123: on the pubkey has it put www-data-deploys on the end [12:55:40] and huh public still shows up with MediaWikiDeploy at the end [12:55:52] yeah fixing [12:56:30] Look on var/www/.ssh and see if the pubkey includes a comment Reception123 [12:56:33] At the end [12:57:04] RhinosF1: ah that would be it, it contains nothing at all [12:57:07] just known_hosts [12:57:26] Reception123: where did the key get saved then [12:57:35] [02puppet] 07RhinosF1 opened pull request 03#1937: mwdeploy: fix key name - 13https://git.io/JuTqO [12:57:46] I think ^ might work [12:57:50] But F1 now [12:57:52] let's try [12:58:02] [02puppet] 07Reception123 closed pull request 03#1937: mwdeploy: fix key name - 13https://git.io/JuTqO [12:58:03] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JuTqn [12:58:05] [02miraheze/puppet] 07RhinosF1 03e63331b - mwdeploy: fix key name (#1937) [13:18:31] Reception123: any better [13:18:52] worse actually :( [13:18:58] just a www-data@mw10.miraheze.org: Permission denied (publickey). this time [13:23:20] That's same error [13:23:35] I'll try and debug in a bit [13:23:40] yes but there was more before [13:23:45] now it's just that one line [13:23:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.26, 2.97 [13:23:59] Reception123: which file was is the key saved in on 11 [13:24:20] The other will only happen first run, that was saying it's not in known hosts [13:25:47] The puppet run should tell you for the file [13:25:53] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.52, 3.05, 2.94 [13:30:13] RhinosF1: nope puppet is quiet [13:30:19] https://www.irccloud.com/pastebin/ilSG6BTV/ [13:45:28] Reception123: it'll be in your puppet run from when it first added [13:53:27] just see [13:53:29] ?[mNotice: /Stage[main]/Mediawiki/Ssh_authorized_key[MediaWikiDeploy]/ensure: created?[0m [13:53:29] ?[0;32mInfo: Computing checksum on file /var/www/.ssh/authorized_keys?[0m [13:54:17] Hey, you free for a minute or two @Reception123? [13:54:29] yeah, but will be away soon for a bit [13:54:40] Reception123: what's in that file [13:54:44] Paste it to me [13:54:58] https://meta.miraheze.org/wiki/Special:CentralAuth/CanYOuDOIt_Can_You_put_spinner_on_finger_for_1_second They're back again. [13:54:59] [url] Global account information for CanYOuDOIt Can You put spinner on finger for 1 second - Miraheze Meta | meta.miraheze.org [13:55:47] RhinosF1: there's keys on every server but mw11 where it's empty [13:55:47] Is this an sre issue darkmatterman450 [13:55:56] Reception123: yeah that's by design [13:56:07] What does it say on 10 [13:56:17] Oh, whoops. I was confused. [13:56:27] I sent Reception123 a message. [13:56:52] https://www.irccloud.com/pastebin/TK2dhFh8/ [13:57:40] I don't get how the key is denied then [13:57:54] As that should match 11 from the staging folder [13:57:57] not sure yeah [13:58:14] Try ssh -vvvvv [13:58:33] if that doesn't give a clue I'll debug after F1 Q3 [13:59:00] https://www.irccloud.com/pastebin/eCqUFONj/ [13:59:59] Reception123: why is it saying key not accessible [14:00:12] fixed that just now but still error [14:00:14] https://www.irccloud.com/pastebin/plsdoTPh/ [14:01:49] Reception123: I'll try debug [14:01:55] Doesn't make sense [14:01:59] ok [14:02:02] and yeah, it doesn't [14:02:04] Might be a config issue with sshd [14:03:15] I see .ssh in https://github.com/miraheze/puppet/blob/d4573b12e6f1b6525800dd34f10f90316866d4fb/modules/ssh/manifests/server.pp#L19 [14:03:15] [url] puppet/server.pp at d4573b12e6f1b6525800dd34f10f90316866d4fb · miraheze/puppet · GitHub | github.com [14:04:34] So no [14:12:50] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.15, 2.96 [14:14:53] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.21, 3.22, 3.01 [14:22:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.58, 3.41, 3.14 [14:30:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.34, 3.67, 3.32 [14:32:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.95, 3.67, 3.36 [14:33:36] JohnLewis: apparently the ssh keys are not working [14:34:51] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.54, 3.25, 3.24 [14:39:29] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.82, 6.21, 5.20 [14:40:22] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.95, 6.44, 5.63 [14:40:34] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.27, 7.88, 6.16 [14:41:27] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.28, 6.09, 5.27 [14:42:15] RhinosF1: try https://github.com/miraheze/puppet/blob/f9de8177dadd357583b0482558f888eccd2964f2/modules/role/manifests/dbbackup.pp#L16 [14:42:16] [url] puppet/dbbackup.pp at f9de8177dadd357583b0482558f888eccd2964f2 · miraheze/puppet · GitHub | github.com [14:42:51] JohnLewis: do we need to specify u/gid [14:42:53] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.16, 6.33, 5.20 [14:43:52] It's marked as optional, but it would create an accessible user which www-data strictly isn't, so perhaps https://github.com/miraheze/puppet/blob/master/modules/users/manifests/user.pp#L48 [14:43:53] [url] puppet/user.pp at master · miraheze/puppet · GitHub | github.com [14:44:00] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.07, 7.09, 5.75 [14:44:53] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.06, 6.57, 5.40 [14:46:01] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.66, 6.69, 5.78 [14:46:16] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.86, 6.40, 5.85 [14:46:54] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.81, 3.98, 3.50 [14:46:56] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.19, 6.92, 5.69 [14:47:06] JohnLewis: In my head it won't matter on existing installs but I can add it if you tell me what it is [14:47:17] [02puppet] 07RhinosF1 opened pull request 03#1938: mwdeploy: convert to users::user - 13https://git.io/JuTaT [14:47:36] It's more on install it matters [14:47:38] I think [14:48:31] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.31, 7.47, 6.90 [14:48:56] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.46, 6.35, 5.62 [14:52:59] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.99, 3.67, 3.54 [14:54:31] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.69, 6.74, 6.74 [14:56:41] Using just users::key is probably better to just add the key to ssh auth [14:56:53] oh ok [14:57:42] [02puppet] 07RhinosF1 edited pull request 03#1938: mwdeploy: convert to users::key - 13https://git.io/JuTaT [14:57:46] [02puppet] 07RhinosF1 closed pull request 03#1938: mwdeploy: convert to users::key - 13https://git.io/JuTaT [14:59:24] [02puppet] 07RhinosF1 opened pull request 03#1939: mwdeploy: use users::key - 13https://git.io/JuTwq [14:59:36] JohnLewis: merge ^? [15:01:03] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.25, 3.98, 3.69 [15:03:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.86, 3.64, 3.61 [15:03:38] Reception123: hopefully ^ will have more luck [15:03:49] That reuses the same logic as data.yaml [15:04:19] ok, great. Let's try it out [15:04:32] [02puppet] 07Reception123 closed pull request 03#1939: mwdeploy: use users::key - 13https://git.io/JuTwq [15:04:33] [02miraheze/puppet] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JuTrG [15:04:35] [02miraheze/puppet] 07RhinosF1 0314795ea - mwdeploy: use users::key (#1939) [15:05:04] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.28, 3.73, 3.64 [15:07:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.03, 3.49, 3.57 [15:07:10] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 8.00, 6.93, 6.12 [15:08:20] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 6.96, 6.70, 6.60 [15:09:09] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 8.13, 7.05, 6.24 [15:10:16] RhinosF1: no luck I'm afraid [15:10:18] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.02, 6.30, 6.46 [15:10:33] Reception123: still same permission denied public key [15:10:50] yup same story I'm afraid [15:10:51] ? [15:10:54] Right [15:11:02] Keys are definitely there [15:11:10] So I'll look in a bit [15:11:12] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.69, 6.69, 6.19 [15:11:28] ok [15:12:26] Reception123: is there anything on the mw10 side [15:12:30] Check syslog [15:12:38] SSH logs should show attempts [15:15:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.72, 2.66, 3.20 [15:15:07] just see https://meta.miraheze.org/wiki/Community_noticeboard#Improving_password_standards_on_Miraheze [15:15:07] [url] Community noticeboard - Miraheze Meta | meta.miraheze.org [15:15:11] whoops wrong copy/paste [15:15:19] *Authentication refused: bad ownership or modes for directory /var/www/.ssh [15:15:35] That's good [15:15:47] It means when I told you to chmod the folder you didn't [15:16:12] Make sure to chown -R www-data [15:16:17]