[00:01:40] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.26, 22.53, 16.32 [00:02:50] PROBLEM - cp12 Current Load on cp12 is CRITICAL: CRITICAL - load average: 3.17, 1.87, 1.30 [00:03:00] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [00:03:43] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.67, 2.94, 2.59 [00:04:49] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.79, 1.75, 1.32 [00:05:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.89, 2.90, 2.61 [00:06:49] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.02, 1.44, 1.26 [00:07:44] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.60, 21.84, 18.35 [00:09:44] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.16, 21.96, 18.74 [00:11:40] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.49, 3.39, 2.89 [00:11:43] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 20.42, 21.22, 18.85 [00:13:40] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.93, 3.27, 2.92 [00:13:45] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.65, 24.29, 20.37 [00:15:06] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.33, 5.43, 4.13 [00:15:45] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.70, 22.62, 20.25 [00:17:04] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.54, 5.77, 4.39 [00:17:44] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 31.54, 26.41, 21.93 [00:19:02] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.00, 5.35, 4.41 [00:22:10] PROBLEM - test4 Puppet on test4 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/puppetlabs.gpg] [00:23:06] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.14, 6.81, 5.18 [00:28:20] PROBLEM - ircwiki.cf - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for ircwiki.cf could not be found [00:28:21] PROBLEM - wiki.aridia.space - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.aridia.space could not be found [00:28:33] PROBLEM - reviwiki.info - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for reviwiki.info could not be found [00:28:34] PROBLEM - icclopedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for icclopedia.org could not be found [00:28:43] PROBLEM - equestripedia.org - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for equestripedia.org could not be found [00:32:55] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:35:09] RECOVERY - ircwiki.cf - reverse DNS on sslhost is OK: SSL OK - ircwiki.cf reverse DNS resolves to cp14.miraheze.org - CNAME FLAT [00:35:09] RECOVERY - wiki.aridia.space - reverse DNS on sslhost is OK: SSL OK - wiki.aridia.space reverse DNS resolves to cp13.miraheze.org - CNAME OK [00:35:28] RECOVERY - equestripedia.org - reverse DNS on sslhost is OK: SSL OK - equestripedia.org reverse DNS resolves to cp14.miraheze.org - CNAME FLAT [00:35:30] RECOVERY - icclopedia.org - reverse DNS on sslhost is OK: SSL OK - icclopedia.org reverse DNS resolves to cp14.miraheze.org - NS RECORDS OK [00:35:32] RECOVERY - reviwiki.info - reverse DNS on sslhost is OK: SSL OK - reviwiki.info reverse DNS resolves to cp13.miraheze.org - CNAME FLAT [00:37:04] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.20, 5.70, 5.91 [00:41:05] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.54, 3.51, 4.95 [00:43:45] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.04, 17.75, 22.92 [00:47:41] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 8.68, 13.05, 19.86 [00:49:56] RECOVERY - test4 Puppet on test4 is OK: OK: Puppet is currently enabled, last run 4 seconds ago with 0 failures [01:45:27] PROBLEM - cp12 Current Load on cp12 is WARNING: WARNING - load average: 1.74, 1.38, 1.09 [01:47:27] RECOVERY - cp12 Current Load on cp12 is OK: OK - load average: 1.36, 1.42, 1.15 [05:14:16] !log renamed princesspictureswiki to spectrumentertainmentwiki [05:14:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:45:11] PROBLEM - memipedia.ga - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'memipedia.ga' expires in 15 day(s) (Thu 07 Oct 2021 05:41:58 GMT +0000). [05:55:20] PROBLEM - ipv6bolivia.tk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ipv6bolivia.tk' expires in 15 day(s) (Thu 07 Oct 2021 05:50:08 GMT +0000). [05:57:17] PROBLEM - wikimas.kr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikimas.kr' expires in 15 day(s) (Thu 07 Oct 2021 05:48:35 GMT +0000). [06:55:05] PROBLEM - iesd.mobagenie.my.id - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - iesd.mobagenie.my.id All nameservers failed to answer the query. [07:01:55] RECOVERY - iesd.mobagenie.my.id - reverse DNS on sslhost is OK: SSL OK - iesd.mobagenie.my.id reverse DNS resolves to cp13.miraheze.org - CNAME OK [07:16:22] [02mw-config] 07Universal-Omega labeled pull request 03#4097: LS: disable jquery migration on testwikis - 13https://git.io/JuP3z [07:16:47] [02CreateWiki] 07Universal-Omega assigned pull request 03#233: Cleanup Echo notifications code and fix Echo email notifications - 13https://git.io/JBSa3 [07:17:14] [02CreateWiki] 07Universal-Omega labeled pull request 03#233: Cleanup Echo notifications code and fix Echo email notifications - 13https://git.io/JBSa3 [07:17:16] [02CreateWiki] 07Universal-Omega labeled pull request 03#233: Cleanup Echo notifications code and fix Echo email notifications - 13https://git.io/JBSa3 [07:17:27] [02CreateWiki] 07Universal-Omega assigned pull request 03#224: load extension/skin entry points automatically - 13https://git.io/Jcqqj [07:17:36] [02CreateWiki] 07Universal-Omega labeled pull request 03#224: load extension/skin entry points automatically - 13https://git.io/Jcqqj [07:18:00] [02ManageWiki] 07Universal-Omega labeled pull request 03#299: Error if database doesn't exist on Special:ManageWiki/core/dbname - 13https://git.io/JuudC [07:18:01] [02ManageWiki] 07Universal-Omega assigned pull request 03#299: Error if database doesn't exist on Special:ManageWiki/core/dbname - 13https://git.io/JuudC [07:18:21] [02ManageWiki] 07Universal-Omega labeled pull request 03#298: Use SiteConfiguration when checking for wiki settings - 13https://git.io/JuE9a [07:18:23] [02ManageWiki] 07Universal-Omega assigned pull request 03#298: Use SiteConfiguration when checking for wiki settings - 13https://git.io/JuE9a [07:19:58] [02ManageWiki] 07Universal-Omega labeled pull request 03#298: Use SiteConfiguration when checking for wiki settings - 13https://git.io/JuE9a [07:20:09] [02ManageWiki] 07Universal-Omega assigned pull request 03#297: Add CI for MediaWiki standards and security - 13https://git.io/JEbi4 [07:20:16] [02ManageWiki] 07Universal-Omega labeled pull request 03#297: Add CI for MediaWiki standards and security - 13https://git.io/JEbi4 [07:20:18] [02ManageWiki] 07Universal-Omega labeled pull request 03#297: Add CI for MediaWiki standards and security - 13https://git.io/JEbi4 [07:21:49] [02WikiDiscover] 07Universal-Omega assigned pull request 03#59: Use SiteConfiguration to get wiki description - 13https://git.io/JuEHb [07:22:50] [02WikiDiscover] 07Universal-Omega labeled pull request 03#59: Use SiteConfiguration to get wiki description - 13https://git.io/JuEHb [07:22:51] [02WikiDiscover] 07Universal-Omega labeled pull request 03#59: Use SiteConfiguration to get wiki description - 13https://git.io/JuEHb [07:23:49] [02MirahezeMagic] 07Universal-Omega labeled pull request 03#286: Overwrite WikiForum captcha messages for ReCaptcha V3 - 13https://git.io/Juwom [07:23:52] [02MirahezeMagic] 07Universal-Omega assigned pull request 03#286: Overwrite WikiForum captcha messages for ReCaptcha V3 - 13https://git.io/Juwom [07:23:59] [02MirahezeMagic] 07Universal-Omega closed pull request 03#286: Overwrite WikiForum captcha messages for ReCaptcha V3 - 13https://git.io/Juwom [07:24:00] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JzlYZ [07:24:02] [02miraheze/MirahezeMagic] 07Universal-Omega 037921f24 - Overwrite WikiForum captcha messages for ReCaptcha V3 (#286) [07:24:03] [02miraheze/MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 [07:24:05] [02MirahezeMagic] 07Universal-Omega deleted branch 03Universal-Omega-patch-2 - 13https://git.io/fQRGX [07:24:19] [02MirahezeMagic] 07Universal-Omega labeled pull request 03#274: T6045: Maintain default Wikimedia sites - 13https://git.io/JBXkb [07:24:58] miraheze/MirahezeMagic - Universal-Omega the build passed. [07:26:24] [02MirahezeMagic] 07Universal-Omega commented on pull request 03#239: Use qmegas/memcache-search for fetching all keys - 13https://git.io/JzlYS [08:28:31] [02mw-config] 07Universal-Omega labeled pull request 03#3991: Remove unnecessary extension loads - 13https://git.io/JcqPl [08:36:38] [02miraheze/.github] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://git.io/Jzlzp [08:36:40] [02miraheze/.github] 07Universal-Omega 03829abb2 - Add MediaWiki tests CI [08:36:41] [02.github] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://git.io/Jf3AX [08:36:43] [02.github] 07Universal-Omega opened pull request 03#1: Add MediaWiki tests CI - 13https://git.io/Jzlzj [08:45:51] [02miraheze/.github] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+1/-0/±0] 13https://git.io/Jzlac [08:45:52] [02miraheze/.github] 07Universal-Omega 0333320a0 - Create sync-tests.yml [08:45:54] [02.github] 07Universal-Omega synchronize pull request 03#1: Add MediaWiki tests CI - 13https://git.io/Jzlzj [10:12:44] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=nationsongcontestwiki /home/reception/NSCw/NSCWM/nsc_pages_full.xml --username-prefix wikia:nsc --report 1 (END - exit=256) [10:12:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:30:10] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.42, 3.12, 2.82 [13:32:08] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.54, 2.88, 2.77 [14:15:34] [02puppet] 07RhinosF1 opened pull request 03#1982: mw11: enable staging - 13https://git.io/Jz8rg [14:18:35] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.50, 3.21, 3.05 [14:19:01] Reception123: depool before merging please [14:20:36] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.16, 3.56, 3.19 [14:22:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.40, 3.72, 3.31 [14:24:31] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.58, 3.92, 3.42 [14:25:10] !log [rhinos@test3] starting deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True} to skip [14:25:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:26:29] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.86, 3.45, 3.31 [14:28:25] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.14, 3.77, 3.44 [14:30:21] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.01, 3.60, 3.43 [14:31:07] * RhinosF1 forgets how long l10n takes [14:31:56] !log [rhinos@test3] finished deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True} to skip - SUCCESS in 407s [14:32:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:32:16] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.07, 3.09, 3.26 [14:32:38] !log [rhinos@test3] starting deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True, 'ignoretime': True} to skip [14:32:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:33:03] !log test with ignore time, rubbish comparison as test3 l10n was v stale [14:33:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:35:43] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.08, 6.70, 5.33 [14:35:47] [02mw-config] 07RhinosF1 closed pull request 03#4097: LS: disable jquery migration on testwikis - 13https://git.io/JuP3z [14:35:48] [02miraheze/mw-config] 07RhinosF1 pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jz8PF [14:35:50] [02miraheze/mw-config] 07RhinosF1 03c289fe7 - LS: disable jquery migration on testwikis (#4097) [14:35:51] [02miraheze/mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-1 [14:35:53] [02mw-config] 07RhinosF1 deleted branch 03RhinosF1-patch-1 - 13https://git.io/vbvb3 [14:36:57] miraheze/mw-config - RhinosF1 the build passed. [14:37:43] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.88, 6.31, 5.36 [14:39:10] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.56, 7.29, 5.97 [14:39:57] !log [rhinos@test3] finished deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True, 'ignoretime': True} to skip - SUCCESS in 438s [14:40:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:41:32] !log [@test3] starting deploy of {'config': True} to skip [14:41:33] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [14:41:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:41:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [14:43:06] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.34, 6.70, 6.05 [14:56:17] [02puppet] 07R4356th opened pull request 03#1983: csp: add *.wikipedia.org to connect-src - 13https://git.io/Jz8yc [15:08:33] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.08, 18.51, 17.20 [15:12:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.73, 3.39, 3.11 [15:12:31] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 19.67, 18.88, 17.63 [15:12:51] PROBLEM - mw10 Puppet on mw10 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:14:03] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.06, 3.23, 3.08 [15:14:22] PROBLEM - mw11 Puppet on mw11 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_MediaWiki core] [15:36:22] RECOVERY - mw11 Puppet on mw11 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:36:48] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [15:42:22] PROBLEM - mw11 Puppet on mw11 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 7 minutes ago with 0 failures [15:43:39] !log stop jobrunner on mw11 & disable puppet on mw* [15:43:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:43:57] PROBLEM - mw11 JobRunner Service on mw11 is CRITICAL: PROCS CRITICAL: 0 processes with args 'redisJobRunnerService' [15:46:06] PROBLEM - mw8 Puppet on mw8 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 12 minutes ago with 0 failures [15:47:28] PROBLEM - mwtask1 Puppet on mwtask1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 6 minutes ago with 0 failures [15:48:04] PROBLEM - mw13 Puppet on mw13 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 11 minutes ago with 0 failures [15:48:32] PROBLEM - mw9 Puppet on mw9 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 14 minutes ago with 0 failures [15:48:47] PROBLEM - mw10 Puppet on mw10 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 13 minutes ago with 0 failures [15:54:46] Voidwalker: merge https://github.com/miraheze/puppet/pull/1844 and https://github.com/miraheze/puppet/pull/1982 [15:54:47] [url] mediawiki: Only clone mediawiki to staging servers by RhinosF1 · Pull Request #1844 · miraheze/puppet · GitHub | github.com [15:54:48] [url] mw11: enable staging by RhinosF1 · Pull Request #1982 · miraheze/puppet · GitHub | github.com [15:54:49] depool mw11 [15:58:36] quick question re 1844, is it not necessary to include '/srv/mediawiki-staging/w' and '/srv/mediawiki-staging/config' as files? in https://github.com/miraheze/puppet/pull/1844/files#diff-fd800f812a93342636f4d07678f65e54b991c79c3e22277240197db8306fc7c9R75 [15:58:36] [url] mediawiki: Only clone mediawiki to staging servers by RhinosF1 · Pull Request #1844 · miraheze/puppet · GitHub | github.com [15:58:57] Voidwalker: no, they're git repos [15:59:32] yeah, should be defined later, so it should be good [16:00:07] Voidwalker: please depool mw11 so i can do the work on there [16:00:32] but merge can happen immediate as puppet is stopped [16:00:34] yup, connecting now [16:00:51] bot are good to merge then? [16:00:55] *both [16:01:04] Voidwalker: yep [16:01:22] [02puppet] 07The-Voidwalker closed pull request 03#1844: mediawiki: Only clone mediawiki to staging servers - 13https://git.io/JB556 [16:01:24] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±2] 13https://git.io/Jz8x9 [16:01:25] [02miraheze/puppet] 07RhinosF1 0372f2a09 - mediawiki: Only clone mediawiki to staging servers (#1844) [16:01:42] [02puppet] 07The-Voidwalker closed pull request 03#1982: mw11: enable staging - 13https://git.io/Jz8rg [16:01:43] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jz8x5 [16:01:45] [02miraheze/puppet] 07RhinosF1 037d7fd46 - mw11: enable staging (#1982) [16:01:46] let me know when puppet is pulled [16:02:45] !log depool mw11 [16:02:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:03:19] puppet is pulled [16:03:47] !log rm w & config git repos on mw11, enable-puppet on mw8 [16:03:50] !log [rhinos@mw8] enabled puppet [16:03:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:03:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:04:46] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw11 [16:05:14] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw11 [16:05:18] [02puppet] 07RhinosF1 opened pull request 03#1984: php: fix broken before - 13https://git.io/Jz8pR [16:05:21] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: 1 backends are down. mw11 [16:05:54] PROBLEM - cp14 Varnish Backends on cp14 is CRITICAL: 1 backends are down. mw11 [16:05:56] [02puppet] 07RhinosF1 synchronize pull request 03#1984: php: fix broken before - 13https://git.io/Jz8pR [16:06:20] Voidwalker: ^ [16:06:26] [02puppet] 07The-Voidwalker closed pull request 03#1984: php: fix broken before - 13https://git.io/Jz8pR [16:06:27] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jz8pK [16:06:29] [02miraheze/puppet] 07RhinosF1 03d7409f0 - php: fix broken before (#1984) [16:06:40] pulled to puppet [16:07:00] PROBLEM - mw11 MediaWiki Rendering on mw11 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 261 bytes in 0.005 second response time [16:07:01] !log enable puppet on mw11 - this will sync config [16:07:03] !log [rhinos@mw8] enabled puppet [16:07:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:07:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:07:33] !log [@mw11] starting deploy of {'config': True} to all [16:07:34] !log [@mw11] DEPLOY ABORTED: Canary check failed for localhost [16:07:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:07:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:10:54] Voidwalker: I'm seeing Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not find resource 'Git::Clone[MediaWiki config]' in parameter 'require' (file: /etc/puppetlabs/puppet/environments/production/modules/mediawiki/manifests/init.pp, line: 177) on node mw8.miraheze.org [16:11:02] but on mw8 staging is false [16:11:10] so that line shouldn't be called [16:11:23] !log [rhinos@mw11] enabled puppet [16:11:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:11:29] https://github.com/miraheze/puppet/blob/master/modules/mediawiki/manifests/init.pp#L73-L119 should be under lookup [16:11:30] [url] puppet/init.pp at master · miraheze/puppet · GitHub | github.com [16:12:03] !log [rhinos@mw11] starting deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True, 'ignoretime': True} to skip [16:12:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:12:15] that looks like invalid syntax to me, at least the hiera key should be in quotes [16:12:32] RhinosF1, https://github.com/miraheze/puppet/blob/master/modules/mediawiki/manifests/init.pp#L177 [16:12:32] [url] puppet/init.pp at master · miraheze/puppet · GitHub | github.com [16:12:37] plus please fix the if{ indentation, that's really hard to read [16:13:03] RECOVERY - mw11 MediaWiki Rendering on mw11 is OK: HTTP OK: HTTP/1.1 200 OK - 20009 bytes in 0.221 second response time [16:13:05] majavah: it works above [16:13:07] Voidwalker: oh [16:13:59] [02puppet] 07RhinosF1 opened pull request 03#1985: mediawiki: git -> file - 13https://git.io/Jz8hA [16:14:09] Voidwalker: ^ [16:15:01] that's not the only require => Git::Clone['MediaWiki config'] in there [16:15:51] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.85, 6.23, 5.09 [16:16:15] !log [rhinos@mw11] finished deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True, 'ignoretime': True} to skip - SUCCESS in 252s [16:16:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:16:49] [02puppet] 07RhinosF1 synchronize pull request 03#1985: mediawiki: git -> file - 13https://git.io/Jz8hA [16:16:56] Voidwalker: ^ [16:17:19] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03The-Voidwalker-patch-1 [+0/-0/±1] 13https://git.io/Jz8jw [16:17:20] [02miraheze/puppet] 07The-Voidwalker 032ce1fbc - fix indentation [16:17:22] [02puppet] 07The-Voidwalker created branch 03The-Voidwalker-patch-1 - 13https://git.io/vbiAS [16:17:25] [02puppet] 07The-Voidwalker opened pull request 03#1986: fix indentation - 13https://git.io/Jz8jK [16:17:35] will do that in a few ^ [16:17:43] ok [16:17:46] mw11 is up [16:17:51] so doing config to all [16:17:53] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 6.80, 6.38, 5.28 [16:18:07] !log [rhinos@mw11] starting deploy of {'config': True} to all [16:18:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:18:27] !log [rhinos@mw11] DEPLOY ABORTED: Canary check failed for mwtask1 [16:18:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:19:02] [02puppet] 07The-Voidwalker closed pull request 03#1985: mediawiki: git -> file - 13https://git.io/Jz8hA [16:19:04] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jz4eU [16:19:05] [02miraheze/puppet] 07RhinosF1 037fb5c5a - mediawiki: git -> file (#1985) [16:19:28] pulled to puppet [16:19:35] !log [www-data@mw11] starting deploy of {'config': True} to all [16:19:44] Voidwalker: ok [16:19:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:20:01] [02puppet] 07The-Voidwalker closed pull request 03#1986: fix indentation - 13https://git.io/Jz8jK [16:20:02] [02puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 - 13https://git.io/vbiAS [16:20:04] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jz4e3 [16:20:06] [02miraheze/puppet] 07The-Voidwalker 03a3598a3 - fix indentation (#1986) [16:20:07] [02miraheze/puppet] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 [16:20:21] !log [www-data@mw11] DEPLOY ABORTED: Canary check failed for mwtask1 [16:20:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:20:30] why is mwtask broke [16:20:40] !log [www-data@mw11] starting deploy of {'config': True, 'force': True} to all [16:20:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:20:48] !log [www-data@mw11] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 7s [16:20:48] pushing to all [16:20:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:21:10] is it? [16:21:50] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.92, 8.05, 6.19 [16:21:53] Voidwalker: according to the canary check [16:21:53] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.89, 7.18, 5.90 [16:21:59] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 12.77, 8.52, 6.46 [16:22:01] that didn't deploy [16:22:34] oh, well, I doubt it is configured to serve any mw traffic [16:22:59] !log [www-data@mw11] starting deploy of {'config': True, 'force': True, 'ignoretime': True} to all [16:23:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:23:08] !log [www-data@mw11] finished deploy of {'config': True, 'force': True, 'ignoretime': True} to all - SUCCESS in 8s [16:23:09] Voidwalker: probably won't proxy then [16:23:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:23:44] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.22, 7.74, 6.30 [16:24:35] hmm, no, I can pull localhost [16:25:42] and mw11 is in the firewall, so I'm not sure [16:25:52] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.59, 7.49, 6.29 [16:27:41] Voidwalker: it's not writing anything to the remotte [16:27:50] https://www.irccloud.com/pastebin/Or08GzpU/ [16:27:51] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.50, 7.14, 6.32 [16:28:02] but nothing happens [16:29:04] ok [16:29:07] i see [16:29:40] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.41, 7.74, 6.98 [16:29:51] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.43, 8.04, 6.74 [16:30:32] [02puppet] 07RhinosF1 opened pull request 03#1987: mwdeploy: bugfixes - 13https://git.io/Jz4vx [16:31:16] !log [www-data@mw11] starting deploy of {'config': True, 'force': True, 'ignoretime': True} to all [16:31:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:31:21] !log [www-data@mw11] finished deploy of {'config': True, 'force': True, 'ignoretime': True} to all - SUCCESS in 5s [16:31:23] Voidwalker: ^ [16:31:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:31:51] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.71, 7.51, 6.70 [16:32:51] [02puppet] 07The-Voidwalker closed pull request 03#1987: mwdeploy: bugfixes - 13https://git.io/Jz4vx [16:32:53] [02miraheze/puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jz4f8 [16:32:54] [02miraheze/puppet] 07RhinosF1 03fb22d65 - mwdeploy: bugfixes (#1987) [16:33:03] live on puppet [16:33:29] !log enable puppet on remaining mw* [16:33:33] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.84, 7.87, 7.16 [16:33:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:34:48] Voidwalker: waiting for puppet to finish [16:35:14] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.33, 6.64, 6.49 [16:35:27] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.08, 7.46, 7.09 [16:35:51] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 9.48, 7.81, 6.95 [16:35:59] RECOVERY - mw11 JobRunner Service on mw11 is OK: PROCS OK: 1 process with args 'redisJobRunnerService' [16:36:04] RECOVERY - mwtask1 Puppet on mwtask1 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures [16:36:09] RECOVERY - mw13 Puppet on mw13 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [16:36:15] !log [www-data@mw11] starting deploy of {'config': True, 'force': True, 'ignoretime': True} to all [16:36:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:36:21] \o/ [16:36:27] !log [www-data@mw11] finished deploy of {'config': True, 'force': True, 'ignoretime': True} to all - SUCCESS in 12s [16:36:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:36:49] RECOVERY - mw10 Puppet on mw10 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [16:36:49] RECOVERY - mw9 Puppet on mw9 is OK: OK: Puppet is currently enabled, last run 6 minutes ago with 0 failures [16:37:03] that's fast [16:37:22] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.06, 7.84, 7.28 [16:37:22] did it work that fast? [16:37:35] Voidwalker: looks like it [16:37:50] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.43, 7.46, 6.92 [16:38:01] that's a pretty big improvement then, I suppose [16:38:14] Voidwalker: going for w now [16:38:18] this will take longer [16:38:59] !log [rhinos@mw11] starting deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True, 'force': True, 'ignoretime': True} to all [16:39:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:39:06] how much longer will you need me? [16:39:06] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.55, 7.97, 7.12 [16:39:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.26, 7.27, 7.13 [16:39:19] Voidwalker: from you, hopefully only a couple of minutes [16:39:25] mw11 should be safe to repool [16:40:50] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 9 backends are healthy [16:40:56] !log repool mw11 [16:41:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:41:14] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 9 backends are healthy [16:41:21] that's good - I should have planned this better, as I have to run out the door soon [16:41:51] RECOVERY - cp3 Varnish Backends on cp3 is OK: All 9 backends are healthy [16:41:51] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.63, 6.36, 6.63 [16:42:06] RECOVERY - cp14 Varnish Backends on cp14 is OK: All 9 backends are healthy [16:42:41] the worst that can happen now is l10n doesn't get auto updated [16:42:44] config is fine [16:42:56] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.64, 5.90, 6.49 [16:43:05] so w will be [16:43:06] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.03, 5.92, 6.64 [16:43:16] test3 has been copying like this for a month [16:44:19] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.89, 6.24, 5.35 [16:45:12] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.90, 4.86, 2.77 [16:45:35] this is about as late as I can go, can you handle it from here? [16:45:43] yes [16:46:17] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.24, 6.43, 5.54 [16:46:22] all right, in that case, all be around in about 3 hours with no computer access until then [16:46:25] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.88, 7.12, 6.22 [16:47:10] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 6.17, 5.39, 3.23 [16:50:12] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.38, 6.64, 6.23 [16:55:11] [02puppet] 07Universal-Omega commented on pull request 03#1983: csp: add *.wikipedia.org to connect-src - 13https://git.io/Jz4kV [16:58:58] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.52, 19.72, 17.14 [17:00:55] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 15.80, 18.29, 16.94 [17:05:47] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.44, 6.74, 6.38 [17:07:57] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 3.87, 5.96, 6.17 [17:08:57] it's on final server [17:09:03] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.95, 21.00, 18.87 [17:11:15] !log w / config synced on all servers, l10n remaining [17:11:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:12:58] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 17.05, 19.92, 19.00 [17:14:00] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.22, 20.45, 17.26 [17:17:29] !log [rhinos@mw11] finished deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True, 'force': True, 'ignoretime': True} to all - FAIL: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 256, 256, 256, 256, 256, 256] in 2309s [17:17:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:17:59] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.89, 18.03, 17.15 [17:20:19] !log [rhinos@mw11] starting deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True, 'force': True} to all [17:20:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:20:28] [02puppet] 07RhinosF1 opened pull request 03#1988: mwdeploy: fix rsync --delete usage; add some logging - 13https://git.io/Jz4mV [17:21:01] seeing faster without ignorettime [17:21:58] !log [rhinos@mw11] finished deploy of {'config': True, 'world': True, 'l10n': True, 'gitinfo': True, 'force': True} to all - FAIL: [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 256, 256, 256, 256, 256, 256] in 98s [17:22:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:23:27] far faster [17:23:43] the fails are l10n [17:25:46] well, seems like Chameleon broke [17:26:15] https://cdn.discordapp.com/attachments/808001911868489748/889925506269589534/unknown.png [17:26:28] link [17:26:43] https://lakehub.ga [17:26:45] [url] Lakehub | lakehub.ga [17:28:58] CosmicAlpha: anything in graylog [17:29:03] my proxy settings are off [17:30:06] @Lake console says VM167:1 Uncaught TypeError: Cannot read property 'getAttribute' of null [17:30:40] no that also on vector [17:31:15] that's because of a JS script I made it wrong and didn't fixed (but it's been there for quite a while) [17:31:23] I'll remove in any case [17:31:39] i don't see anything obvious [17:32:59] by the way, i didn't changed anything on the wiki in the past 10 days [17:33:28] it's composer [17:33:55] !log [rhinos@mw11] starting deploy of {'folders': 'w/extensions/Bootstrap/vendor'} to all [17:33:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:34:16] !log [rhinos@mw11] DEPLOY ABORTED: Canary check failed for mwtask1 [17:34:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:34:36] !log [rhinos@mw11] starting deploy of {'force': True, 'folders': 'w/extensions/Bootstrap/vendor'} to all [17:34:43] !log [rhinos@mw11] finished deploy of {'force': True, 'folders': 'w/extensions/Bootstrap/vendor'} to all - SUCCESS in 6s [17:34:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:34:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:35:29] CosmicAlpha: i've tried composer update but nothing [17:35:43] !log [rhinos@mw11] starting deploy of {'force': True, 'folders': 'w/extensions/Bootstrap/vendor', 'ignoretime': True} to all [17:35:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:35:52] !log [rhinos@mw11] finished deploy of {'force': True, 'folders': 'w/extensions/Bootstrap/vendor', 'ignoretime': True} to all - SUCCESS in 8s [17:35:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:36:21] RhinosF1: I'm not on PC right now so don't know. Will be soon though. [17:36:38] CosmicAlpha: I've ran composer install/update but nothing [17:36:45] and puppet should do anyway [17:36:51] pls run in staging [17:37:43] @Lake composer was ran so no idea [17:40:32] Hmm... interesting [17:41:30] CosmicAlpha: anything in logs? [17:42:35] Looking now [17:47:06] Nothing in logs. [17:47:21]