[02:27:23] [02WikiForge/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/dns/compare/d60d96366735...b589f202a199 [02:27:24] [02WikiForge/dns] 07AgentIsai 03b589f20 - Add records [02:28:08] [02WikiForge/dns] 07AgentIsai pushed 031 commit to 03master [+1/-0/±0] 13https://github.com/WikiForge/dns/compare/b589f202a199...5f3e301cf025 [02:28:10] [02WikiForge/dns] 07AgentIsai 035f3e301 - Add your.wf [03:01:44] RECOVERY - mon1 Backups Grafana on mon1 is OK: FILE_AGE OK: /var/log/grafana-backup.log is 102 seconds old and 349 bytes [03:17:01] [02WikiForge/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/dns/compare/5f3e301cf025...f60332dc388f [03:17:04] [02WikiForge/dns] 07AgentIsai 03f60332d - Fix AAAA record [03:19:53] RECOVERY - ns2 Auth DNS on ns2 is OK: DNS OK: 0.229 seconds response time. inside.wf returns 208.110.84.235,2604:4300:a:37::116 [03:26:21] [02WikiForge/dns] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/dns/compare/f60332dc388f...4304420ba243 [03:26:24] [02WikiForge/dns] 07AgentIsai 034304420 - Add cp*/ns* [03:50:17] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.38.134.38: Connection reset by peer [03:50:19] PROBLEM - cp3 Varnish Backends on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [03:50:19] PROBLEM - cp4 Current Load on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.38.134.38: Connection reset by peer [03:50:19] PROBLEM - cp3 Nginx Backend for mail1 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [03:50:21] PROBLEM - cp4 APT on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.38.134.38: Connection reset by peer [03:50:23] PROBLEM - cp3 Nginx Backend for mw2 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [03:50:27] PROBLEM - cp4 Nginx Backend for mail1 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.38.134.38: Connection reset by peer [03:50:27] PROBLEM - cp4 Nginx Backend for phorge1 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.38.134.38: Connection reset by peer [03:50:30] PROBLEM - cp4 HTTP 4xx/5xx ERROR Rate on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.38.134.38: Connection reset by peer [03:50:30] PROBLEM - cp3 Nginx Backend for mw1 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [03:50:36] PROBLEM - cp3 Puppet on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [03:50:36] PROBLEM - cp3 Current Load on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [03:50:37] PROBLEM - cp3 Nginx Backend for puppet1 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [03:50:38] PROBLEM - cp4 NTP time on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.38.134.38: Connection reset by peer [03:50:41] PROBLEM - cp3 APT on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [03:50:44] PROBLEM - cp3 PowerDNS Recursor on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [03:50:48] PROBLEM - cp4 Nginx Backend for test1 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.38.134.38: Connection reset by peer [03:50:48] PROBLEM - cp4 Disk Space on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.38.134.38: Connection reset by peer [03:50:51] PROBLEM - cp3 Nginx Backend for mwdedi2 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [03:50:52] PROBLEM - cp4 PowerDNS Recursor on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.38.134.38: Connection reset by peer [03:50:53] PROBLEM - cp3 Disk Space on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [03:50:54] PROBLEM - cp3 Nginx Backend for mwdedi1 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [03:51:00] PROBLEM - cp4 Nginx Backend for mwdedi1 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.38.134.38: Connection reset by peer [03:51:03] PROBLEM - cp4 Nginx Backend for mw1 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.38.134.38: Connection reset by peer [03:51:03] PROBLEM - cp4 Nginx Backend for puppet1 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.38.134.38: Connection reset by peer [03:51:03] PROBLEM - cp3 NTP time on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [03:51:10] PROBLEM - cp4 Nginx Backend for mwdedi2 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.38.134.38: Connection reset by peer [03:51:11] PROBLEM - cp3 Nginx Backend for phorge1 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [03:51:12] PROBLEM - cp4 Nginx Backend for mw2 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.38.134.38: Connection reset by peer [03:51:14] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.38.134.38: Connection reset by peer [03:51:14] PROBLEM - cp3 Nginx Backend for test1 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [03:51:15] PROBLEM - cp3 HTTP 4xx/5xx ERROR Rate on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [05:16:01] [02WikiForge/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/c421164cfcb0...6f1fa6431eef [05:16:04] [02WikiForge/puppet] 07AgentIsai 036f1fa64 - Add matomo1 [05:39:29] [02WikiForge/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/6f1fa6431eef...321368052183 [05:39:32] [02WikiForge/puppet] 07AgentIsai 033213680 - swap for wikiforge.net [05:50:24] PROBLEM - cp5 HTTP 4xx/5xx ERROR Rate on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [05:50:26] PROBLEM - cp5 Current Load on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [05:50:28] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [05:50:29] PROBLEM - cp5 Nginx Backend for mw1 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [05:50:35] PROBLEM - cp5 Nginx Backend for mwdedi1 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [05:50:35] PROBLEM - cp5 NTP time on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [05:50:36] PROBLEM - cp5 Nginx Backend for mw2 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [05:50:37] PROBLEM - cp5 Nginx Backend for matomo1 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [05:50:43] PROBLEM - cp5 APT on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [05:50:44] [02WikiForge/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/321368052183...22f217761031 [05:50:46] [02WikiForge/puppet] 07AgentIsai 0322f2177 - Add your.wf to nginx [05:50:49] PROBLEM - cp5 Nginx Backend for puppet1 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [05:50:50] PROBLEM - cp3 Nginx Backend for matomo1 on cp3 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.68.214.246: Connection reset by peer [05:50:54] PROBLEM - cp5 Nginx Backend for test1 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [05:50:55] PROBLEM - cp4 Nginx Backend for matomo1 on cp4 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.38.134.38: Connection reset by peer [05:50:56] PROBLEM - cp5 Puppet on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [05:51:02] PROBLEM - cp5 Nginx Backend for mail1 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [05:51:04] PROBLEM - cp5 Disk Space on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [05:51:04] PROBLEM - cp2 Nginx Backend for matomo1 on cp2 is CRITICAL: CHECK_NRPE: Error - Could not connect to 158.69.62.132: Connection reset by peer [05:51:09] PROBLEM - cp5 Nginx Backend for mwdedi2 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [05:51:16] PROBLEM - cp5 PowerDNS Recursor on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [05:51:19] PROBLEM - cp5 Nginx Backend for phorge1 on cp5 is CRITICAL: CHECK_NRPE: Error - Could not connect to 15.235.192.162: Connection reset by peer [05:52:33] [02WikiForge/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/22f217761031...7b41062cb2bd [05:52:34] [02WikiForge/puppet] 07AgentIsai 037b41062 - Add error_page to mediawiki-includes [05:55:25] [02WikiForge/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/7b41062cb2bd...322437378452 [05:55:28] [02WikiForge/puppet] 07AgentIsai 033224373 - Add inside.wf to nginx [06:03:24] [02WikiForge/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/322437378452...c47c41dbe511 [06:03:25] [02WikiForge/puppet] 07AgentIsai 03c47c41d - Revert [06:05:11] [02WikiForge/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/c47c41dbe511...90d52860dc44 [06:05:14] [02WikiForge/puppet] 07AgentIsai 0390d5286 - Pool mw1 and mw2 [06:07:54] [02WikiForge/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/90d52860dc44...3c032ba40927 [06:07:57] [02WikiForge/puppet] 07AgentIsai 033c032ba - Add your.wf to Varnish [06:12:18] [02WikiForge/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/puppet/compare/3c032ba40927...c52f9fee2ad8 [06:12:19] [02WikiForge/puppet] 07AgentIsai 03c52f9fe - Fix root name [06:43:31] [02WikiForge/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiForge/mw-config/compare/a8163d4f90b7...469e51b885dc [06:43:33] [02WikiForge/mw-config] 07AgentIsai 03469e51b - Add wgCdnServers [06:43:53] !log [agent@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, mw3, jobrunner1] [06:43:54] !log [agent@jobrunner1] DEPLOY ABORTED: Canary check failed for hub.wikiforge.net@mw1.inside.wf [06:43:57] !log [agent@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, mw3, jobrunner1] [06:44:00] !log [agent@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'force': True, 'versions': ['1.40']} to [mw1, mw2, mw3, jobrunner1] - SUCCESS in 2s [06:44:30] WikiForge/mw-config - AgentIsai the build passed. [06:50:21] PROBLEM - cp6 Nginx Backend for phorge1 on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer [06:50:23] PROBLEM - cp6 Puppet on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer [06:50:23] PROBLEM - cp6 Current Load on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer [06:50:28] PROBLEM - cp6 Nginx Backend for mw1 on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer [06:50:31] PROBLEM - cp6 HTTP 4xx/5xx ERROR Rate on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer [06:50:34] PROBLEM - cp6 PowerDNS Recursor on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer [06:50:38] PROBLEM - cp6 Disk Space on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer [06:50:42] PROBLEM - cp6 Nginx Backend for puppet1 on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer [06:50:48] PROBLEM - cp6 NTP time on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer [06:50:59] PROBLEM - cp6 Nginx Backend for matomo1 on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer [06:51:01] PROBLEM - cp6 Nginx Backend for mw2 on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer [06:51:02] PROBLEM - cp6 Nginx Backend for test1 on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer [06:51:05] PROBLEM - cp6 Nginx Backend for mwdedi1 on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer [06:51:11] PROBLEM - cp6 Nginx Backend for mwdedi2 on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer [06:51:12] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer [06:51:13] PROBLEM - cp6 APT on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer [06:51:13] PROBLEM - cp6 Nginx Backend for mail1 on cp6 is CRITICAL: CHECK_NRPE: Error - Could not connect to 51.161.131.113: Connection reset by peer