[00:02:05] RECOVERY - cp1 Disk Space on cp1 is OK: DISK OK - free space: / 6327MiB (11% inode=98%); [00:13:35] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [00:24:19] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/73dfcdf72dba...7dbeb406f3bc [00:24:20] [02WikiTideOrg/puppet] 07Universal-Omega 037dbeb40 - -wfglobal [00:26:02] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/7dbeb406f3bc...5831e7b0a7a7 [00:26:05] [02WikiTideOrg/puppet] 07Universal-Omega 035831e7b - -wfglobal/prodglobal [01:34:32] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [02:17:36] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [02:33:49] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [02:34:01] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [03:03:11] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [03:46:54] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [03:59:51] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/WikiTideOrg/puppet/compare/5831e7b0a7a7...c4c04f1a7c29 [03:59:54] [02WikiTideOrg/puppet] 07Universal-Omega 03c4c04f1 - Cleanup some private config [04:01:44] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/c4c04f1a7c29...76b7b8c2b8c9 [04:01:45] [02WikiTideOrg/puppet] 07Universal-Omega 0376b7b8c - Use single quotes [05:34:58] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [06:55:04] [02WikiTideOrg/dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/dns/compare/860554ff7413...593f04ec38c4 [06:55:05] [02WikiTideOrg/dns] 07Universal-Omega 03593f04e - Add DNS record [07:28:19] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=leoniawikitide --no-updates --username-prefix=fandom:leonia /home/reception/{file} (START) [07:28:20] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=leoniawikitide --no-updates --username-prefix=fandom:leonia /home/reception/{file} (END - exit=256) [07:28:21] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=leoniawikitide (START) [07:28:22] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [07:28:22] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=leoniawikitide (END - exit=0) [07:28:23] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=leoniawikitide --active --update (END - exit=0) [07:28:25] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [07:28:28] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [07:28:30] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [07:28:33] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [07:40:45] [02WikiTideOrg/mw-config] 07Reception123 pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/ca908b7e2bc5...4b6e27682907 [07:40:48] [02WikiTideOrg/mw-config] 07Reception123 034b6e276 - use file-path/file-name for ImportDump script [07:41:04] !log [reception@jobrunner1] starting deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] [07:41:07] !log [reception@jobrunner1] finished deploy of {'pull': 'config', 'config': True, 'versions': ['1.40']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [07:41:07] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [07:41:10] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [07:41:36] WikiTideOrg/mw-config - Reception123 the build passed. [07:42:24] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=leoniawikitide --no-updates --username-prefix=fandom:leonia /home/reception/leoniawikitide-20231124141631.xml (START) [07:42:28] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [07:43:54] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/importDump.php --wiki=leoniawikitide --no-updates --username-prefix=fandom:leonia /home/reception/leoniawikitide-20231124141631.xml (END - exit=0) [07:43:55] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=leoniawikitide (START) [07:43:56] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [07:43:59] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [07:45:43] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/rebuildall.php --wiki=leoniawikitide (END - exit=0) [07:45:44] !log [reception@jobrunner1] sudo -u www-data php /srv/mediawiki/1.40/maintenance/run.php /srv/mediawiki/1.40/maintenance/initSiteStats.php --wiki=leoniawikitide --active --update (END - exit=0) [07:45:46] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [07:45:48] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [08:01:07] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [08:01:08] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [08:01:10] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [08:01:13] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [08:01:41] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [08:46:05] PROBLEM - cp1 Disk Space on cp1 is WARNING: DISK WARNING - free space: / 6073MiB (10% inode=98%); [09:27:02] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [12:55:23] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [13:59:32] PROBLEM - puppet1 Disk Space on puppet1 is WARNING: DISK WARNING - free space: / 878MiB (10% inode=70%); [15:33:16] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [16:32:40] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [16:35:01] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [17:04:31] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.405 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [17:33:38] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [17:36:51] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [18:28:09] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/WikiTideMagic/compare/24fcb5f00df6...d0df16fe148b [18:28:10] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega 03d0df16f - Fix CentralAuth tables being deleted when wiki is deleted [18:32:46] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/WikiTideMagic/compare/d0df16fe148b...012d1f525af6 [18:32:49] [02WikiTideOrg/WikiTideMagic] 07Universal-Omega 03012d1f5 - Remove $wgAWSBucketName from tests [18:36:23] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [18:48:20] [02WikiTideOrg/mw-config] 07Universal-Omega pushed 031 commit to 03main [+0/-0/±1] 13https://github.com/WikiTideOrg/mw-config/compare/4b6e27682907...7fdcdb214e1c [18:48:22] [02WikiTideOrg/mw-config] 07Universal-Omega 037fdcdb2 - Remove wgCreateWikiUseSecureContainers and add wgCreateWikiUseJobQueue [18:48:26] !log [@jobrunner1] starting deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] [18:48:29] !log [@jobrunner1] finished deploy of {'config': True, 'versions': ['']} to [mw1, mw2, jobrunner1] - SUCCESS in 2s [18:48:30] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [18:48:33] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [18:49:10] WikiTideOrg/mw-config - Universal-Omega the build passed. [19:01:13] !log [@test1] starting deploy of {'config': True, 'versions': ['']} to [test1] [19:01:14] !log [@test1] finished deploy of {'config': True, 'versions': ['']} to [test1] - SUCCESS in 0s [19:01:16] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:01:19] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [19:02:35] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/76b7b8c2b8c9...04c15ddb91e7 [19:02:38] [02WikiTideOrg/puppet] 07Universal-Omega 0304c15dd - Don't use --no-log [19:03:29] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [19:07:19] PROBLEM - mail1 Puppet on mail1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 24 minutes ago with 1 failures [19:08:13] PROBLEM - test1 Puppet on test1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 6 minutes ago with 7 failures [19:08:48] PROBLEM - graylog1 Puppet on graylog1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 3 minutes ago with 0 failures [19:08:50] PROBLEM - cp3 Puppet on cp3 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 18 minutes ago with 0 failures [19:08:52] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 17 minutes ago with 0 failures [19:08:52] PROBLEM - os1 Puppet on os1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 12 minutes ago with 0 failures [19:09:05] PROBLEM - swiftproxy1 Puppet on swiftproxy1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 9 minutes ago with 0 failures [19:09:10] PROBLEM - cp4 Puppet on cp4 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 30 minutes ago with 0 failures [19:09:11] PROBLEM - cloud1 Puppet on cloud1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 29 minutes ago with 0 failures [19:09:34] PROBLEM - cp5 Puppet on cp5 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 17 minutes ago with 0 failures [19:09:40] PROBLEM - cp6 Puppet on cp6 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 22 minutes ago with 0 failures [19:09:44] PROBLEM - phorge1 Puppet on phorge1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 29 minutes ago with 0 failures [19:09:44] PROBLEM - prometheus1 Puppet on prometheus1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 29 minutes ago with 0 failures [19:09:45] PROBLEM - swiftobject1 Puppet on swiftobject1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 25 minutes ago with 0 failures [19:09:53] PROBLEM - db1 Puppet on db1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 14 minutes ago with 0 failures [19:09:59] PROBLEM - jobrunner1 Puppet on jobrunner1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 21 minutes ago with 0 failures [19:10:07] PROBLEM - bast1 Puppet on bast1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 13 minutes ago with 0 failures [19:10:07] PROBLEM - mem1 Puppet on mem1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 5 minutes ago with 0 failures [19:10:16] PROBLEM - puppet1 Puppet on puppet1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 21 minutes ago with 0 failures [19:10:23] PROBLEM - jobchron1 Puppet on jobchron1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 23 minutes ago with 0 failures [19:10:26] PROBLEM - services1 Puppet on services1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 4 minutes ago with 0 failures [19:10:31] PROBLEM - bots1 Puppet on bots1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 21 minutes ago with 0 failures [19:10:33] PROBLEM - cp2 Puppet on cp2 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 18 minutes ago with 0 failures [19:10:35] PROBLEM - ldap1 Puppet on ldap1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 29 minutes ago with 0 failures [19:10:41] PROBLEM - swiftac1 Puppet on swiftac1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 10 minutes ago with 0 failures [19:13:40] PROBLEM - cp1 Puppet on cp1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 18 minutes ago with 0 failures [19:15:17] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 12 minutes ago with 0 failures [19:15:17] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 8 minutes ago with 0 failures [19:27:28] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/04c15ddb91e7...02ed598fb79b [19:27:30] [02WikiTideOrg/puppet] 07Universal-Omega 0302ed598 - Add Icinga2 to firewall rule [19:29:52] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/02ed598fb79b...357c3297e1b3 [19:29:55] [02WikiTideOrg/puppet] 07Universal-Omega 03357c329 - Add icinga2 to firewall [19:31:38] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/357c3297e1b3...3adc07e39a27 [19:31:39] [02WikiTideOrg/puppet] 07Universal-Omega 033adc07e - Add icinga2 to firewall rule [19:34:38] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/3adc07e39a27...dd8a3f199b29 [19:34:41] [02WikiTideOrg/puppet] 07Universal-Omega 03dd8a3f1 - Add Prometheus and Icinga2 to MW firewall [19:38:23] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/dd8a3f199b29...41a6ee2bf644 [19:38:25] [02WikiTideOrg/puppet] 07Universal-Omega 0341a6ee2 - Add icinga2 to firewall [19:39:07] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [19:40:44] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/41a6ee2bf644...ddceaf0ff9ff [19:40:46] [02WikiTideOrg/puppet] 07Universal-Omega 03ddceaf0 - Add icinga2 to firewall rule [19:46:20] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/ddceaf0ff9ff...602879340666 [19:46:22] [02WikiTideOrg/puppet] 07Universal-Omega 036028793 - puppetdb: add firewall rule for icinga2 [19:51:09] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/602879340666...4248329136de [19:51:11] [02WikiTideOrg/puppet] 07Universal-Omega 034248329 - opensearch: add ip4 to firewall [19:51:54] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/4248329136de...2e6aa9e8be6f [19:51:55] [02WikiTideOrg/puppet] 07Universal-Omega 032e6aa9e - elasticsearch: add ip4 to firewall [19:54:17] [02puppet] 07Universal-Omega closed pull request 03#19: Enable ferm - 13https://github.com/WikiTideOrg/puppet/pull/19 [19:54:19] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/2e6aa9e8be6f...bfe2c0a9c3a0 [19:54:22] [02WikiTideOrg/puppet] 07Universal-Omega 03bfe2c0a - Enable ferm [19:54:23] [02WikiTideOrg/puppet] 07Universal-Omega deleted branch 03ferm [19:54:25] [02puppet] 07Universal-Omega deleted branch 03ferm - 13https://github.com/WikiTideOrg/puppet [19:55:44] RECOVERY - phorge1 Puppet on phorge1 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [20:00:07] RECOVERY - bast1 Puppet on bast1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [20:08:50] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [20:10:15] PROBLEM - mon1 Puppet on mon1 is UNKNOWN: NRPE: Unable to read output [20:11:00] PROBLEM - mon1 APT on mon1 is CRITICAL: APT CRITICAL: 33 packages available for upgrade (2 critical updates). [20:12:05] PROBLEM - mon1 Check correctness of the icinga configuration on mon1 is CRITICAL: Icinga configuration contains errors [20:12:07] PROBLEM - cp4 Puppet on cp4 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:16:08] RECOVERY - mon1 Check correctness of the icinga configuration on mon1 is OK: Icinga configuration is correct [20:16:22] PROBLEM - mon1 ferm_active on mon1 is CRITICAL: ERROR ferm input drop default policy not set, ferm might not have been started correctly [20:16:26] PROBLEM - mon1 Puppet on mon1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 4 minutes ago with 4 failures [20:16:33] PROBLEM - cp4 Puppet on cp4 is WARNING: WARNING: Puppet is currently disabled, message: Universal Omega -- testing ferm, last run 1 hour ago with 0 failures [20:27:26] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/bfe2c0a9c3a0...54ea3f8daebe [20:27:28] [02WikiTideOrg/puppet] 07Universal-Omega 0354ea3f8 - Don't install icingaweb2-module-monitoring [20:28:14] PROBLEM - mon1 Puppet on mon1 is UNKNOWN: NRPE: Unable to read output [20:28:26] RECOVERY - mon1 ferm_active on mon1 is OK: OK ferm input default policy is set [20:31:57] PROBLEM - mon1 Puppet on mon1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 3 minutes ago with 0 failures [20:32:14] RECOVERY - puppet1 Puppet on puppet1 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures [20:33:40] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 12 seconds ago with 0 failures [20:35:45] PROBLEM - newcascadia.net - reverse DNS on sslhost is WARNING: LifetimeTimeout: The resolution lifetime expired after 5.404 seconds: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered The DNS operation timed out. [20:36:26] PROBLEM - cp1 Varnish Backends on cp1 is CRITICAL: 3 backends are down. mw1 mw2 mediawiki [20:36:28] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:36:33] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 38 seconds ago with 0 failures [20:37:52] PROBLEM - cp1 HTTPS on cp1 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2571 bytes in 0.014 second response time [20:42:25] PROBLEM - cp2 Disk Space on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:42:44] PROBLEM - cp2 Nginx Backend for phorge1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:42:44] PROBLEM - cp2 APT on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:42:44] PROBLEM - cp2 NTP time on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:42:57] PROBLEM - cp2 Nginx Backend for mail1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:42:57] PROBLEM - cp2 Nginx Backend for puppet1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:43:01] PROBLEM - cp2 Nginx Backend for matomo1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:43:06] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:43:08] PROBLEM - cp2 Nginx Backend for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:43:09] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:43:11] PROBLEM - cp2 Nginx Backend for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:43:14] PROBLEM - cp2 Current Load on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:43:18] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:43:21] PROBLEM - cp2 Nginx Backend for test1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [20:46:27] RECOVERY - cp2 Current Load on cp2 is OK: LOAD OK - total load average: 0.52, 0.37, 0.28 [20:46:37] RECOVERY - cp2 NTP time on cp2 is OK: NTP OK: Offset -0.003519684076 secs [20:46:45] PROBLEM - cp2 Puppet on cp2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 57 seconds ago with 0 failures [20:46:54] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 2% [20:47:18] RECOVERY - cp2 Disk Space on cp2 is OK: DISK OK - free space: / 6295MiB (32% inode=94%); [20:47:38] RECOVERY - cp2 APT on cp2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [20:47:40] RECOVERY - cp2 Nginx Backend for phorge1 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8200 [20:47:53] RECOVERY - cp2 Nginx Backend for matomo1 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8204 [20:47:54] RECOVERY - cp2 Nginx Backend for mail1 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8202 [20:47:55] RECOVERY - cp2 Nginx Backend for puppet1 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8201 [20:47:56] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [20:48:02] RECOVERY - cp2 Nginx Backend for mw1 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8100 [20:48:04] RECOVERY - cp2 Nginx Backend for mw2 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8101 [20:48:06] RECOVERY - cp2 Nginx Backend for test1 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8180 [20:55:14] RECOVERY - cp1 HTTPS on cp1 is OK: HTTP OK: HTTP/2 200 - 2916 bytes in 0.041 second response time [20:55:21] RECOVERY - cp1 Varnish Backends on cp1 is OK: All 7 backends are healthy [20:56:34] PROBLEM - mw1 Puppet on mw1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 19 minutes ago with 0 failures [20:57:38] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 24 minutes ago with 0 failures [20:57:39] !log (logging for the record) ferm broke mw connection with cp1 (but not other cps) and cp2 (and probably other cps) connection with backends. [20:57:42] Logged the message at https://meta.wikitide.org/wiki/Tech:Server_admin_log [21:01:47] RECOVERY - db1 Puppet on db1 is OK: OK: Puppet is currently enabled, last run 28 seconds ago with 0 failures [21:03:56] PROBLEM - nexttide.org - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query nexttide.org. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [21:06:49] RECOVERY - graylog1 Puppet on graylog1 is OK: OK: Puppet is currently enabled, last run 21 seconds ago with 0 failures [21:09:45] RECOVERY - os1 Puppet on os1 is OK: OK: Puppet is currently enabled, last run 25 seconds ago with 0 failures [21:11:17] RECOVERY - bots1 Puppet on bots1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:14:11] RECOVERY - mem1 Puppet on mem1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:15:32] RECOVERY - swiftac1 Puppet on swiftac1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:17:16] RECOVERY - ldap1 Puppet on ldap1 is OK: OK: Puppet is currently enabled, last run 1 second ago with 0 failures [21:19:26] RECOVERY - services1 Puppet on services1 is OK: OK: Puppet is currently enabled, last run 32 seconds ago with 0 failures [21:20:40] RECOVERY - jobchron1 Puppet on jobchron1 is OK: OK: Puppet is currently enabled, last run 3 seconds ago with 0 failures [21:22:33] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [21:23:26] RECOVERY - prometheus1 Puppet on prometheus1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [21:25:50] RECOVERY - swiftobject1 Puppet on swiftobject1 is OK: OK: Puppet is currently enabled, last run 14 seconds ago with 0 failures [21:26:57] PROBLEM - mail1 Puppet on mail1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 11 seconds ago with 1 failures. Failed resources (up to 3 shown): File[/etc/opendkim/keys/mail.private] [21:30:43] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [21:32:06] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 2 minutes ago with 0 failures [21:34:32] RECOVERY - swiftproxy1 Puppet on swiftproxy1 is OK: OK: Puppet is currently enabled, last run 10 seconds ago with 0 failures [21:37:43] PROBLEM - test1 Puppet on test1 is CRITICAL: CRITICAL: Puppet has 7 failures. Last run 8 seconds ago with 7 failures. Failed resources (up to 3 shown): File[/srv/mediawiki/1.39/skins/Femiwiki/node_modules],File[/srv/mediawiki/1.40/skins/Femiwiki/node_modules],File[/srv/mediawiki/1.41/skins/Femiwiki/node_modules],Exec[DataTransfer-REL1_39 composer] [21:38:11] PROBLEM - valorantjp.wiki - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query valorantjp.wiki. IN NS: Server 1.1.1.1 UDP port 53 answered The DNS operation timed out.; Server 1.1.1.1 UDP port 53 answered SERVFAIL [22:02:43] RECOVERY - nexttide.org - reverse DNS on sslhost is OK: SSL OK - nexttide.org reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [22:06:55] RECOVERY - valorantjp.wiki - reverse DNS on sslhost is OK: SSL OK - valorantjp.wiki reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [22:09:38] RECOVERY - puppet1 puppetdb on puppet1 is OK: TCP OK - 0.000 second response time on 10.0.0.100 port 8081 [22:10:22] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:12:33] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures [22:13:34] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [22:14:48] PROBLEM - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is CRITICAL: CRITICAL - NGINX Error Rate is 76% [22:15:01] PROBLEM - cp1 Varnish Backends on cp1 is CRITICAL: 3 backends are down. mw1 mw2 mediawiki [22:15:32] PROBLEM - cp2 Nginx Backend for matomo1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:15:36] PROBLEM - ns1 Puppet on ns1 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:15:43] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:16:00] PROBLEM - cp2 NTP time on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:16:03] PROBLEM - cp2 Disk Space on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:16:08] PROBLEM - cp2 APT on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:16:11] PROBLEM - cp2 Nginx Backend for mail1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:16:15] PROBLEM - cp2 Nginx Backend for mw1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:16:19] PROBLEM - ns1 Puppet on ns1 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 1 minute ago with 0 failures [22:16:23] PROBLEM - cp2 Nginx Backend for puppet1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:16:24] PROBLEM - cp1 HTTPS on cp1 is CRITICAL: HTTP CRITICAL: HTTP/2 503 - 2571 bytes in 0.015 second response time [22:16:33] PROBLEM - cp2 Nginx Backend for phorge1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:16:33] PROBLEM - cp2 Current Load on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:16:34] PROBLEM - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:16:44] PROBLEM - cp2 Nginx Backend for mw2 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:16:48] RECOVERY - cp1 HTTP 4xx/5xx ERROR Rate on cp1 is OK: OK - NGINX Error Rate is 27% [22:17:03] PROBLEM - cp2 Nginx Backend for test1 on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:17:18] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [22:18:20] RECOVERY - cp1 HTTPS on cp1 is OK: HTTP OK: HTTP/2 200 - 2939 bytes in 0.031 second response time [22:18:42] RECOVERY - cp2 Nginx Backend for mw2 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8101 [22:18:43] RECOVERY - cp2 Nginx Backend for mail1 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8202 [22:18:43] RECOVERY - cp2 Nginx Backend for mw1 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8100 [22:18:45] RECOVERY - cp2 Current Load on cp2 is OK: LOAD OK - total load average: 0.00, 0.04, 0.06 [22:18:47] RECOVERY - cp2 Nginx Backend for phorge1 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8200 [22:18:47] RECOVERY - cp2 HTTP 4xx/5xx ERROR Rate on cp2 is OK: OK - NGINX Error Rate is 5% [22:18:53] RECOVERY - cp2 Nginx Backend for puppet1 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8201 [22:18:59] RECOVERY - cp2 Nginx Backend for test1 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8180 [22:19:14] PROBLEM - cp2 Puppet on cp2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 8 minutes ago with 0 failures [22:19:34] PROBLEM - mw2 Puppet on mw2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 5 minutes ago with 0 failures [22:20:24] RECOVERY - cp2 Nginx Backend for matomo1 on cp2 is OK: TCP OK - 0.000 second response time on localhost port 8204 [22:20:33] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [22:20:47] RECOVERY - cp2 NTP time on cp2 is OK: NTP OK: Offset -0.004055023193 secs [22:20:55] RECOVERY - cp2 Disk Space on cp2 is OK: DISK OK - free space: / 6285MiB (32% inode=94%); [22:21:03] RECOVERY - cp2 APT on cp2 is OK: APT OK: 1 packages available for upgrade (0 critical updates). [22:25:48] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/commit/5a3f34cace57 [22:25:50] [02WikiTideOrg/puppet] 07Universal-Omega 035a3f34c - Fix firewall [22:25:53] [02puppet] 07Universal-Omega created branch 03fix-firewall - 13https://github.com/WikiTideOrg/puppet [22:25:55] [02puppet] 07Universal-Omega opened pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [22:26:25] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/5a3f34cace57...b94fd04f26c3 [22:26:27] [02WikiTideOrg/puppet] 07Universal-Omega 03b94fd04 - Fix firewall [22:26:30] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [22:30:34] PROBLEM - mw1 Puppet on mw1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:31:52] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [22:31:53] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/b94fd04f26c3...6ad40e0e62dd [22:31:56] [02WikiTideOrg/puppet] 07Universal-Omega 036ad40e0 - Fix [22:32:28] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [22:32:29] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/6ad40e0e62dd...cf537833c7a7 [22:32:30] [02WikiTideOrg/puppet] 07Universal-Omega 03cf53783 - Fix [22:34:26] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [22:34:29] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/cf537833c7a7...bb605549cb59 [22:34:31] [02WikiTideOrg/puppet] 07Universal-Omega 03bb60554 - Fix [22:34:51] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [22:34:54] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/bb605549cb59...cae6bfd27aac [22:34:56] [02WikiTideOrg/puppet] 07Universal-Omega 03cae6bfd - Fix [22:36:28] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [22:36:30] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/cae6bfd27aac...97eaa6e2bf52 [22:36:33] [02WikiTideOrg/puppet] 07Universal-Omega 0397eaa6e - Fix [22:37:25] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [22:37:27] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/97eaa6e2bf52...d71d018903a1 [22:37:29] [02WikiTideOrg/puppet] 07Universal-Omega 03d71d018 - Fix [22:38:07] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [22:38:10] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/d71d018903a1...fd79f47ef76e [22:38:12] [02WikiTideOrg/puppet] 07Universal-Omega 03fd79f47 - Fix [22:50:37] PROBLEM - jobrunner1 Puppet on jobrunner1 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [22:53:31] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [22:53:33] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/fd79f47ef76e...edc89440ac75 [22:53:35] [02WikiTideOrg/puppet] 07Universal-Omega 03edc8944 - Fix [22:58:26] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [22:58:27] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/edc89440ac75...9f5fd23edf5b [22:58:29] [02WikiTideOrg/puppet] 07Universal-Omega 039f5fd23 - Fix [22:59:25] PROBLEM - mw1 MediaWiki Rendering on mw1 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [23:00:18] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [23:00:20] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/9f5fd23edf5b...c3e6dfd8f54d [23:00:23] [02WikiTideOrg/puppet] 07Universal-Omega 03c3e6dfd - fix [23:00:33] RECOVERY - mw1 Puppet on mw1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:01:21] PROBLEM - cp4 Varnish Backends on cp4 is CRITICAL: 1 backends are down. mw1 [23:01:51] PROBLEM - cp2 Varnish Backends on cp2 is CRITICAL: 1 backends are down. mw1 [23:01:57] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [23:02:00] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/c3e6dfd8f54d...a7f850a0c8f4 [23:02:02] [02WikiTideOrg/puppet] 07Universal-Omega 03a7f850a - Fix [23:02:05] PROBLEM - cp5 Varnish Backends on cp5 is CRITICAL: 1 backends are down. mw1 [23:02:40] PROBLEM - cp6 Varnish Backends on cp6 is CRITICAL: 1 backends are down. mw1 [23:03:00] RECOVERY - cp1 Varnish Backends on cp1 is OK: All 7 backends are healthy [23:03:06] RECOVERY - cp4 Varnish Backends on cp4 is OK: All 7 backends are healthy [23:03:11] RECOVERY - cp2 Varnish Backends on cp2 is OK: All 7 backends are healthy [23:03:17] RECOVERY - cp5 Varnish Backends on cp5 is OK: All 7 backends are healthy [23:03:21] RECOVERY - cp6 Varnish Backends on cp6 is OK: All 7 backends are healthy [23:03:28] PROBLEM - mw1 MediaWiki Rendering on mw1 is WARNING: HTTP WARNING: HTTP/1.1 404 Not Found - 8191 bytes in 0.327 second response time [23:04:08] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [23:04:10] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/a7f850a0c8f4...936d3a86befb [23:04:13] [02WikiTideOrg/puppet] 07Universal-Omega 03936d3a8 - Revert [23:06:57] RECOVERY - cp1 Puppet on cp1 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [23:11:05] PROBLEM - cp2 Puppet on cp2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [23:13:01] PROBLEM - cp2 Puppet on cp2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 3 minutes ago with 0 failures [23:14:37] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/936d3a86befb...e19e72c08977 [23:14:40] [02WikiTideOrg/puppet] 07Universal-Omega 03e19e72c - Fix [23:14:42] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [23:14:57] RECOVERY - cp2 Puppet on cp2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:15:33] WikiTideOrg/puppet - Universal-Omega the build has errored. [23:19:34] RECOVERY - mw2 Puppet on mw2 is OK: OK: Puppet is currently enabled, last run 37 seconds ago with 0 failures [23:19:44] PROBLEM - cp2 Puppet on cp2 is WARNING: WARNING: Puppet is currently disabled, message: reason not specified, last run 4 minutes ago with 0 failures [23:20:19] RECOVERY - jobrunner1 Puppet on jobrunner1 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [23:32:05] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/e19e72c08977...372e04325bd0 [23:32:08] [02WikiTideOrg/puppet] 07Universal-Omega 03372e043 - - [23:32:10] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [23:33:01] WikiTideOrg/puppet - Universal-Omega the build has errored. [23:33:27] RECOVERY - test1 JobChron Service on test1 is OK: PROCS OK: 1 process with args 'redisJobChronService' [23:34:04] RECOVERY - newcascadia.net - reverse DNS on sslhost is OK: SSL OK - newcascadia.net reverse DNS resolves to cp1.wikitide.net - NS RECORDS OK [23:37:28] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/54ea3f8daebe...71e3955db3b2 [23:37:29] [02WikiTideOrg/puppet] 07Universal-Omega 0371e3955 - Remove unused [23:46:17] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+1/-0/±0] 13https://github.com/WikiTideOrg/puppet/compare/372e04325bd0...8626eeac18bd [23:46:20] [02WikiTideOrg/puppet] 07Universal-Omega 038626eea - Add varnish_public_ip facter [23:46:21] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [23:47:16] WikiTideOrg/puppet - Universal-Omega the build has errored. [23:48:14] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+1/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/8626eeac18bd...3825af04c6da [23:48:15] [02WikiTideOrg/puppet] 07Universal-Omega 033825af0 - Fix and rename [23:48:19] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [23:49:06] WikiTideOrg/puppet - Universal-Omega the build has errored. [23:49:24] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/3825af04c6da...974250e125f0 [23:49:25] [02WikiTideOrg/puppet] 07Universal-Omega 03974250e - Add debugging [23:49:27] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [23:49:34] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/974250e125f0...402b46fe21c1 [23:49:36] [02WikiTideOrg/puppet] 07Universal-Omega 03402b46f - - [23:49:37] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [23:50:14] WikiTideOrg/puppet - Universal-Omega the build has errored. [23:50:24] WikiTideOrg/puppet - Universal-Omega the build has errored. [23:50:55] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/402b46fe21c1...ed5e1f6cc009 [23:50:57] [02WikiTideOrg/puppet] 07Universal-Omega 03ed5e1f6 - Fix [23:50:59] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [23:51:48] WikiTideOrg/puppet - Universal-Omega the build has errored. [23:52:19] PROBLEM - puppet1 Puppet on puppet1 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_puppet] [23:53:26] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+1/-1/±0] 13https://github.com/WikiTideOrg/puppet/compare/ed5e1f6cc009...ccffc1b89e58 [23:53:27] [02WikiTideOrg/puppet] 07Universal-Omega 03ccffc1b - Fix [23:53:30] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [23:53:33] WikiTideOrg/puppet - Universal-Omega the build has errored. [23:58:06] [02WikiTideOrg/puppet] 07Universal-Omega pushed 031 commit to 03fix-firewall [+0/-0/±1] 13https://github.com/WikiTideOrg/puppet/compare/ccffc1b89e58...737085cae711 [23:58:07] [02WikiTideOrg/puppet] 07Universal-Omega 03737085c - Fix [23:58:08] [02puppet] 07Universal-Omega synchronize pull request 03#20: Fix firewall - 13https://github.com/WikiTideOrg/puppet/pull/20 [23:58:59] WikiTideOrg/puppet - Universal-Omega the build has errored.