[00:00:57] miraheze/CreateWiki - Universal-Omega the build passed. [00:02:03] [02mw-config] 07dmehus synchronize pull request 03#4315: On hold reason - 13https://git.io/Jy9RE [00:02:26] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.23, 20.10, 18.56 [00:02:31] [02mw-config] 07AgentIsai commented on pull request 03#4315: On hold reason - 13https://git.io/Jy9Rh [00:02:42] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:03:13] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:03:39] PROBLEM - graylog2 Puppet on graylog2 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 3 minutes ago with 13 failures. Failed resources (up to 3 shown): File[/etc/apt/trusted.gpg.d/graylog-keyring.gpg],File[/etc/prometheus-es-exporter],File[/etc/logrotate.d/puppet],File[/etc/update-motd.d/97-last-puppet-run] [00:03:39] miraheze/mw-config - dmehus the build passed. [00:04:24] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 20.34, 19.82, 18.63 [00:04:42] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.352 second response time [00:05:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.11, 3.80, 3.68 [00:05:11] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 1.360 second response time [00:07:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.95, 4.19, 3.83 [00:09:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.29, 3.67, 3.67 [00:09:18] [02mw-config] 07dmehus commented on pull request 03#4315: On hold reason - 13https://git.io/Jy9Ev [00:09:42] [02mw-config] 07dmehus edited a comment on pull request 03#4315: On hold reason - 13https://git.io/Jy9Ev [00:10:17] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 6 datacenters are down: 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:11:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.93, 4.25, 3.89 [00:12:10] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 22.88, 20.88, 19.36 [00:12:12] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 7 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 2001:41d0:801:2000::4c25/cpweb, 2001:41d0:801:2000::1b80/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:12:19] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:12:35] Agent, or CosmicAlpha, could either of you update the topic in this channel? I think we can probably update the "Status:" to just say "Up" or "Online" or something now :) [00:12:50] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.87, 7.46, 6.84 [00:12:54] .op [00:12:54] Attempting to OP... [00:12:54] !log [@mw11] starting deploy of {'config': True} to ovlon [00:12:58] ty :) [00:13:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.96, 3.73, 3.75 [00:13:21] note you don't technically need to op to update to the topic, as long as you have the "t" flag [00:13:23] .deop [00:13:23] Attempting to OP... [00:13:33] PROBLEM - mw9 MediaWiki Rendering on mw9 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:13:35] dmehus: yeah, or I can always use ChanServ to do my bidding :P [00:13:36] oh you did it through the bot [00:13:39] PROBLEM - cp21 Stunnel Http for mw9 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:42] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:13:43] yeah I use ChanServ :P [00:13:50] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 55s [00:14:14] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:14:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:14:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:15:26] PROBLEM - cp20 Stunnel Http for mw9 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:15:39] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 1.860 second response time [00:16:09] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jy9EB [00:16:11] [02miraheze/mw-config] 07Universal-Omega 030f65b58 - Update Defines.php [00:16:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:16:12] [02mw-config] 07Universal-Omega synchronize pull request 03#4302: set ManageWikiNamespaces' variable defaults - 13https://git.io/JyRnp [00:16:19] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.013 second response time [00:16:20] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 5.674 second response time [00:17:13] miraheze/mw-config - Universal-Omega the build passed. [00:17:32] [02mw-config] 07Universal-Omega closed pull request 03#4302: set ManageWikiNamespaces' variable defaults - 13https://git.io/JyRnp [00:17:34] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 [00:17:35] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy9Ei [00:17:37] [02miraheze/mw-config] 07Universal-Omega 03bce8450 - set ManageWikiNamespaces' variable defaults (#4302) [00:17:38] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [00:17:58] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:18:02] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.15, 22.92, 20.71 [00:18:28] [02mw-config] 07Universal-Omega commented on pull request 03#4315: On hold reason - 13https://git.io/Jy9ES [00:18:32] miraheze/mw-config - Universal-Omega the build passed. [00:19:44] PROBLEM - cp30 Stunnel Http for mw9 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:20:03] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.89, 5.89, 4.73 [00:20:07] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 5 datacenters are down: 198.244.148.90/cpweb, 149.56.140.43/cpweb, 149.56.141.75/cpweb, 2607:5300:201:3100::929a/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:20:51] PROBLEM - cp31 Stunnel Http for mw9 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:21:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.30, 3.50, 3.59 [00:21:26] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.253 second response time [00:21:29] RECOVERY - cp20 Stunnel Http for mw9 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 2.022 second response time [00:21:44] RECOVERY - cp30 Stunnel Http for mw9 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.324 second response time [00:21:49] RECOVERY - cp21 Stunnel Http for mw9 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 0.028 second response time [00:21:52] RECOVERY - mw9 MediaWiki Rendering on mw9 is OK: HTTP OK: HTTP/1.1 200 OK - 20524 bytes in 0.351 second response time [00:21:56] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 4 datacenters are down: 51.195.220.68/cpweb, 198.244.148.90/cpweb, 149.56.140.43/cpweb, 2607:5300:201:3100::5ebc/cpweb [00:21:58] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.02, 5.54, 4.74 [00:22:00] PROBLEM - cp30 Stunnel Http for mw8 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:16] PROBLEM - cp21 Stunnel Http for mw12 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:38] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:22:47] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:22:59] RECOVERY - cp31 Stunnel Http for mw9 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14556 bytes in 8.101 second response time [00:23:01] PROBLEM - cp20 Stunnel Http for mw12 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.58, 3.03, 3.40 [00:23:18] PROBLEM - cp31 Stunnel Http for mw12 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:19] PROBLEM - cp20 Stunnel Http for mw8 on cp20 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:23:31] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 7.460 second response time [00:23:33] PROBLEM - mw8 MediaWiki Rendering on mw8 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:23:53] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.88, 5.74, 4.89 [00:23:54] PROBLEM - cp31 Stunnel Http for mw8 on cp31 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:24:46] !log [@test3] starting deploy of {'config': True} to skip [00:24:47] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [00:25:37] RECOVERY - mw8 MediaWiki Rendering on mw8 is OK: HTTP OK: HTTP/1.1 200 OK - 20514 bytes in 6.756 second response time [00:25:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:26:00] RECOVERY - cp31 Stunnel Http for mw8 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 6.462 second response time [00:26:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:26:07] RECOVERY - cp30 Stunnel Http for mw8 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 6.568 second response time [00:27:58] PROBLEM - cp30 Stunnel Http for mw12 on cp30 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [00:28:44] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.005 second response time [00:28:58] !log [@mw11] starting deploy of {'config': True} to ovlon [00:29:21] RECOVERY - cp20 Stunnel Http for mw8 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.012 second response time [00:29:24] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 26s [00:29:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:29:37] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.60, 5.77, 5.23 [00:29:56] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [00:30:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:30:35] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.97, 6.06, 6.59 [00:31:32] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 12.54, 7.90, 6.04 [00:31:45] RECOVERY - cp31 Stunnel Http for mw12 on cp31 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 9.477 second response time [00:32:02] RECOVERY - cp30 Stunnel Http for mw12 on cp30 is OK: HTTP OK: HTTP/1.1 200 OK - 14565 bytes in 0.311 second response time [00:32:25] RECOVERY - cp21 Stunnel Http for mw12 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.018 second response time [00:33:00] RECOVERY - graylog2 Puppet on graylog2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [00:33:07] RECOVERY - cp20 Stunnel Http for mw12 on cp20 is OK: HTTP OK: HTTP/1.1 200 OK - 14557 bytes in 0.063 second response time [00:33:12] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03revert-4302-Universal-Omega-patch-1 [+0/-0/±1] 13https://git.io/Jy9uc [00:33:13] [02miraheze/mw-config] 07Universal-Omega 0312571cb - Revert "set ManageWikiNamespaces' variable defaults (#4302)" [00:33:15] [02mw-config] 07Universal-Omega created branch 03revert-4302-Universal-Omega-patch-1 - 13https://git.io/vbvb3 [00:33:16] [02mw-config] 07Universal-Omega opened pull request 03#4316: Revert "set ManageWikiNamespaces' variable defaults" - 13https://git.io/Jy9uC [00:33:20] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20526 bytes in 2.914 second response time [00:33:20] [02mw-config] 07Universal-Omega closed pull request 03#4316: Revert "set ManageWikiNamespaces' variable defaults" - 13https://git.io/Jy9uC [00:33:22] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy9uW [00:33:23] [02miraheze/mw-config] 07Universal-Omega 035bbad69 - Revert "set ManageWikiNamespaces' variable defaults (#4302)" (#4316) [00:33:25] [02miraheze/mw-config] 07Universal-Omega deleted branch 03revert-4302-Universal-Omega-patch-1 [00:33:26] [02mw-config] 07Universal-Omega deleted branch 03revert-4302-Universal-Omega-patch-1 - 13https://git.io/vbvb3 [00:33:55] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [00:34:19] miraheze/mw-config - Universal-Omega the build passed. [00:34:34] miraheze/mw-config - Universal-Omega the build passed. [00:35:15] [02mw-config] 07dmehus commented on pull request 03#4315: On hold reason - 13https://git.io/Jy9u2 [00:35:25] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.55, 5.68, 5.56 [00:35:41] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.44, 22.31, 23.46 [00:35:49] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.51, 3.37 [00:35:52] PROBLEM - wiki.raghuveer.net - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.raghuveer.net could not be found [00:39:09] [02mw-config] 07Universal-Omega commented on pull request 03#4315: On hold reason - 13https://git.io/Jy9uQ [00:39:15] [02mw-config] 07Universal-Omega closed pull request 03#4315: On hold reason - 13https://git.io/Jy9RE [00:39:17] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy9u7 [00:39:18] [02miraheze/mw-config] 07dmehus 03e6413b2 - On hold reason (#4315) [00:39:25] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.22, 4.31, 5.03 [00:39:38] [02miraheze/mw-config] 07github-actions[bot] pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy9u5 [00:39:40] [02miraheze/mw-config] 07github-actions 0314f8776 - CI: lint code to MediaWiki standards [00:39:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.62, 3.15, 3.25 [00:40:16] miraheze/mw-config - Universal-Omega the build passed. [00:40:28] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy9ud [00:40:29] [02miraheze/mw-config] 07Universal-Omega 03f37a2d6 - Update LocalSettings.php [00:41:05] !log [@mw11] starting deploy of {'config': True} to ovlon [00:41:26] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 20s [00:41:34] miraheze/mw-config - Universal-Omega the build passed. [00:41:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:42:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:42:08] [02mw-config] 07dmehus commented on pull request 03#4315: On hold reason - 13https://git.io/Jy9up [00:43:35] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.86, 4.03, 3.56 [00:47:28] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.97, 3.72, 3.56 [00:47:40] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [00:49:14] RECOVERY - wiki.raghuveer.net - reverse DNS on sslhost is OK: SSL OK - wiki.raghuveer.net reverse DNS resolves to cp20.miraheze.org - CNAME OK [00:49:22] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.68, 21.29, 21.48 [00:50:15] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 9.86, 7.07, 6.10 [00:51:19] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.17, 21.41, 21.52 [00:52:13] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.67, 6.79, 6.11 [00:54:06] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.16, 5.36, 4.99 [00:54:28] !log [@test3] starting deploy of {'config': True} to skip [00:54:29] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [00:54:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:54:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:12] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.67, 3.12, 3.34 [00:57:53] PROBLEM - wiki.raghuveer.net - reverse DNS on sslhost is WARNING: Traceback (most recent call last): File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 148, in main() File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 129, in main records = check_records(args.hostname) File "/usr/lib/nagios/plugins/check_reverse_dns.py", line 79, in check_records cname = str(dns_resolver.query(hostname, 'CNAME')[0]) File "/usr/lib/pyth [00:57:53] ist-packages/dns/resolver.py", line 1002, in query raise NXDOMAIN(qnames=qnames_to_try, responses=nxdomain_responses)dns.resolver.NXDOMAIN: None of DNS query names exist: wiki.raghuveer.net., wiki.raghuveer.net. [00:57:56] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.19, 4.70, 4.85 [00:59:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.51, 4.00, 3.61 [01:01:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.18, 3.75, 3.57 [01:03:17] is that added error reporting? [01:03:33] I don't think I've seen those tracebacks for custom domain/SSL errors [01:04:52] [02miraheze/dns] 07JohnFLewis pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyHiV [01:04:53] [02miraheze/dns] 07JohnFLewis 038737813 - initial push for SCSVG IP allocations [01:05:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.35, 3.47, 3.46 [01:05:10] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 16.33, 18.59, 20.03 [01:06:29] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JyHQh [01:06:30] [02miraheze/puppet] 07paladox 03572d4d5 - postgresql: Add support for bullseye [01:06:32] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [01:06:33] [02puppet] 07paladox opened pull request 03#2186: postgresql: Add support for bullseye - 13https://git.io/JyH7n [01:07:36] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JyHxY [01:07:38] [02miraheze/puppet] 07paladox 031e98202 - Update master.pp [01:07:39] [02puppet] 07paladox synchronize pull request 03#2186: postgresql: Add support for bullseye - 13https://git.io/JyH7n [01:08:06] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JyHjW [01:08:08] [02miraheze/puppet] 07paladox 039d949e2 - Update user.pp [01:08:09] [02puppet] 07paladox synchronize pull request 03#2186: postgresql: Add support for bullseye - 13https://git.io/JyH7n [01:09:02] [02puppet] 07paladox closed pull request 03#2186: postgresql: Add support for bullseye - 13https://git.io/JyH7n [01:09:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.50, 3.47 [01:09:04] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±3] 13https://git.io/JyQUd [01:09:05] [02miraheze/puppet] 07paladox 03d251983 - postgresql: Add support for bullseye (#2186) [01:09:07] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [01:09:08] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [01:11:18] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JyQZG [01:11:20] [02miraheze/puppet] 07paladox 0327c1d42 - puppetdb: Add support for postgresql under bullseye [01:13:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.90, 3.18, 3.35 [01:14:59] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-0/±0] 13https://git.io/JyQrX [01:15:01] [02miraheze/puppet] 07paladox 037127688 - Setup puppet111 [01:15:02] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [01:15:04] [02puppet] 07paladox opened pull request 03#2187: Setup puppet111 - 13https://git.io/JyQrd [01:15:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JyQoK [01:15:15] [02miraheze/puppet] 07paladox 03a26312b - Update puppet111.yaml [01:15:17] [02puppet] 07paladox synchronize pull request 03#2187: Setup puppet111 - 13https://git.io/JyQrd [01:15:36] paladox, did we ever upgrade to Debian Bullseye? [01:15:43] No [01:15:46] oh [01:15:53] apart from mw8 and mwtask1 [01:15:57] oh and test3 [01:15:59] ah [01:16:09] so just some pieces of MW infra then [01:16:39] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/JyQ1y [01:16:40] [02miraheze/puppet] 07paladox 0385f7694 - Update site.pp [01:16:42] [02puppet] 07paladox synchronize pull request 03#2187: Setup puppet111 - 13https://git.io/JyQrd [01:16:50] dmehus: yeh [01:22:27] RECOVERY - db13 Disk Space on db13 is OK: DISK OK - free space: / 49154 MB (11% inode=98%); [01:25:03] ah [01:27:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.31, 4.05, 3.64 [01:28:01] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy7Ih [01:28:02] [02miraheze/mw-config] 07Universal-Omega 03d10e20c - Update CreateWiki subdomain blacklist entry for bastion [01:28:03] I'm getting 502s and 503s yet I see no icinga warnings [01:28:32] Agent: everything works fine for me. [01:28:43] oh, I see [01:28:52] Agent, surprisingly everything is fine for me, too, and my DT is working again [01:28:53] seems the tab I proxy via test3 is the one that doesn't work [01:29:02] miraheze/mw-config - Universal-Omega the build passed. [01:29:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.64, 3.84, 3.62 [01:31:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.46, 4.03, 3.71 [01:33:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.02, 3.75, 3.66 [01:37:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.01, 4.14, 3.80 [01:39:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.55, 3.76, 3.72 [01:41:41] !log [@mw11] starting deploy of {'config': True} to ovlon [01:41:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:41:52] !log [@mw11] finished deploy of {'config': True} to ovlon - SUCCESS in 10s [01:41:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:43:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.19, 3.63, 3.65 [01:49:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 3.78, 3.71 [01:54:14] !log [@test3] starting deploy of {'config': True} to skip [01:54:15] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [01:54:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:54:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:05:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.94, 3.68, 3.58 [02:13:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.01, 3.57, 3.59 [02:17:03] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.48, 3.50, 3.51 [02:19:03] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.89, 3.50, 3.50 [02:20:27] PROBLEM - db13 Disk Space on db13 is WARNING: DISK WARNING - free space: / 48948 MB (10% inode=98%); [02:21:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.61, 3.66, 3.55 [02:21:34] PROBLEM - cp21 Stunnel Http for mw8 on cp21 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [02:22:26] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.48, 6.70, 6.11 [02:23:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.70, 3.58 [02:23:30] RECOVERY - cp21 Stunnel Http for mw8 on cp21 is OK: HTTP OK: HTTP/1.1 200 OK - 14546 bytes in 0.014 second response time [02:24:25] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.94, 6.24, 6.01 [02:27:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.92, 4.06, 3.72 [02:31:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.83, 3.70 [02:37:03] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jy7B3 [02:37:04] [02miraheze/puppet] 07paladox 038fba149 - base::puppet: Set release to buster [02:37:06] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [02:37:07] [02puppet] 07paladox opened pull request 03#2188: base::puppet: Set release to buster - 13https://git.io/Jy7BZ [02:37:14] [02puppet] 07paladox closed pull request 03#2188: base::puppet: Set release to buster - 13https://git.io/Jy7BZ [02:37:15] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy7Bc [02:37:17] [02miraheze/puppet] 07paladox 032402142 - base::puppet: Set release to buster (#2188) [02:37:18] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [02:37:20] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [02:37:40] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:39:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.16, 2.90, 3.35 [02:40:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [02:53:15] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.16, 7.30, 6.24 [02:55:13] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 6.66, 7.07, 6.28 [02:56:48] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [02:57:03] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 6.91, 6.85, 6.33 [02:57:11] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.27, 6.36, 6.11 [02:58:43] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:00:57] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 4.83, 6.23, 6.22 [03:11:19] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.72, 6.70, 5.58 [03:13:19] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.56, 6.27, 5.57 [03:21:13] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.36, 3.80, 3.38 [03:21:14] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 1 datacenter is down: 149.56.141.75/cpweb [03:22:48] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.47, 6.66, 6.12 [03:23:09] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.00, 3.51, 3.33 [03:23:12] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [03:25:05] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.37, 3.13, 3.21 [03:26:45] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.83, 6.43, 6.16 [03:27:59] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.78, 7.41, 6.38 [03:28:39] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 1 datacenter is down: 2001:41d0:801:2000::1b80/cpweb [03:29:58] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.70, 7.13, 6.40 [03:30:34] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [03:31:58] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.55, 5.77, 5.99 [03:35:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:37:53] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.72, 3.56, 3.31 [03:38:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:39:49] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.22, 3.84, 3.45 [03:43:41] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.33, 3.60, 3.44 [03:45:37] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.06, 3.17, 3.31 [03:53:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 6.18, 5.37, 4.12 [04:00:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jy5l9 [04:00:55] [02miraheze/puppet] 07paladox 03a31b480 - postgresql: Support bullseye part 2 [04:00:57] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://git.io/vbiAS [04:00:58] [02puppet] 07paladox opened pull request 03#2189: postgresql: Support bullseye part 2 - 13https://git.io/Jy5lH [04:01:17] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.70, 3.62, 3.81 [04:01:48] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://git.io/Jy5lQ [04:01:50] [02miraheze/puppet] 07paladox 03c6f7074 - Update master.conf.erb [04:01:52] [02puppet] 07paladox synchronize pull request 03#2189: postgresql: Support bullseye part 2 - 13https://git.io/Jy5lH [04:02:43] [02puppet] 07paladox closed pull request 03#2189: postgresql: Support bullseye part 2 - 13https://git.io/Jy5lH [04:02:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy5lb [04:02:46] [02miraheze/puppet] 07paladox 03fe8e45e - postgresql: Support bullseye part 2 (#2189) [04:02:48] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [04:02:49] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://git.io/vbiAS [04:03:58] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.07, 6.66, 6.23 [04:05:09] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.15, 3.77, 3.83 [04:05:58] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.29, 5.76, 5.95 [04:07:05] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.80, 3.38, 3.68 [04:08:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:11:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.92, 3.56, 3.63 [04:11:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:13:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.87, 3.53, 3.61 [04:17:12] [02puppet] 07paladox closed pull request 03#2187: Setup puppet111 - 13https://git.io/JyQrd [04:17:14] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-0/±1] 13https://git.io/Jy58w [04:17:15] [02miraheze/puppet] 07paladox 03a1ac083 - Setup puppet111 (#2187) [04:17:17] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [04:17:18] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [04:17:51] PROBLEM - dc-multiverse.dcwikis.com - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['ns-cloud-a1.googledomains.com.', 'ns-cloud-a2.googledomains.com.', 'ns-cloud-a3.googledomains.com.', 'ns-cloud-a4.googledomains.com.'], 'CNAME': None} [04:19:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.43, 2.97, 3.35 [04:19:38] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Jy58o [04:19:40] [02miraheze/puppet] 07paladox 03a1f2903 - puppetserver: Fix using rsyslog [04:19:41] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [04:19:43] [02puppet] 07paladox opened pull request 03#2190: puppetserver: Fix using rsyslog - 13https://git.io/Jy58K [04:20:47] [02puppet] 07paladox closed pull request 03#2190: puppetserver: Fix using rsyslog - 13https://git.io/Jy58K [04:20:49] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy58X [04:20:50] [02miraheze/puppet] 07paladox 03659ecad - puppetserver: Fix using rsyslog (#2190) [04:20:52] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [04:20:53] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [04:21:17] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 7.30, 6.81, 6.09 [04:23:13] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 6.43, 6.64, 6.11 [04:26:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy58x [04:26:21] [02miraheze/puppet] 07paladox 032c19c5a - puppetserver: Fix support for rsyslog part 2 [04:36:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:42:40] PROBLEM - wiki.macc.nyc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.macc.nyc' expires in 15 day(s) (Sun 16 Jan 2022 04:38:17 GMT +0000). [04:42:40] PROBLEM - wiki.counterculturelabs.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.counterculturelabs.org' expires in 15 day(s) (Sun 16 Jan 2022 04:39:56 GMT +0000). [04:43:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:45:30] PROBLEM - pastport.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pastport.org' expires in 15 day(s) (Sun 16 Jan 2022 04:43:10 GMT +0000). [04:48:13] PROBLEM - miraheze.gq - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'miraheze.gq' expires in 15 day(s) (Sun 16 Jan 2022 04:40:54 GMT +0000). [04:48:39] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:48:51] PROBLEM - podpedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'podpedia.org' expires in 15 day(s) (Sun 16 Jan 2022 04:41:58 GMT +0000). [04:50:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Jy544 [04:50:39] [02miraheze/puppet] 07paladox 0325f658e - base::dns: Support using ipv6 address for the resolver [04:50:40] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [04:50:42] [02puppet] 07paladox opened pull request 03#2191: base::dns: Support using ipv6 address for the resolver - 13https://git.io/Jy54B [04:50:46] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy54R [04:50:48] [02miraheze/ssl] 07MirahezeSSLBot 03d00b788 - Bot: Update SSL cert for wiki.counterculturelabs.org [04:51:52] PROBLEM - idleon.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'idleon.info' expires in 15 day(s) (Sun 16 Jan 2022 04:45:16 GMT +0000). [04:52:39] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [04:52:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy54g [04:52:52] [02miraheze/ssl] 07MirahezeSSLBot 03547e8e4 - Bot: Update SSL cert for wiki.macc.nyc [04:53:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Jy542 [04:53:25] [02miraheze/puppet] 07paladox 034dea63c - Update dns.pp [04:53:26] [02puppet] 07paladox synchronize pull request 03#2191: base::dns: Support using ipv6 address for the resolver - 13https://git.io/Jy54B [04:53:58] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-1/±0] 13https://git.io/Jy54a [04:54:00] [02miraheze/puppet] 07paladox 03450c743 - Rename modules/base/files/dns/resolv.conf to modules/base/templates/dns/resolv.conf.erb [04:54:01] [02puppet] 07paladox synchronize pull request 03#2191: base::dns: Support using ipv6 address for the resolver - 13https://git.io/Jy54B [04:54:54] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Jy54r [04:54:56] [02miraheze/puppet] 07paladox 03c8a30b3 - Update dns.pp [04:54:57] [02puppet] 07paladox synchronize pull request 03#2191: base::dns: Support using ipv6 address for the resolver - 13https://git.io/Jy54B [04:55:30] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Jy54K [04:55:31] [02miraheze/puppet] 07paladox 03ad9f00e - Update puppet111.yaml [04:55:33] [02puppet] 07paladox synchronize pull request 03#2191: base::dns: Support using ipv6 address for the resolver - 13https://git.io/Jy54B [04:56:13] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Jy546 [04:56:14] [02miraheze/puppet] 07paladox 034d3dc34 - Update resolv.conf.erb [04:56:16] [02puppet] 07paladox synchronize pull request 03#2191: base::dns: Support using ipv6 address for the resolver - 13https://git.io/Jy54B [04:56:35] [02puppet] 07paladox closed pull request 03#2191: base::dns: Support using ipv6 address for the resolver - 13https://git.io/Jy54B [04:56:37] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+1/-1/±2] 13https://git.io/Jy54P [04:56:38] [02miraheze/puppet] 07paladox 030d0e657 - base::dns: Support using ipv6 address for the resolver (#2191) [04:56:39] [02puppet] 07paladox deleted branch 03paladox-patch-4 - 13https://git.io/vbiAS [04:56:41] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-4 [04:56:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy54X [04:56:53] [02miraheze/ssl] 07MirahezeSSLBot 03b018bb9 - Bot: Update SSL cert for miraheze.gq [04:57:07] PROBLEM - wiki.erisly.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.erisly.com' expires in 15 day(s) (Sun 16 Jan 2022 04:50:13 GMT +0000). [04:57:51] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy54M [04:57:52] [02miraheze/ssl] 07MirahezeSSLBot 037dd85d0 - Bot: Update SSL cert for pastport.org [04:58:26] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy54y [04:58:28] [02miraheze/ssl] 07MirahezeSSLBot 03305f3dc - Bot: Update SSL cert for idleon.info [04:59:52] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.52, 3.10, 2.96 [05:00:03] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy54H [05:00:05] [02miraheze/ssl] 07MirahezeSSLBot 03ffc6093 - Bot: Update SSL cert for wiki.erisly.com [05:00:41] PROBLEM - lcn.zfc.id.lv - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'lcn.zfc.id.lv' expires in 15 day(s) (Sun 16 Jan 2022 04:52:25 GMT +0000). [05:00:55] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy54Q [05:00:56] [02miraheze/puppet] 07paladox 03d0aeb24 - fix [05:02:09] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy547 [05:02:11] [02miraheze/ssl] 07MirahezeSSLBot 0376e675c - Bot: Update SSL cert for podpedia.org [05:03:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.04, 3.33, 3.11 [05:07:31] [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jy5Bf [05:07:32] [02miraheze/ssl] 07MirahezeSSLBot 03d64b0db - Bot: Update SSL cert for lcn.zfc.id.lv [05:07:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-0/±0] 13https://git.io/Jy5BJ [05:07:35] [02miraheze/puppet] 07paladox 039fb8fab - Install new gluster servers [05:07:37] [02puppet] 07paladox created branch 03paladox-patch-4 - 13https://git.io/vbiAS [05:07:38] [02puppet] 07paladox opened pull request 03#2192: Install new gluster servers - 13https://git.io/Jy5BU [05:08:02] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-0/±0] 13https://git.io/Jy5BI [05:08:03] [02miraheze/puppet] 07paladox 0386c267c - Create gluster121.yaml [05:08:05] [02puppet] 07paladox synchronize pull request 03#2192: Install new gluster servers - 13https://git.io/Jy5BU [05:08:17] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-1/±0] 13https://git.io/Jy5Bt [05:08:18] [02miraheze/puppet] 07paladox 03c5945aa - Rename hieradata/gluster111.yaml to hieradata/hosts/gluster111.yaml [05:08:20] [02puppet] 07paladox synchronize pull request 03#2192: Install new gluster servers - 13https://git.io/Jy5BU [05:08:27] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-1/±0] 13https://git.io/Jy5Bm [05:08:29] [02miraheze/puppet] 07paladox 039e66c44 - Rename hieradata/gluster121.yaml to hieradata/hosts/gluster121.yaml [05:08:30] [02puppet] 07paladox synchronize pull request 03#2192: Install new gluster servers - 13https://git.io/Jy5BU [05:08:52] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+1/-0/±0] 13https://git.io/Jy5B3 [05:08:53] [02miraheze/puppet] 07paladox 03be30de7 - Create gluster101.yaml [05:08:55] [02puppet] 07paladox synchronize pull request 03#2192: Install new gluster servers - 13https://git.io/Jy5BU [05:09:45] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-4 [+0/-0/±1] 13https://git.io/Jy5BG [05:09:46] [02miraheze/puppet] 07paladox 0368c43f3 - Update gluster101.yaml [05:09:47] [02puppet] 07paladox synchronize pull request 03#2192: Install new gluster servers - 13https://git.io/Jy5BU [05:09:54]