[00:00:08] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN
[00:00:37] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is UNKNOWN: UNKNOWN - NGINX Error Rate is UNKNOWN
[00:02:08] <icinga-miraheze>	 PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is CRITICAL: CRITICAL - NGINX Error Rate is 100%
[00:02:35] <icinga-miraheze>	 PROBLEM - cp27 HTTP 4xx/5xx ERROR Rate on cp27 is CRITICAL: CRITICAL - NGINX Error Rate is 100%
[00:04:17] <icinga-miraheze>	 RECOVERY - cp36 Disk Space on cp36 is OK: DISK OK - free space: / 21755MiB (24% inode=98%);
[00:04:47] <icinga-miraheze>	 RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 21766MiB (24% inode=98%);
[02:01:45] <botifico>	 [02miraheze/ImportDump] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/40b80dd6f9be...881a922e51b7
[02:01:47] <botifico>	 [02miraheze/ImportDump] 07Universal-Omega 03881a922 - Increase the snippet length for failed import notifications to 1000 characters
[02:10:30] <miraheze-github>	 miraheze/ImportDump - Universal-Omega the build passed.
[02:15:30] <icinga-miraheze>	 [Grafana] !sre FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1
[02:20:30] <icinga-miraheze>	 [Grafana] !sre RESOLVED: High Job Queue Backlog https://grafana.miraheze.org/d/GtxbP1Xnk?orgId=1
[02:31:11] <botifico>	 [02miraheze/puppet] 07AgentIsai pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c032ad64fd67...21ae2d830fb8
[02:31:12] <botifico>	 [02miraheze/puppet] 07AgentIsai 0321ae2d8 - Add #miraheze-interwiki-feed to relay
[03:00:40] <icinga-miraheze>	 PROBLEM - mon181 Backups Grafana on mon181 is CRITICAL: FILE_AGE CRITICAL: /var/log/grafana-backup.log is 1209622 seconds old and 92 bytes
[03:22:51] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.98 ms
[03:24:51] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 164.34 ms
[03:36:50] <botifico>	 [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/f9a5d150e8b0
[03:36:53] <botifico>	 [02miraheze/mw-config] 07AgentIsai 03f9a5d15 - Load SecurePoll on Meta
[03:36:55] <botifico>	 [02mw-config] 07AgentIsai created branch 03AgentIsai-patch-1 - 13https://github.com/miraheze/mw-config
[03:36:57] <botifico>	 [02mw-config] 07AgentIsai opened pull request 03#5496: Load SecurePoll on Meta - 13https://github.com/miraheze/mw-config/pull/5496
[03:37:55] <miraheze-github>	 miraheze/mw-config - AgentIsai the build passed.
[03:48:15] <botifico>	 [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03AgentIsai-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f9a5d150e8b0...7bda4d2044de
[03:48:17] <botifico>	 [02miraheze/mw-config] 07AgentIsai 037bda4d2 - Allow electionadmin on Meta to create polls
[03:48:18] <botifico>	 [02mw-config] 07AgentIsai synchronize pull request 03#5496: Load SecurePoll on Meta - 13https://github.com/miraheze/mw-config/pull/5496
[03:48:54] <botifico>	 [02mw-config] 07AgentIsai closed pull request 03#5496: Load SecurePoll on Meta - 13https://github.com/miraheze/mw-config/pull/5496
[03:48:56] <botifico>	 [02miraheze/mw-config] 07AgentIsai deleted branch 03AgentIsai-patch-1
[03:48:57] <botifico>	 [02miraheze/mw-config] 07AgentIsai pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/mw-config/compare/f23ba171bd9c...666f2c441eda
[03:48:58] <botifico>	 [02miraheze/mw-config] 07AgentIsai 03666f2c4 - Load SecurePoll on Meta (#5496)
[03:49:01] <botifico>	 [02mw-config] 07AgentIsai deleted branch 03AgentIsai-patch-1 - 13https://github.com/miraheze/mw-config
[03:49:10] <miraheze-github>	 miraheze/mw-config - AgentIsai the build passed.
[03:49:47] <miraheze-github>	 miraheze/mw-config - AgentIsai the build passed.
[03:50:17] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=metawiki /srv/mediawiki/1.41/extensions/SecurePoll/sql/mysql/tables-generated.sql (END - exit=0)
[03:50:24] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:50:31] <MirahezeLSBot>	 !log [agent@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[03:50:38] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:50:42] <MirahezeLSBot>	 !log [agent@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s
[03:50:49] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:53:04] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/resetWikiCaches.php --wiki=metawiki (END - exit=0)
[03:53:05] <MirahezeLSBot>	 !log [agent@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/MirahezeMagic/maintenance/resetWikiCaches.php --wiki=metawiki (END - exit=0)
[03:53:11] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:53:12] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[03:53:13] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[03:53:19] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:53:27] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[03:53:34] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:07:29] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'config': True} to all
[04:07:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[04:07:37] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s
[04:07:45] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[06:57:04] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/2d1190f58bfc...15df86b7a39f
[06:57:05] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 0315df86b - Bot: Add SSL cert for wiki.jerryly.top
[07:17:38] <icinga-miraheze>	 PROBLEM - wiki.jerryly.top - reverse DNS on sslhost is WARNING: SSL WARNING - rDNS OK but records conflict. {'NS': ['dns31.hichina.com.', 'dns32.hichina.com.'], 'CNAME': 'lime.miraheze.org.'}
[07:56:41] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 158.59 ms
[07:58:44] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 163.11 ms
[08:11:24] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.72 ms
[08:15:27] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.79 ms
[09:15:58] <botifico>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/f674da8a5722
[09:16:01] <botifico>	 [02miraheze/mw-config] 07Reception123 03f674da8 - change ImportDump rights
[09:16:03] <botifico>	 [02mw-config] 07Reception123 created branch 03Reception123-patch-1 - 13https://github.com/miraheze/mw-config
[09:16:05] <botifico>	 [02mw-config] 07Reception123 opened pull request 03#5497: change ImportDump rights - 13https://github.com/miraheze/mw-config/pull/5497
[09:17:09] <miraheze-github>	 miraheze/mw-config - Reception123 the build passed.
[09:21:13] <botifico>	 [02mw-config] 07Reception123 synchronize pull request 03#5497: change ImportDump rights - 13https://github.com/miraheze/mw-config/pull/5497
[09:21:14] <botifico>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f674da8a5722...405729159838
[09:21:16] <botifico>	 [02miraheze/mw-config] 07Reception123 034057291 - Update LocalSettings.php
[09:22:07] <miraheze-github>	 miraheze/mw-config - Reception123 the build passed.
[09:25:06] <botifico>	 [02mw-config] 07Reception123 synchronize pull request 03#5497: change ImportDump rights - 13https://github.com/miraheze/mw-config/pull/5497
[09:25:09] <botifico>	 [02miraheze/mw-config] 07Reception123 pushed 031 commit to 03Reception123-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/405729159838...1c1d0abd5dd4
[09:25:12] <botifico>	 [02miraheze/mw-config] 07Reception123 031c1d0ab - Update LocalSettings.php
[09:25:55] <miraheze-github>	 miraheze/mw-config - Reception123 the build passed.
[09:26:15] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.12 ms
[09:28:16] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 184.25 ms
[09:38:23] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 178.76 ms
[09:40:24] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 167.74 ms
[10:02:23] <icinga-miraheze>	 PROBLEM - cloud16 Puppet on cloud16 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2]
[10:30:22] <icinga-miraheze>	 RECOVERY - cloud16 Puppet on cloud16 is OK: OK: Puppet is currently enabled, last run 56 seconds ago with 0 failures
[12:26:24] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 169.35 ms
[12:28:27] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.55 ms
[13:04:26] <icinga-miraheze>	 PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 2 packages available for upgrade (1 critical updates).
[13:41:44] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 175.19 ms
[13:43:46] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 162.56 ms
[13:58:20] <botifico>	 [02RequestSSL] 07redbluegreenhat synchronize pull request 03#40: T11699: Create a job to automate CNAME checks - 13https://github.com/miraheze/RequestSSL/pull/40
[14:02:04] <miraheze-github>	 miraheze/RequestSSL - redbluegreenhat the build has errored.
[14:12:45] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 180.77 ms
[14:18:55] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 182.57 ms
[14:22:46] <botifico>	 [02RequestSSL] 07redbluegreenhat synchronize pull request 03#40: T11699: Create a job to automate CNAME checks - 13https://github.com/miraheze/RequestSSL/pull/40
[14:23:52] <botifico>	 [02RequestSSL] 07redbluegreenhat reviewed pull request 03#40 commit - 13https://github.com/miraheze/RequestSSL/pull/40#discussion_r1518866260
[14:24:38] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=universopediawiki /srv/mediawiki/1.41/extensions/CheckUser/schema/mysql/patch-cu_private_event-def.sql (END - exit=0)
[14:24:46] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:25:08] <botifico>	 [02RequestSSL] 07redbluegreenhat synchronize pull request 03#40: T11699: Create a job to automate CNAME checks - 13https://github.com/miraheze/RequestSSL/pull/40
[14:25:32] <botifico>	 [02RequestSSL] 07redbluegreenhat reviewed pull request 03#40 commit - 13https://github.com/miraheze/RequestSSL/pull/40#discussion_r1518866575
[14:25:52] <miraheze-github>	 miraheze/RequestSSL - redbluegreenhat the build has errored.
[14:26:08] <botifico>	 [02RequestSSL] 07redbluegreenhat reviewed pull request 03#40 commit - 13https://github.com/miraheze/RequestSSL/pull/40#discussion_r1518866685
[14:26:13] <MirahezeLSBot>	 !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/sql.php --wiki=universopediawiki /srv/mediawiki/1.41/extensions/CheckUser/schema/mysql/patch-cu_changes-add-cuc_only_for_read_old.sql (END - exit=0)
[14:26:20] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[14:28:00] <miraheze-github>	 miraheze/RequestSSL - redbluegreenhat the build has errored.
[14:29:16] <botifico>	 [02RequestSSL] 07redbluegreenhat reviewed pull request 03#40 commit - 13https://github.com/miraheze/RequestSSL/pull/40#discussion_r1518867348
[14:32:36] <botifico>	 [02RequestSSL] 07redbluegreenhat synchronize pull request 03#40: T11699: Create a job to automate CNAME checks - 13https://github.com/miraheze/RequestSSL/pull/40
[14:32:51] <botifico>	 [02RequestSSL] 07redbluegreenhat reviewed pull request 03#40 commit - 13https://github.com/miraheze/RequestSSL/pull/40#discussion_r1518870498
[14:34:41] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 170.57 ms
[14:35:35] <miraheze-github>	 miraheze/RequestSSL - redbluegreenhat the build has errored.
[14:36:44] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 172.45 ms
[14:36:51] <botifico>	 [02RequestSSL] 07redbluegreenhat reviewed pull request 03#40 commit - 13https://github.com/miraheze/RequestSSL/pull/40#discussion_r1518871160
[14:40:33] <botifico>	 [02RequestSSL] 07redbluegreenhat reviewed pull request 03#40 commit - 13https://github.com/miraheze/RequestSSL/pull/40#discussion_r1518871879
[14:50:58] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 187.48 ms
[14:53:01] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 225.10 ms
[14:57:12] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 181.27 ms
[15:01:19] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 180.60 ms
[15:05:32] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 223.03 ms
[15:07:35] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 182.52 ms
[15:16:55] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 181.06 ms
[15:23:08] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 184.62 ms
[15:40:47] <icinga-miraheze>	 PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 177.94 ms
[15:42:50] <icinga-miraheze>	 RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 176.37 ms
[15:44:11] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 171.71 ms
[15:46:14] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 174.03 ms
[16:31:40] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is WARNING: PING WARNING - Packet loss = 0%, RTA = 318.61 ms
[16:33:46] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 250.58 ms
[16:37:53] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 179.60 ms
[16:48:16] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 170.21 ms
[16:50:19] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 168.47 ms
[17:24:30] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.16 ms
[17:26:34] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 174.75 ms
[17:44:04] <botifico>	 [02miraheze/mw-config] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/666f2c441eda...2cae7e1eb0fe
[17:44:06] <botifico>	 [02miraheze/mw-config] 07MacFan4000 032cae7e1 - Restore interwiki permission for meta sysops
[17:44:37] <MirahezeLSBot>	 !log [macfan@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all
[17:44:44] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:44:47] <MirahezeLSBot>	 !log [macfan@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 10s
[17:44:55] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:44:57] <miraheze-github>	 miraheze/mw-config - MacFan4000 the build passed.
[17:53:21] <MirahezeLSBot>	 !log [@test151] starting deploy of {'config': True} to test151
[17:53:22] <MirahezeLSBot>	 !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s
[17:53:28] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[17:53:36] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:00:45] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 174.09 ms
[18:06:57] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 176.51 ms
[18:07:42] <MirahezeLSBot>	 !log [@mwtask171] starting deploy of {'config': True} to all
[18:07:50] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[18:07:50] <MirahezeLSBot>	 !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 8s
[18:07:58] <MirahezeLogbot>	 Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log
[19:01:53] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 168.14 ms
[19:03:56] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 171.11 ms
[20:03:25] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 176.06 ms
[20:05:29] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.94 ms
[20:09:47] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 28%, RTA = 175.62 ms
[20:16:02] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.01 ms
[20:26:25] <icinga-miraheze>	 PROBLEM - wiki.grab-tools.live - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.grab-tools.live' expires in 15 day(s) (Tue 26 Mar 2024 08:09:24 PM GMT +0000).
[20:26:43] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/15df86b7a39f...e57140a8527e
[20:26:44] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 03e57140a - Bot: Update SSL cert for wiki.grab-tools.live
[20:27:43] <icinga-miraheze>	 PROBLEM - wiki.cutefame.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cutefame.net' expires in 15 day(s) (Tue 26 Mar 2024 08:10:28 PM GMT +0000).
[20:27:55] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e57140a8527e...03d01fa71ee0
[20:27:56] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 0303d01fa - Bot: Update SSL cert for wiki.cutefame.net
[20:29:33] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 172.61 ms
[20:31:36] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 176.84 ms
[20:37:04] <icinga-miraheze>	 PROBLEM - cp41 Puppet on cp41 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/etc/logrotate.d/varnishlog_logs]
[20:38:59] <icinga-miraheze>	 PROBLEM - gogigantic.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'gogigantic.wiki' expires in 15 day(s) (Tue 26 Mar 2024 08:22:01 PM GMT +0000).
[20:39:11] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/03d01fa71ee0...55c64191c555
[20:39:12] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 0355c6419 - Bot: Update SSL cert for gogigantic.wiki
[20:42:04] <icinga-miraheze>	 PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 16%, RTA = 167.07 ms
[20:52:25] <icinga-miraheze>	 RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 169.53 ms
[20:55:21] <icinga-miraheze>	 RECOVERY - wiki.grab-tools.live - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.grab-tools.live' will expire on Sat 08 Jun 2024 07:26:37 PM GMT +0000.
[20:56:26] <icinga-miraheze>	 RECOVERY - wiki.cutefame.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cutefame.net' will expire on Sat 08 Jun 2024 07:27:49 PM GMT +0000.
[21:05:03] <icinga-miraheze>	 RECOVERY - cp41 Puppet on cp41 is OK: OK: Puppet is currently enabled, last run 2 seconds ago with 0 failures
[21:36:37] <icinga-miraheze>	 PROBLEM - worldtobuild.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'worldtobuild.wiki' expires in 15 day(s) (Tue 26 Mar 2024 09:24:34 PM GMT +0000).
[21:36:39] <icinga-miraheze>	 RECOVERY - gogigantic.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'gogigantic.wiki' will expire on Sat 08 Jun 2024 07:39:04 PM GMT +0000.
[21:36:49] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/55c64191c555...ce661b9a010d
[21:36:52] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 03ce661b9 - Bot: Update SSL cert for worldtobuild.wiki
[21:37:16] <icinga-miraheze>	 PROBLEM - pandorastale.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pandorastale.wiki' expires in 15 day(s) (Tue 26 Mar 2024 09:10:22 PM GMT +0000).
[21:37:27] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ce661b9a010d...3cb56da7ae1e
[21:37:29] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 033cb56da - Bot: Update SSL cert for pandorastale.wiki
[21:37:34] <icinga-miraheze>	 PROBLEM - mockgovernments.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mockgovernments.com' expires in 15 day(s) (Tue 26 Mar 2024 09:20:56 PM GMT +0000).
[21:37:43] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3cb56da7ae1e...5e8274d1924e
[21:37:46] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 035e8274d - Bot: Update SSL cert for mockgovernments.com
[22:06:22] <icinga-miraheze>	 RECOVERY - worldtobuild.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'worldtobuild.wiki' will expire on Sat 08 Jun 2024 08:36:43 PM GMT +0000.
[22:06:33] <icinga-miraheze>	 RECOVERY - pandorastale.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'pandorastale.wiki' will expire on Sat 08 Jun 2024 08:37:22 PM GMT +0000.
[22:07:22] <icinga-miraheze>	 RECOVERY - mockgovernments.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mockgovernments.com' will expire on Sat 08 Jun 2024 08:37:37 PM GMT +0000.
[22:17:03] <icinga-miraheze>	 PROBLEM - cp36 Disk Space on cp36 is WARNING: DISK WARNING - free space: / 9711MiB (10% inode=98%);
[22:27:15] <icinga-miraheze>	 PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 9705MiB (10% inode=98%);
[22:40:12] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mahdiruiz.line.pm All nameservers failed to answer the query.
[23:09:47] <icinga-miraheze>	 PROBLEM - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is WARNING: NoNameservers: All nameservers failed to answer the query wiki.mahdiruiz.line.pm. IN CNAME: Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered The DNS operation timed out.; Server 2606:4700:4700::1111 UDP port 53 answered SERVFAIL
[23:18:03] <icinga-miraheze>	 PROBLEM - wiki.solarconquest.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.solarconquest.net' expires in 15 day(s) (Tue 26 Mar 2024 11:04:24 PM GMT +0000).
[23:18:15] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5e8274d1924e...357a3cd74be8
[23:18:17] <botifico>	 [02miraheze/ssl] 07MirahezeSSLBot 03357a3cd - Bot: Update SSL cert for wiki.solarconquest.net
[23:39:24] <icinga-miraheze>	 RECOVERY - wiki.mahdiruiz.line.pm - reverse DNS on sslhost is OK: SSL OK - wiki.mahdiruiz.line.pm reverse DNS resolves to cp37.wikitide.net - CNAME OK
[23:47:56] <icinga-miraheze>	 RECOVERY - wiki.solarconquest.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.solarconquest.net' will expire on Sat 08 Jun 2024 10:18:09 PM GMT +0000.