[00:02:31] !log [@mwtask181] starting deploy of {'config': True} to all [00:02:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:02:42] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 11s [00:02:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:07:47] !log [@mwtask171] starting deploy of {'config': True} to all [00:07:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:07:57] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 9s [00:08:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:24:05] !log [@test151] starting deploy of {'config': True} to test151 [00:24:06] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [00:24:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:24:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:47:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:51:39] !log [void@puppet181] Upgraded packages apache2, apache2-bin, apache2-data, and apache2-utils on graphite151 [00:51:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:52:30] !log [void@puppet181] Upgraded packages Write, IO, and The on changeprop151 [00:52:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:53:18] !log [void@puppet181] Upgraded packages apache2, apache2-bin, apache2-data, and apache2-utils on mwtask171 [00:53:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:53:34] RECOVERY - graphite151 APT on graphite151 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [00:54:40] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 64 packages available for upgrade (0 critical updates). [00:55:34] !log [void@puppet181] Upgraded packages apache2, apache2-bin, apache2-data, and apache2-utils on prometheus151 [00:55:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:52] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 53 packages available for upgrade (0 critical updates). [00:56:07] !log [void@puppet181] Upgraded packages apache2, apache2-bin, apache2-data, and apache2-utils on mwtask181 [00:56:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:56:28] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 64 packages available for upgrade (0 critical updates). [00:57:32] !log [void@puppet181] Upgraded packages apache2, apache2-bin, apache2-data, and apache2-utils on test151 [00:57:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:59:22] RECOVERY - test151 APT on test151 is OK: APT OK: 76 packages available for upgrade (0 critical updates). [03:05:23] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 33.45, 23.69, 19.61 [03:28:21] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=bignatecommentswiki /home/reception/bignatecomments_pages_full.xml --username-prefix=wikia:big-nate-comments (END - exit=0) [03:28:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:54:55] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.69, 18.31, 23.40 [03:56:54] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 29.74, 23.37, 24.70 [04:11:01] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [04:13:02] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [107 system event log (SEL) entries present] [04:30:35] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 11.24, 17.50, 23.65 [04:36:35] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 11.05, 13.43, 19.85 [05:39:31] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [05:41:32] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [108 system event log (SEL) entries present] [05:58:11] [02dns] 07BlankEclair opened pull request 03#535: T12333: Add tools.ff8.wiki -> ff8-speedruns.github.io CNAME - 13https://github.com/miraheze/dns/pull/535 [05:59:25] [02dns] 07Universal-Omega closed pull request 03#535: T12333: Add tools.ff8.wiki -> ff8-speedruns.github.io CNAME - 13https://github.com/miraheze/dns/pull/535 [05:59:28] [02dns] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/be1b59a6e7e8...b526b73c69da [05:59:29] [02dns] 07BlankEclair 03b526b73 - T12333: Add tools.ff8.wiki -> ff8-speedruns.github.io CNAME (#535) [06:07:04] [02mw-config] 07BlankEclair synchronize pull request 03#5598: T12214: Avoid redirects if there are query parameters - 13https://github.com/miraheze/mw-config/pull/5598 [06:07:06] [02mw-config] 07BlankEclair edited pull request 03#5598: T12214: Avoid redirects if action=raw is set - 13https://github.com/miraheze/mw-config/pull/5598 [06:08:08] miraheze/mw-config - BlankEclair the build passed. [06:14:24] [02puppet] 07waki285 opened pull request 03#3882: T12186: Add translate.googleapis.com to CSP - 13https://github.com/miraheze/puppet/pull/3882 [06:14:29] [02puppet] 07coderabbitai[bot] commented on pull request 03#3882: T12186: Add translate.googleapis.com to CSP - 13https://github.com/miraheze/puppet/pull/3882#issuecomment-2227766038 [06:52:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:03:52] [02puppet] 07Universal-Omega closed pull request 03#3882: T12186: Add translate.googleapis.com to CSP - 13https://github.com/miraheze/puppet/pull/3882 [07:03:55] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/7e1e29491b85...384b61324352 [07:03:56] [02puppet] 07waki285 03384b613 - T12186: Add translate.googleapis.com to CSP (#3882) [07:04:19] [02puppet] 07Universal-Omega closed pull request 03#3880: Remove dead domain from CSP - 13https://github.com/miraheze/puppet/pull/3880 [07:04:22] [02puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/384b61324352...c5c93c3bcfe0 [07:04:23] [02puppet] 07BlankEclair 03c5c93c3 - Remove dead domain from CSP (#3880) [07:07:59] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [07:10:00] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [109 system event log (SEL) entries present] [07:22:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:08:56] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [08:24:12] PROBLEM - wiki.orvyn.ca - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [08:24:19] PROBLEM - thepolicyhub.org.uk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - thepolicyhub.org.uk All nameservers failed to answer the query. [08:30:17] PROBLEM - www.sekaipedia.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.sekaipedia.org All nameservers failed to answer the query. [08:30:40] PROBLEM - en.noblework.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - en.noblework.org All nameservers failed to answer the query. [08:38:22] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [08:53:54] RECOVERY - wiki.orvyn.ca - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.orvyn.ca' will expire on Sun 15 Sep 2024 02:02:22 PM GMT +0000. [08:54:13] RECOVERY - thepolicyhub.org.uk - reverse DNS on sslhost is OK: SSL OK - thepolicyhub.org.uk reverse DNS resolves to cp36.wikitide.net - CNAME FLAT [08:59:21] RECOVERY - en.noblework.org - reverse DNS on sslhost is OK: SSL OK - en.noblework.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [09:00:05] RECOVERY - www.sekaipedia.org - reverse DNS on sslhost is OK: SSL OK - www.sekaipedia.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [09:08:44] PROBLEM - wiki.wubbygame.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.wubbygame.com All nameservers failed to answer the query. [09:08:58] PROBLEM - www.samafia.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - www.samafia.org All nameservers failed to answer the query. [09:37:58] RECOVERY - wiki.wubbygame.com - reverse DNS on sslhost is OK: SSL OK - wiki.wubbygame.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [09:38:59] RECOVERY - www.samafia.org - reverse DNS on sslhost is OK: SSL OK - www.samafia.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [10:04:56] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [10:06:57] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [111 system event log (SEL) entries present] [10:18:39] PROBLEM - wiki.mc.marochuru.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mc.marochuru.net All nameservers failed to answer the query. [10:20:58] [02puppet] 07RhinosF1 opened pull request 03#3883: add Cache-Tag to mediawiki responses - 13https://github.com/miraheze/puppet/pull/3883 [10:21:05] [02puppet] 07coderabbitai[bot] commented on pull request 03#3883: add Cache-Tag to mediawiki responses - 13https://github.com/miraheze/puppet/pull/3883#issuecomment-2228165077 [10:23:41] [02puppet] 07redbluegreenhat closed pull request 03#3883: add Cache-Tag to mediawiki responses - 13https://github.com/miraheze/puppet/pull/3883 [10:23:43] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/c5c93c3bcfe0...6e6afd08b308 [10:23:46] [02puppet] 07RhinosF1 036e6afd0 - add Cache-Tag to mediawiki responses (#3883) [10:28:23] [02puppet] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/6e6afd08b308...938dd53dfdb9 [10:28:24] [02puppet] 07redbluegreenhat 03938dd53 - use tabs [10:31:19] PROBLEM - tep.wiki - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - tep.wiki All nameservers failed to answer the query. [10:32:30] PROBLEM - valsora.tep.wiki - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [10:33:56] PROBLEM - wiki.secondrenaissance.net - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.secondrenaissance.net All nameservers failed to answer the query. [10:35:12] PROBLEM - pwiki.drydraytonvillagehall.org.uk - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - pwiki.drydraytonvillagehall.org.uk All nameservers failed to answer the query. [10:47:19] RECOVERY - wiki.mc.marochuru.net - reverse DNS on sslhost is OK: SSL OK - wiki.mc.marochuru.net reverse DNS resolves to cp36.wikitide.net - CNAME OK [10:57:10] PROBLEM - tep.wiki - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [10:58:16] PROBLEM - wiki.candelabrem.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.candelabrem.com All nameservers failed to answer the query. [10:58:27] PROBLEM - amps.wiki.gd - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [11:02:53] RECOVERY - wiki.secondrenaissance.net - reverse DNS on sslhost is OK: SSL OK - wiki.secondrenaissance.net reverse DNS resolves to cp36.wikitide.net - CNAME OK [11:03:13] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [11:03:54] RECOVERY - pwiki.drydraytonvillagehall.org.uk - reverse DNS on sslhost is OK: SSL OK - pwiki.drydraytonvillagehall.org.uk reverse DNS resolves to cp36.wikitide.net - CNAME OK [11:05:14] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [112 system event log (SEL) entries present] [11:15:48] PROBLEM - fid.koymi.net - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [11:25:57] PROBLEM - db181 Backups SQL on db181 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864167 seconds old and 137002 bytes [11:26:51] RECOVERY - tep.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tep.wiki' will expire on Wed 09 Oct 2024 09:10:56 PM GMT +0000. [11:27:33] RECOVERY - wiki.candelabrem.com - reverse DNS on sslhost is OK: SSL OK - wiki.candelabrem.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [11:27:53] RECOVERY - amps.wiki.gd - LetsEncrypt on sslhost is OK: OK - Certificate 'amps.wiki.gd' will expire on Mon 09 Sep 2024 05:49:32 PM GMT +0000. [11:29:56] RECOVERY - tep.wiki - reverse DNS on sslhost is OK: SSL OK - tep.wiki reverse DNS resolves to cp36.wikitide.net - CNAME OK [11:30:30] RECOVERY - valsora.tep.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'valsora.tep.wiki' will expire on Sun 29 Sep 2024 03:32:02 PM GMT +0000. [11:44:36] RECOVERY - fid.koymi.net - LetsEncrypt on sslhost is OK: OK - Certificate 'fid.koymi.net' will expire on Wed 25 Sep 2024 05:11:11 PM GMT +0000. [11:51:26] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [11:53:27] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [113 system event log (SEL) entries present] [12:02:57] PROBLEM - db181 Backups SQL on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:03:53] PROBLEM - db181 PowerDNS Recursor on db181 is CRITICAL: CRITICAL - Plugin timed out while executing system call [12:04:12] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 77.44, 37.86, 15.51 [12:04:36] PROBLEM - db181 Puppet on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:05:08] PROBLEM - db181 Backups SQL on db181 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 866517 seconds old and 137002 bytes [12:05:36] PROBLEM - db181 APT on db181 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [12:07:56] RECOVERY - db181 APT on db181 is OK: APT OK: 61 packages available for upgrade (0 critical updates). [12:08:02] RECOVERY - db181 PowerDNS Recursor on db181 is OK: DNS OK: 0.075 seconds response time. wikitide.net returns 2602:294:0:b13::110,2602:294:0:b23::112,38.46.223.205,38.46.223.206 [12:09:20] PROBLEM - wiki.mxlinuxusers.de - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.mxlinuxusers.de All nameservers failed to answer the query. [12:09:26] RECOVERY - db181 Puppet on db181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [12:13:31] [02CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/8586c6c03865...f3671ff2cf0a [12:13:33] [02CreateWiki] 07translatewiki 03f3671ff - Localisation updates from https://translatewiki.net. [12:13:35] [02ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/b734bfc6f5fb...269385816d02 [12:13:38] [02ImportDump] 07translatewiki 032693858 - Localisation updates from https://translatewiki.net. [12:13:41] [02DataDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/DataDump/compare/7a788c957193...722db5a89d2a [12:13:42] [02DataDump] 07translatewiki 03722db5a - Localisation updates from https://translatewiki.net. [12:13:45] [02WikiDiscover] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/WikiDiscover/compare/f4496b1ed733...2879adb157d2 [12:13:47] [02WikiDiscover] 07translatewiki 032879adb - Localisation updates from https://translatewiki.net. [12:13:48] [02YouTube] 07translatewiki pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/YouTube/compare/b5e21828bd80...a0bda790b2c0 [12:13:49] [02YouTube] 07translatewiki 03a0bda79 - Localisation updates from https://translatewiki.net. [12:13:51] [02MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/d451938e0c81...4a0f2ec158bb [12:13:54] [02MirahezeMagic] 07translatewiki 034a0f2ec - Localisation updates from https://translatewiki.net. [12:17:50] miraheze/YouTube - translatewiki the build passed. [12:18:00] miraheze/MirahezeMagic - translatewiki the build has errored. [12:18:25] miraheze/DataDump - translatewiki the build passed. [12:18:55] miraheze/WikiDiscover - translatewiki the build passed. [12:23:05] miraheze/CreateWiki - translatewiki the build passed. [12:23:48] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [12:23:49] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:23:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:01] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [12:24:02] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:09] PROBLEM - db181 Current Load on db181 is WARNING: LOAD WARNING - total load average: 0.42, 3.09, 11.57 [12:24:09] miraheze/ImportDump - translatewiki the build passed. [12:24:14] !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 [12:24:15] !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:25:36] PROBLEM - history.sdtef.org - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [12:28:09] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.30, 1.56, 9.00 [12:29:09] PROBLEM - db161 Backups SQL on db161 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864184 seconds old and 138055 bytes [12:32:51] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:32:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:02] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 11s [12:33:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:13] !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:33:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:23] !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 9s [12:33:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:41] !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:37:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:50] !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s [12:37:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:01] !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:38:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:10] !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 9s [12:38:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:39:11] RECOVERY - wiki.mxlinuxusers.de - reverse DNS on sslhost is OK: SSL OK - wiki.mxlinuxusers.de reverse DNS resolves to cp36.wikitide.net - CNAME OK [12:51:47] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [12:53:47] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [114 system event log (SEL) entries present] [12:55:13] RECOVERY - history.sdtef.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.sdtef.org' will expire on Wed 09 Oct 2024 04:01:19 PM GMT +0000. [14:16:09] PROBLEM - psycho.engineering - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'psycho.engineering' expires in 15 day(s) (Wed 31 Jul 2024 02:10:28 PM GMT +0000). [14:16:23] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/42be862cbd72...9ee397aa4ca7 [14:16:24] [02ssl] 07WikiTideSSLBot 039ee397a - Bot: Update SSL cert for psycho.engineering [14:27:45] RECOVERY - wiki.blockate.com - reverse DNS on sslhost is OK: SSL OK - wiki.blockate.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [14:32:36] PROBLEM - electowiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'electowiki.org' expires in 15 day(s) (Wed 31 Jul 2024 02:10:14 PM GMT +0000). [14:32:47] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9ee397aa4ca7...b28a8d30258f [14:32:48] [02ssl] 07WikiTideSSLBot 03b28a8d3 - Bot: Update SSL cert for electowiki.org [14:35:46] PROBLEM - evilgeniuswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'evilgeniuswiki.com' expires in 15 day(s) (Wed 31 Jul 2024 02:07:03 PM GMT +0000). [14:35:57] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b28a8d30258f...2096c0f0fe8f [14:36:00] [02ssl] 07WikiTideSSLBot 032096c0f - Bot: Update SSL cert for evilgeniuswiki.com [14:36:51] PROBLEM - storytime.jdstroy.cf - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'storytime.jdstroy.cf' expires in 15 day(s) (Wed 31 Jul 2024 02:18:12 PM GMT +0000). [14:37:05] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2096c0f0fe8f...e029c7ed2e3d [14:37:08] [02ssl] 07WikiTideSSLBot 03e029c7e - Bot: Update SSL cert for storytime.jdstroy.cf [14:52:07] PROBLEM - wiki.blockate.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.blockate.com' expires in 14 day(s) (Mon 29 Jul 2024 08:33:10 PM GMT +0000). [14:52:18] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e029c7ed2e3d...4a463068eda3 [14:52:21] [02ssl] 07WikiTideSSLBot 034a46306 - Bot: Update SSL cert for wiki.blockate.com [15:02:34] RECOVERY - electowiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'electowiki.org' will expire on Sun 13 Oct 2024 01:32:41 PM GMT +0000. [15:05:01] RECOVERY - evilgeniuswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'evilgeniuswiki.com' will expire on Sun 13 Oct 2024 01:35:51 PM GMT +0000. [15:05:42] RECOVERY - storytime.jdstroy.cf - LetsEncrypt on sslhost is OK: OK - Certificate 'storytime.jdstroy.cf' will expire on Sun 13 Oct 2024 01:37:00 PM GMT +0000. [15:14:27] RECOVERY - psycho.engineering - LetsEncrypt on sslhost is OK: OK - Certificate 'psycho.engineering' will expire on Sun 13 Oct 2024 01:16:15 PM GMT +0000. [15:16:18] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:18:33] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [15:20:33] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [117 system event log (SEL) entries present] [15:21:15] RECOVERY - wiki.blockate.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.blockate.com' will expire on Sun 13 Oct 2024 01:52:13 PM GMT +0000. [15:43:52] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:06:48] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [16:08:49] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [118 system event log (SEL) entries present] [16:12:59] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/4a463068eda3...6e6b398b4c0e [16:13:01] [02ssl] 07WikiTideSSLBot 036e6b398 - Bot: Add SSL cert for vesc.wiki [16:13:30] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/ssl/compare/6e6b398b4c0e...955305f8ad7d [16:13:32] [02ssl] 07WikiTideSSLBot 03955305f - Bot: Add SSL cert for keitaiwiki.com [16:14:58] PROBLEM - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is WARNING: WARNING - NGINX Error Rate is 59% [16:15:00] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/955305f8ad7d...223880c2b79d [16:15:02] [02ssl] 07WikiTideSSLBot 03223880c - Bot: Update SSL cert for www.gengbaike.top [16:17:31] [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/223880c2b79d...5f6ff08c57ee [16:17:34] [02ssl] 07BlankEclair 035f6ff08 - T12311: Redirect gimkit.miraheze.org to gimkit.wiki (#788) [16:17:36] [02ssl] 07MacFan4000 closed pull request 03#788: T12311: Redirect gimkit.miraheze.org to gimkit.wiki - 13https://github.com/miraheze/ssl/pull/788 [16:22:59] RECOVERY - cp26 HTTP 4xx/5xx ERROR Rate on cp26 is OK: OK - NGINX Error Rate is 26% [16:32:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:40:37] RECOVERY - www.gengbaike.top - LetsEncrypt on sslhost is OK: OK - Certificate 'www.gengbaike.top' will expire on Sun 13 Oct 2024 03:14:54 PM GMT +0000. [17:03:34] miraheze/ssl - MacFan4000 the build has errored. [17:15:06] ^ not an actual failure just a GitHub actions server glitch [17:35:17] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [17:37:16] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 122 system event log (SEL) entries present] [18:25:14] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [18:27:14] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 124 system event log (SEL) entries present] [18:29:07] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.52, 19.30, 17.05 [18:31:05] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 19.33, 19.52, 17.43 [18:43:37] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [18:45:04] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [18:45:35] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0008938610554 secs [18:55:26] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.63, 17.95, 17.12 [18:57:23] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.39, 18.12, 17.29 [19:02:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [19:08:28] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.89, 18.62, 17.84 [19:10:28] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 12.10, 15.90, 16.93 [19:15:41] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [19:47:34] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [20:01:01] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [20:02:20] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [20:03:01] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 128 system event log (SEL) entries present] [20:15:01] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:50:54] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [20:52:54] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 130 system event log (SEL) entries present] [21:35:07] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [21:37:07] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 132 system event log (SEL) entries present] [21:57:10] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.00, 17.75, 14.31 [21:59:07] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.71, 17.10, 14.49 [22:12:20] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [23:43:33] !log [void@mwtask181] starting deploy of {'versions': '1.41', 'upgrade_skins': 'Citizen'} to all [23:43:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:43:46] !log [void@mwtask181] finished deploy of {'versions': '1.41', 'upgrade_skins': 'Citizen'} to all - SUCCESS in 13s [23:43:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log