[00:58:23] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/deleteArchivedFiles.php --wiki=kagagawiki --delete (END - exit=0) [00:58:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:07:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:17:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [01:30:53] !log [void@mwtask181] starting deploy of {'folders': 'extensions/MultiPurge'} to mwtask181 [01:30:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:31:18] !log [void@mwtask181] starting deploy of {'folders': '1.41/extensions/MultiPurge'} to mwtask181 [01:31:19] !log [void@mwtask181] finished deploy of {'folders': '1.41/extensions/MultiPurge'} to mwtask181 - SUCCESS in 0s [01:31:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:31:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:25:01] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [02:27:02] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 80 system event log (SEL) entries present] [03:57:20] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:02:20] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:59:44] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [05:01:45] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [82 system event log (SEL) entries present] [06:06:12] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [06:08:12] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [83 system event log (SEL) entries present] [06:56:32] [02puppet] 07BlankEclair opened pull request 03#3880: Remove dead domain from CSP - 13https://github.com/miraheze/puppet/pull/3880 [06:56:37] [02puppet] 07coderabbitai[bot] commented on pull request 03#3880: Remove dead domain from CSP - 13https://github.com/miraheze/puppet/pull/3880#issuecomment-2227218121 [07:14:33] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [07:16:34] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [84 system event log (SEL) entries present] [08:12:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:31:17] PROBLEM - db171 Backups SQL on db171 is WARNING: FILE_AGE WARNING: /var/log/sql-backup.log is 864164 seconds old and 89892 bytes [10:07:33] PROBLEM - hastursnotebook.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hastursnotebook.org' expires in 15 day(s) (Tue 30 Jul 2024 09:48:39 AM GMT +0000). [10:07:48] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d0b0d28c82e1...2e0184e37f8f [10:07:50] [02ssl] 07WikiTideSSLBot 032e0184e - Bot: Update SSL cert for hastursnotebook.org [10:15:58] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:32:48] !log [reception@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/maintenance/importDump.php --wiki=cyberpediawiki /home/reception/undertale_pages_full.xml --username-prefix=wikia:undertale (END - exit=35072) [10:32:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:37:06] RECOVERY - hastursnotebook.org - LetsEncrypt on sslhost is OK: OK - Certificate 'hastursnotebook.org' will expire on Sat 12 Oct 2024 09:07:42 AM GMT +0000. [10:45:38] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 29 seconds ago with 0 failures [10:47:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:59:56] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o