[00:00:02] [02ssl] 07WikiTideSSLBot 03df8e043 - Bot: Update SSL cert for marinebiodiversitymatrix.org [00:20:11] RECOVERY - thelonsdalebattalion.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thelonsdalebattalion.co.uk' will expire on Tue 08 Oct 2024 10:51:41 PM GMT +0000. [00:29:00] RECOVERY - marinebiodiversitymatrix.org - LetsEncrypt on sslhost is OK: OK - Certificate 'marinebiodiversitymatrix.org' will expire on Tue 08 Oct 2024 10:59:51 PM GMT +0000. [00:37:02] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'permanentfuturelab.wiki' expires in 15 day(s) (Sat 27 Jul 2024 12:13:59 AM GMT +0000). [00:37:32] RECOVERY - wiki.gesamtschule-nordkirchen.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gesamtschule-nordkirchen.de' will expire on Tue 08 Oct 2024 10:39:37 PM GMT +0000. [00:43:03] PROBLEM - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'permanentfuturelab.wiki' expires in 15 day(s) (Sat 27 Jul 2024 12:13:59 AM GMT +0000). [00:51:01] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [00:52:56] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [19 system event log (SEL) entries present] [01:03:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 28.51, 23.29, 18.56 [01:05:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.39, 22.04, 18.63 [01:11:12] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.38, 19.67, 18.77 [01:29:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.85, 20.53, 18.69 [01:37:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.97, 23.14, 20.52 [01:39:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.81, 22.38, 20.59 [01:43:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.22, 24.24, 21.68 [01:45:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.94, 23.81, 21.84 [02:25:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.96, 22.94, 21.94 [02:26:42] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [02:28:43] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [21 system event log (SEL) entries present] [02:29:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.21, 22.38, 21.99 [02:37:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.51, 21.95, 21.88 [02:39:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.50, 21.97, 21.88 [02:47:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:00:38] RECOVERY - mon181 Backups Grafana on mon181 is OK: FILE_AGE OK: /var/log/grafana-backup.log is 19 seconds old and 92 bytes [03:13:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.48, 21.74, 21.28 [03:14:42] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [03:15:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 18.54, 20.37, 20.83 [03:16:43] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 24 system event log (SEL) entries present] [03:21:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.13, 21.85, 21.21 [03:23:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.37, 21.94, 21.36 [03:24:13] PROBLEM - adadevelopersacademy.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'adadevelopersacademy.wiki' expires in 15 day(s) (Sat 27 Jul 2024 03:13:46 AM GMT +0000). [03:24:28] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/df8e04375445...71fc1851a1ba [03:24:30] [02ssl] 07WikiTideSSLBot 0371fc185 - Bot: Update SSL cert for adadevelopersacademy.wiki [03:26:20] PROBLEM - www.durawiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.durawiki.com' expires in 15 day(s) (Sat 27 Jul 2024 03:07:43 AM GMT +0000). [03:26:33] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/71fc1851a1ba...20bcd1da56c3 [03:26:34] [02ssl] 07WikiTideSSLBot 0320bcd1d - Bot: Update SSL cert for www.durawiki.com [03:29:10] PROBLEM - wiki.tulpa.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.tulpa.info' expires in 15 day(s) (Sat 27 Jul 2024 03:18:22 AM GMT +0000). [03:29:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.91, 23.94, 22.37 [03:29:21] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/20bcd1da56c3...dabf7d5bbcd8 [03:29:22] [02ssl] 07WikiTideSSLBot 03dabf7d5 - Bot: Update SSL cert for wiki.tulpa.info [03:30:01] PROBLEM - wiki.autocountsoft.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.autocountsoft.com' expires in 15 day(s) (Sat 27 Jul 2024 03:12:15 AM GMT +0000). [03:30:13] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/dabf7d5bbcd8...7d225e65cfa7 [03:30:14] [02ssl] 07WikiTideSSLBot 037d225e6 - Bot: Update SSL cert for wiki.autocountsoft.com [03:31:53] PROBLEM - publictestwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'publictestwiki.com' expires in 15 day(s) (Sat 27 Jul 2024 03:16:29 AM GMT +0000). [03:37:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.41, 23.34, 23.05 [03:39:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.43, 24.92, 23.67 [03:39:58] PROBLEM - www.publictestwiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'publictestwiki.com' expires in 15 day(s) (Sat 27 Jul 2024 03:16:29 AM GMT +0000). [03:41:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.79, 23.49, 23.28 [03:41:40] PROBLEM - www.zenbuddhism.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.zenbuddhism.info' expires in 15 day(s) (Sat 27 Jul 2024 03:20:46 AM GMT +0000). [03:41:52] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7d225e65cfa7...bc6adcf51a73 [03:41:55] [02ssl] 07WikiTideSSLBot 03bc6adcf - Bot: Update SSL cert for www.zenbuddhism.info [03:43:10] PROBLEM - wiki.make717.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.make717.org' expires in 15 day(s) (Sat 27 Jul 2024 03:17:42 AM GMT +0000). [03:43:21] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bc6adcf51a73...a949414cf48c [03:43:24] [02ssl] 07WikiTideSSLBot 03a949414 - Bot: Update SSL cert for wiki.make717.org [03:53:57] RECOVERY - adadevelopersacademy.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'adadevelopersacademy.wiki' will expire on Wed 09 Oct 2024 02:24:22 AM GMT +0000. [03:55:10] RECOVERY - www.durawiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.durawiki.com' will expire on Wed 09 Oct 2024 02:26:27 AM GMT +0000. [03:57:12] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 13.07, 17.51, 20.27 [03:57:50] RECOVERY - wiki.tulpa.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.tulpa.info' will expire on Wed 09 Oct 2024 02:29:16 AM GMT +0000. [03:59:32] RECOVERY - wiki.autocountsoft.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.autocountsoft.com' will expire on Wed 09 Oct 2024 02:30:07 AM GMT +0000. [04:02:42] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [04:04:43] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 26 system event log (SEL) entries present] [04:10:50] RECOVERY - www.zenbuddhism.info - LetsEncrypt on sslhost is OK: OK - Certificate 'www.zenbuddhism.info' will expire on Wed 09 Oct 2024 02:41:46 AM GMT +0000. [04:12:19] RECOVERY - wiki.make717.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.make717.org' will expire on Wed 09 Oct 2024 02:43:15 AM GMT +0000. [04:21:39] PROBLEM - changeprop151 Puppet on changeprop151 is CRITICAL: CRITICAL: Puppet last ran 12 hours ago [04:58:50] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [05:00:45] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [27 system event log (SEL) entries present] [05:50:42] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [05:52:43] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [28 system event log (SEL) entries present] [06:12:01] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.80, 20.23, 17.19 [06:13:55] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 31.85, 23.66, 18.73 [06:15:49] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.25, 22.42, 18.90 [06:19:38] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.49, 22.81, 19.73 [06:23:26] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.79, 22.34, 20.30 [06:27:14] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 15.76, 19.22, 19.51 [06:31:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.66, 20.28, 19.87 [06:33:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.29, 21.42, 20.31 [06:35:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.41, 22.25, 20.76 [06:39:12] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 16.51, 19.97, 20.26 [06:43:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.49, 20.60, 20.45 [06:47:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:55:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.94, 22.80, 21.50 [06:57:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 17.50, 20.72, 20.89 [07:05:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.83, 23.55, 21.90 [07:09:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.15, 22.87, 22.07 [07:25:12] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 18.93, 19.53, 20.38 [07:33:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.36, 20.24, 20.30 [07:35:12] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.70, 19.38, 19.98 [07:47:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.57, 22.42, 20.78 [07:59:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 28.71, 23.51, 21.74 [08:01:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.91, 23.68, 22.04 [08:03:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.57, 24.37, 22.49 [08:07:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.65, 22.43, 22.19 [08:21:12] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.65, 19.31, 20.23 [08:27:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.36, 21.87, 21.16 [08:30:42] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [08:32:43] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [30 system event log (SEL) entries present] [08:45:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.73, 22.58, 21.74 [08:47:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.32, 21.53, 21.43 [08:50:51] PROBLEM - enc.for.uz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'enc.for.uz' expires in 15 day(s) (Sat 27 Jul 2024 08:40:42 AM GMT +0000). [08:51:06] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a949414cf48c...a25f1682a173 [08:51:08] [02ssl] 07WikiTideSSLBot 03a25f168 - Bot: Update SSL cert for enc.for.uz [08:55:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.68, 22.40, 21.84 [08:57:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.03, 20.96, 21.38 [09:01:07] PROBLEM - nonciclopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'nonciclopedia.org' expires in 15 day(s) (Sat 27 Jul 2024 08:39:58 AM GMT +0000). [09:01:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.74, 22.37, 21.77 [09:01:19] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a25f1682a173...8f788c18d95d [09:01:22] [02ssl] 07WikiTideSSLBot 038f788c1 - Bot: Update SSL cert for nonciclopedia.org [09:02:53] PROBLEM - pwiki.arkcls.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pwiki.arkcls.com' expires in 15 day(s) (Sat 27 Jul 2024 09:00:44 AM GMT +0000). [09:03:04] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8f788c18d95d...4409c9a9b020 [09:03:06] [02ssl] 07WikiTideSSLBot 034409c9a - Bot: Update SSL cert for pwiki.arkcls.com [09:03:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.43, 22.53, 21.89 [09:07:06] PROBLEM - runzeppelin.ru - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'runzeppelin.ru' expires in 15 day(s) (Sat 27 Jul 2024 09:05:03 AM GMT +0000). [09:07:22] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4409c9a9b020...4e1bd02d543e [09:07:23] [02ssl] 07WikiTideSSLBot 034e1bd02 - Bot: Update SSL cert for runzeppelin.ru [09:14:42] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [09:16:43] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [30 system event log (SEL) entries present] [09:20:11] RECOVERY - enc.for.uz - LetsEncrypt on sslhost is OK: OK - Certificate 'enc.for.uz' will expire on Wed 09 Oct 2024 07:51:00 AM GMT +0000. [09:23:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.35, 22.35, 21.91 [09:25:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.67, 22.75, 22.11 [09:30:14] RECOVERY - nonciclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'nonciclopedia.org' will expire on Wed 09 Oct 2024 08:01:13 AM GMT +0000. [09:32:16] RECOVERY - pwiki.arkcls.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.arkcls.com' will expire on Wed 09 Oct 2024 08:02:59 AM GMT +0000. [09:36:11] RECOVERY - runzeppelin.ru - LetsEncrypt on sslhost is OK: OK - Certificate 'runzeppelin.ru' will expire on Wed 09 Oct 2024 08:07:14 AM GMT +0000. [09:44:14] PROBLEM - stablestate.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'stablestate.org' expires in 15 day(s) (Sat 27 Jul 2024 09:33:50 AM GMT +0000). [09:44:25] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4e1bd02d543e...054c5b2a6b5d [09:44:26] [02ssl] 07WikiTideSSLBot 03054c5b2 - Bot: Update SSL cert for stablestate.org [09:45:07] PROBLEM - www.mh142.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mh142.com' expires in 15 day(s) (Sat 27 Jul 2024 09:40:05 AM GMT +0000). [09:55:20] PROBLEM - wiki.starship.digital - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.starship.digital' expires in 15 day(s) (Sat 27 Jul 2024 09:28:10 AM GMT +0000). [09:55:32] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/054c5b2a6b5d...9eb2fde394e6 [09:55:33] [02ssl] 07WikiTideSSLBot 039eb2fde - Bot: Update SSL cert for wiki.starship.digital [09:57:12] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 16.35, 18.98, 20.13 [10:03:56] PROBLEM - mh142.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mh142.com' expires in 15 day(s) (Sat 27 Jul 2024 09:40:05 AM GMT +0000). [10:04:17] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9eb2fde394e6...abb2e6efaab9 [10:04:19] [02ssl] 07WikiTideSSLBot 03abb2e6e - Bot: Update SSL cert for mh142.com [10:10:42] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [10:11:16] PROBLEM - wiki.thunis.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.thunis.eu' expires in 15 day(s) (Sat 27 Jul 2024 10:05:50 AM GMT +0000). [10:11:28] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/abb2e6efaab9...752d0b44435e [10:11:30] [02ssl] 07WikiTideSSLBot 03752d0b4 - Bot: Update SSL cert for wiki.thunis.eu [10:12:43] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [30 system event log (SEL) entries present] [10:14:14] PROBLEM - www.programming.red - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'programming.red' expires in 15 day(s) (Sat 27 Jul 2024 10:08:36 AM GMT +0000). [10:14:19] PROBLEM - programming.red - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'programming.red' expires in 15 day(s) (Sat 27 Jul 2024 10:08:36 AM GMT +0000). [10:14:37] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/752d0b44435e...4dd500e031ce [10:14:40] [02ssl] 07WikiTideSSLBot 034dd500e - Bot: Update SSL cert for programming.red [10:15:10] PROBLEM - dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Sat 27 Jul 2024 10:01:56 AM GMT +0000). [10:24:40] RECOVERY - wiki.starship.digital - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.starship.digital' will expire on Wed 09 Oct 2024 08:55:26 AM GMT +0000. [10:29:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.40, 19.89, 19.17 [10:31:12] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.99, 19.95, 19.29 [10:31:49] PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'dariawiki.org' expires in 15 day(s) (Sat 27 Jul 2024 10:01:56 AM GMT +0000). [10:32:53] RECOVERY - mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Wed 09 Oct 2024 09:04:11 AM GMT +0000. [10:40:03] RECOVERY - wiki.thunis.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.thunis.eu' will expire on Wed 09 Oct 2024 09:11:22 AM GMT +0000. [10:41:12] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.97, 21.02, 20.12 [10:42:14] RECOVERY - stablestate.org - LetsEncrypt on sslhost is OK: OK - Certificate 'stablestate.org' will expire on Wed 09 Oct 2024 08:44:19 AM GMT +0000. [10:42:58] RECOVERY - www.programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Wed 09 Oct 2024 09:14:31 AM GMT +0000. [10:43:08] RECOVERY - programming.red - LetsEncrypt on sslhost is OK: OK - Certificate 'programming.red' will expire on Wed 09 Oct 2024 09:14:31 AM GMT +0000. [10:43:08] PROBLEM - kunwok.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kunwok.org' expires in 15 day(s) (Sat 27 Jul 2024 10:31:56 AM GMT +0000). [10:43:12] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.66, 20.22, 19.96 [10:43:20] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4dd500e031ce...7760a4f69d0b [10:43:22] [02ssl] 07WikiTideSSLBot 037760a4f - Bot: Update SSL cert for kunwok.org [10:44:21] RECOVERY - www.mh142.com - LetsEncrypt on sslhost is OK: OK - Certificate 'mh142.com' will expire on Wed 09 Oct 2024 09:04:11 AM GMT +0000. [10:47:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:50:01] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.28, 20.97, 20.39 [10:53:49] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.30, 20.12, 20.17 [10:56:28] PROBLEM - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'encyclopediaofastrobiology.org' expires in 15 day(s) (Sat 27 Jul 2024 10:33:47 AM GMT +0000). [10:56:39] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7760a4f69d0b...f6e1242962b6 [10:56:42] [02ssl] 07WikiTideSSLBot 03f6e1242 - Bot: Update SSL cert for encyclopediaofastrobiology.org [11:01:26] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.10, 19.66, 20.03 [11:11:01] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [11:12:55] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [30 system event log (SEL) entries present] [11:17:55] PROBLEM - podpedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'podpedia.org' expires in 15 day(s) (Sat 27 Jul 2024 11:02:39 AM GMT +0000). [11:18:07] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f6e1242962b6...894892f78022 [11:18:10] [02ssl] 07WikiTideSSLBot 03894892f - Bot: Update SSL cert for podpedia.org [11:25:25] RECOVERY - encyclopediaofastrobiology.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediaofastrobiology.org' will expire on Wed 09 Oct 2024 09:56:34 AM GMT +0000. [11:29:27] PROBLEM - monarchists.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'monarchists.wiki' expires in 15 day(s) (Sat 27 Jul 2024 11:01:07 AM GMT +0000). [11:29:39] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/894892f78022...d330ed78530c [11:29:40] [02ssl] 07WikiTideSSLBot 03d330ed7 - Bot: Update SSL cert for monarchists.wiki [11:41:09] PROBLEM - vitriol.spoon.army - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vitriol.spoon.army' expires in 15 day(s) (Sat 27 Jul 2024 11:36:52 AM GMT +0000). [11:41:21] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d330ed78530c...a2fb2fa604d3 [11:41:22] [02ssl] 07WikiTideSSLBot 03a2fb2fa - Bot: Update SSL cert for vitriol.spoon.army [11:41:24] RECOVERY - kunwok.org - LetsEncrypt on sslhost is OK: OK - Certificate 'kunwok.org' will expire on Wed 09 Oct 2024 09:43:14 AM GMT +0000. [11:42:06] PROBLEM - wiki.whentheycry.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.whentheycry.org' expires in 15 day(s) (Sat 27 Jul 2024 11:34:36 AM GMT +0000). [11:42:17] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a2fb2fa604d3...a6304fbe7603 [11:42:19] [02ssl] 07WikiTideSSLBot 03a6304fb - Bot: Update SSL cert for wiki.whentheycry.org [11:58:24] RECOVERY - monarchists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'monarchists.wiki' will expire on Wed 09 Oct 2024 10:29:33 AM GMT +0000. [12:07:20] PROBLEM - vise.dayid.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'vise.dayid.org' expires in 15 day(s) (Sat 27 Jul 2024 11:59:36 AM GMT +0000). [12:07:34] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a6304fbe7603...5449528aeb2b [12:07:35] [02ssl] 07WikiTideSSLBot 035449528 - Bot: Update SSL cert for vise.dayid.org [12:10:57] RECOVERY - vitriol.spoon.army - LetsEncrypt on sslhost is OK: OK - Certificate 'vitriol.spoon.army' will expire on Wed 09 Oct 2024 10:41:15 AM GMT +0000. [12:11:41] RECOVERY - wiki.whentheycry.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.whentheycry.org' will expire on Wed 09 Oct 2024 10:42:11 AM GMT +0000. [12:13:45] [02landing] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/landing/compare/8d195a9c5131...a822bf47bcfc [12:13:47] [02landing] 07translatewiki 03a822bf4 - Localisation updates from https://translatewiki.net. [12:13:48] [02CreateWiki] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/184b5cd3760d...325e6027b70e [12:13:50] [02CreateWiki] 07translatewiki 03325e602 - Localisation updates from https://translatewiki.net. [12:13:52] [02MatomoAnalytics] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MatomoAnalytics/compare/7d15dc2dd601...1269e1e0a355 [12:13:54] [02MatomoAnalytics] 07translatewiki 031269e1e - Localisation updates from https://translatewiki.net. [12:13:57] [02MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+1/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/a6ceed97bcab...d451938e0c81 [12:14:00] [02MirahezeMagic] 07translatewiki 03d451938 - Localisation updates from https://translatewiki.net. [12:14:39] miraheze/landing - translatewiki the build passed. [12:14:42] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [12:16:17] PROBLEM - www.lab612.at - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.lab612.at' expires in 15 day(s) (Sat 27 Jul 2024 12:04:24 PM GMT +0000). [12:16:29] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5449528aeb2b...09a4067b43a0 [12:16:31] [02ssl] 07WikiTideSSLBot 0309a4067 - Bot: Update SSL cert for www.lab612.at [12:16:43] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [30 system event log (SEL) entries present] [12:16:46] RECOVERY - podpedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'podpedia.org' will expire on Wed 09 Oct 2024 10:18:01 AM GMT +0000. [12:17:32] miraheze/MirahezeMagic - translatewiki the build has errored. [12:18:40] miraheze/MatomoAnalytics - translatewiki the build passed. [12:22:55] miraheze/CreateWiki - translatewiki the build passed. [12:23:50] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [12:23:51] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:23:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:23:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:03] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [12:24:04] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:14] !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 [12:24:15] !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:48] !log [@mwtask181] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:32:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:58] !log [@mwtask181] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 10s [12:33:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:10] !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:33:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:20] !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 10s [12:33:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:36:40] RECOVERY - vise.dayid.org - LetsEncrypt on sslhost is OK: OK - Certificate 'vise.dayid.org' will expire on Wed 09 Oct 2024 11:07:25 AM GMT +0000. [12:37:55] !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [12:38:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:04] !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 9s [12:38:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:15] !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:38:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:24] !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 9s [12:38:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:39:14] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.84, 20.56, 19.91 [12:41:12] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 15.05, 18.42, 19.20 [13:01:10] PROBLEM - bristolstudenthousingcoop.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bristolstudenthousingcoop.org' expires in 15 day(s) (Sat 27 Jul 2024 12:37:08 PM GMT +0000). [13:01:19] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/09a4067b43a0...3ce50df94a9f [13:01:21] [02ssl] 07WikiTideSSLBot 033ce50df - Bot: Update SSL cert for bristolstudenthousingcoop.org [13:04:09] PROBLEM - bebaskanpengetahuan.id - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'bebaskanpengetahuan.id' expires in 15 day(s) (Sat 27 Jul 2024 12:38:55 PM GMT +0000). [13:04:21] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3ce50df94a9f...7e8c765f2f7e [13:04:23] [02ssl] 07WikiTideSSLBot 037e8c765 - Bot: Update SSL cert for bebaskanpengetahuan.id [13:10:45] PROBLEM - portalsofphereon.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.portalsofphereon.com' expires in 15 day(s) (Sat 27 Jul 2024 01:00:01 PM GMT +0000). [13:16:03] RECOVERY - www.lab612.at - LetsEncrypt on sslhost is OK: OK - Certificate 'www.lab612.at' will expire on Wed 09 Oct 2024 11:16:23 AM GMT +0000. [13:17:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [13:20:42] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [13:22:43] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [30 system event log (SEL) entries present] [13:29:03] PROBLEM - www.portalsofphereon.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.portalsofphereon.com' expires in 15 day(s) (Sat 27 Jul 2024 01:00:01 PM GMT +0000). [13:29:18] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7e8c765f2f7e...991533959cf6 [13:29:20] [02ssl] 07WikiTideSSLBot 039915339 - Bot: Update SSL cert for www.portalsofphereon.com [13:30:14] RECOVERY - bristolstudenthousingcoop.org - LetsEncrypt on sslhost is OK: OK - Certificate 'bristolstudenthousingcoop.org' will expire on Wed 09 Oct 2024 12:01:13 PM GMT +0000. [13:34:03] RECOVERY - bebaskanpengetahuan.id - LetsEncrypt on sslhost is OK: OK - Certificate 'bebaskanpengetahuan.id' will expire on Wed 09 Oct 2024 12:04:15 PM GMT +0000. [13:57:35] RECOVERY - www.portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Wed 09 Oct 2024 12:29:12 PM GMT +0000. [14:10:13] RECOVERY - portalsofphereon.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.portalsofphereon.com' will expire on Wed 09 Oct 2024 12:29:12 PM GMT +0000. [14:40:42] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [14:42:43] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [30 system event log (SEL) entries present] [14:59:55] [02mw-config] 07redbluegreenhat opened pull request 03#5605: Change schema location of UrlShortener in 1.42 - 13https://github.com/miraheze/mw-config/pull/5605 [15:00:49] miraheze/mw-config - redbluegreenhat the build has errored. [15:01:21] [02mw-config] 07redbluegreenhat synchronize pull request 03#5605: Change schema location of UrlShortener in 1.42 - 13https://github.com/miraheze/mw-config/pull/5605 [15:02:21] miraheze/mw-config - redbluegreenhat the build passed. [15:02:46] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:04:39] [02mediawiki-repos] 07redbluegreenhat pushed 031 commit to 03revert-24-socialtools-checkout-commit [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/commit/4326dc35e6c9 [15:04:40] [02mediawiki-repos] 07redbluegreenhat 034326dc3 - Revert "checkout a specific commit on the social tools extensions (#24)" [15:04:43] [02mediawiki-repos] 07redbluegreenhat created branch 03revert-24-socialtools-checkout-commit - 13https://github.com/miraheze/mediawiki-repos [15:04:46] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0004786550999 secs [15:04:54] [02mediawiki-repos] 07redbluegreenhat opened pull request 03#26: Revert "checkout a specific commit on the social tools extensions" - 13https://github.com/miraheze/mediawiki-repos/pull/26 [15:04:59] [02mediawiki-repos] 07coderabbitai[bot] commented on pull request 03#26: Revert "checkout a specific commit on the social tools extensions" - 13https://github.com/miraheze/mediawiki-repos/pull/26#issuecomment-2223178469 [15:06:06] [02mediawiki-repos] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/3289dedde02a...1bff30f6f8cf [15:06:09] [02mediawiki-repos] 07redbluegreenhat 031bff30f - Revert "checkout a specific commit on the social tools extensions (#24)" (#26) [15:06:09] [02mediawiki-repos] 07redbluegreenhat closed pull request 03#26: Revert "checkout a specific commit on the social tools extensions" - 13https://github.com/miraheze/mediawiki-repos/pull/26 [15:06:10] [02mediawiki-repos] 07redbluegreenhat deleted branch 03revert-24-socialtools-checkout-commit [15:06:11] [02mediawiki-repos] 07redbluegreenhat deleted branch 03revert-24-socialtools-checkout-commit - 13https://github.com/miraheze/mediawiki-repos [15:14:51] !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'all'} to test151 [15:14:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:17:39] !log interrupted mwdeploy [15:17:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:23:28] [02mediawiki-repos] 07redbluegreenhat created branch 03socialtools-master - 13https://github.com/miraheze/mediawiki-repos [15:23:30] [02mediawiki-repos] 07redbluegreenhat pushed 031 commit to 03socialtools-master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/commit/06b064b67ff0 [15:23:32] [02mediawiki-repos] 07redbluegreenhat 0306b064b - Use the master branch on the social tools extensions [15:23:42] [02mediawiki-repos] 07redbluegreenhat opened pull request 03#27: Use the master branch on the social tools extensions - 13https://github.com/miraheze/mediawiki-repos/pull/27 [15:23:46] [02mediawiki-repos] 07coderabbitai[bot] commented on pull request 03#27: Use the master branch on the social tools extensions - 13https://github.com/miraheze/mediawiki-repos/pull/27#issuecomment-2223236895 [15:24:26] [02mediawiki-repos] 07redbluegreenhat pushed 031 commit to 03socialtools-master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/06b064b67ff0...f488f3cba0a8 [15:24:27] [02mediawiki-repos] 07redbluegreenhat 03f488f3c - oops [15:24:29] [02mediawiki-repos] 07redbluegreenhat synchronize pull request 03#27: Use the master branch on the social tools extensions - 13https://github.com/miraheze/mediawiki-repos/pull/27 [15:25:18] [02mediawiki-repos] 07redbluegreenhat closed pull request 03#27: Use the master branch on the social tools extensions - 13https://github.com/miraheze/mediawiki-repos/pull/27 [15:25:20] [02mediawiki-repos] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/1bff30f6f8cf...ea1e891db4cc [15:25:21] [02mediawiki-repos] 07redbluegreenhat 03ea1e891 - Use the master branch on the social tools extensions (#27) [15:25:22] [02mediawiki-repos] 07redbluegreenhat deleted branch 03socialtools-master [15:25:23] [02mediawiki-repos] 07redbluegreenhat deleted branch 03socialtools-master - 13https://github.com/miraheze/mediawiki-repos [15:28:04] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 3 minutes ago with 10 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_41 AJAXPoll],Exec[git_checkout_MediaWiki-REL1_41 Comments],Exec[git_checkout_MediaWiki-REL1_41 QuizGame],Exec[git_checkout_MediaWiki-REL1_41 VoteNY] [15:35:44] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 2 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_41 Comments],Exec[git_checkout_MediaWiki-REL1_42 Comments] [15:38:57] [02mediawiki-repos] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/ea1e891db4cc...9aeb383d709e [15:38:59] [02mediawiki-repos] 07redbluegreenhat 039aeb383 - fix Comments branch [15:41:06] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 2 failures. Last run 3 minutes ago with 2 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_41 Comments],Exec[git_checkout_MediaWiki-REL1_42 Comments] [15:43:54] !log [alex@test151] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'all'} to test151 [15:43:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:44:12] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:46:25] !log [alex@test151] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'all'} to test151 - SUCCESS in 150s [15:46:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:46:52] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [15:47:08] !log [alex@test151] starting deploy of {'upgrade_world': True, 'versions': '1.42', 'upgrade_extensions': 'all', 'upgrade_skins': 'all'} to test151 [15:47:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:48:47] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [30 system event log (SEL) entries present] [15:55:10] !log [alex@test151] finished deploy of {'upgrade_world': True, 'versions': '1.42', 'upgrade_extensions': 'all', 'upgrade_skins': 'all'} to test151 - SUCCESS in 482s [15:55:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [15:58:40] [02mw-config] 07redbluegreenhat synchronize pull request 03#5605: Change schema location of UrlShortener in 1.42 - 13https://github.com/miraheze/mw-config/pull/5605 [15:59:37] miraheze/mw-config - redbluegreenhat the build passed. [16:03:44] RECOVERY - mwtask181 Puppet on mwtask181 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:09:04] RECOVERY - mwtask171 Puppet on mwtask171 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:12:54] [02mw-config] 07redbluegreenhat closed pull request 03#5605: Change schema location of UrlShortener in 1.42 - 13https://github.com/miraheze/mw-config/pull/5605 [16:12:56] [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/230ea0bfbb47...f9a26469b0fb [16:12:58] [02mw-config] 07redbluegreenhat 03f9a2646 - Change schema location of UrlShortener in 1.42 (#5605) [16:14:00] miraheze/mw-config - redbluegreenhat the build passed. [16:14:21] !log [alex@test151] starting deploy of {'pull': 'config', 'config': True} to test151 [16:14:22] !log [alex@test151] finished deploy of {'pull': 'config', 'config': True} to test151 - SUCCESS in 0s [16:14:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:14:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:23:24] !log [alex@test151] starting deploy of {'upgrade_world': True, 'versions': '1.41', 'upgrade_extensions': 'all', 'upgrade_skins': 'all'} to test151 [16:23:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:24:16] I'm going to try to get the list of SQL patches and possible maint script that need running for the upgrade. Upgrading 1.41 to make sure we don't miss anything [16:24:58] [1/21] ``` [16:24:58] [2/21] /srv/mediawiki-staging/1.42/extensions/CheckUser/schema/mysql/cu_useragent.sql [16:24:59] [3/21] /srv/mediawiki-staging/1.42/extensions/CheckUser/schema/mysql/patch-cu_changes-add-cuc_agent_id.sql [16:24:59] [4/21] /srv/mediawiki-staging/1.42/extensions/CheckUser/schema/mysql/patch-cu_changes-modify-cuc_id-bigint.sql [16:24:59] [5/21] /srv/mediawiki-staging/1.42/extensions/CheckUser/schema/mysql/patch-cu_log-add-cul_result_id.sql [16:25:00] [6/21] /srv/mediawiki-staging/1.42/extensions/CheckUser/schema/mysql/patch-cu_log_event-add-cule_agent_id.sql [16:25:00] [7/21] /srv/mediawiki-staging/1.42/extensions/CheckUser/schema/mysql/patch-cu_private_event-add-cupe_agent_id.sql [16:25:00] [8/21] /srv/mediawiki-staging/1.42/extensions/CheckUser/schema/mysql/patch-cu_private_event-modify-cupe_actor-nullable.sql [16:25:01] [9/21] /srv/mediawiki-staging/1.42/extensions/SecurePoll/cli/wm-scripts/u4c202404/u4c202404_tables.sql [16:25:01] [10/21] /srv/mediawiki-staging/1.42/extensions/SecurePoll/cli/wm-scripts/u4c2024/u4c2024_tables.sql [16:25:01] [11/21] /srv/mediawiki-staging/1.42/extensions/GlobalBlocking/sql/mysql/patch-add-gb_target_central_id.sql [16:25:01] [12/21] /srv/mediawiki-staging/1.42/extensions/GlobalBlocking/sql/mysql/patch-add-gbw_target_central_id.sql [16:25:02] [13/21] /srv/mediawiki-staging/1.42/extensions/GlobalBlocking/sql/mysql/patch-modify-gb_by-default.sql [16:25:02] [14/21] /srv/mediawiki-staging/1.42/extensions/CentralAuth/schema/mysql/patch-add-global_user_autocreate_serial.sql [16:25:03] [15/21] /srv/mediawiki-staging/1.42/extensions/CentralAuth/schema/mysql/patch-add-uas_year.sql [16:25:03] [16/21] /srv/mediawiki-staging/1.42/maintenance/archives/patch-block_target.sql [16:25:04] [17/21] /srv/mediawiki-staging/1.42/maintenance/archives/patch-drop-cl_collation_ext.sql [16:25:04] [18/21] /srv/mediawiki-staging/1.42/maintenance/archives/patch-iwlinks-drop-iwl_prefix_from_title.sql [16:25:05] [19/21] /srv/mediawiki-staging/1.42/maintenance/archives/patch-sites-drop_indexes.sql [16:25:05] [20/21] /srv/mediawiki-staging/1.42/maintenance/archives/patch-user_autocreate_serial-uas_year.sql``` [16:25:06] [21/21] Are the SQL patches I'm getting before doing the upgrade [16:26:43] !log interrupted deploy [16:26:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:32:33] !log [@mwtask181] starting deploy of {'config': True} to all [16:32:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:32:44] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 10s [16:32:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:37] !log [@mwtask171] starting deploy of {'config': True} to all [16:37:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:37:50] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 13s [16:37:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [16:54:42] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [16:56:43] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [32 system event log (SEL) entries present] [17:01:13] PROBLEM - history.sdtef.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'history.sdtef.org' expires in 15 day(s) (Sat 27 Jul 2024 04:31:19 PM GMT +0000). [17:01:26] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/991533959cf6...c205025bfe2e [17:01:29] [02ssl] 07WikiTideSSLBot 03c205025 - Bot: Update SSL cert for history.sdtef.org [17:22:36] PROBLEM - infectowiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'infectowiki.com' expires in 15 day(s) (Sat 27 Jul 2024 04:56:30 PM GMT +0000). [17:22:50] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c205025bfe2e...8c77d38dbb6a [17:22:51] [02ssl] 07WikiTideSSLBot 038c77d38 - Bot: Update SSL cert for infectowiki.com [17:30:26] RECOVERY - history.sdtef.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.sdtef.org' will expire on Wed 09 Oct 2024 04:01:19 PM GMT +0000. [17:35:10] PROBLEM - icclopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'icclopedia.org' expires in 15 day(s) (Sat 27 Jul 2024 05:33:08 PM GMT +0000). [17:35:22] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8c77d38dbb6a...f772d88e8f6d [17:35:25] [02ssl] 07WikiTideSSLBot 03f772d88 - Bot: Update SSL cert for icclopedia.org [17:47:04] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:52:11] RECOVERY - infectowiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'infectowiki.com' will expire on Wed 09 Oct 2024 04:22:42 PM GMT +0000. [18:00:32] PROBLEM - history.estill.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'history.estill.org' expires in 15 day(s) (Sat 27 Jul 2024 05:44:33 PM GMT +0000). [18:00:43] PROBLEM - index.pdcommunity.ir - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'index.pdcommunity.ir' expires in 15 day(s) (Sat 27 Jul 2024 05:29:12 PM GMT +0000). [18:00:43] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f772d88e8f6d...4b02eba2a7c7 [18:00:45] [02ssl] 07WikiTideSSLBot 034b02eba - Bot: Update SSL cert for history.estill.org [18:00:58] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4b02eba2a7c7...b928665fb633 [18:01:01] [02ssl] 07WikiTideSSLBot 03b928665 - Bot: Update SSL cert for index.pdcommunity.ir [18:03:49] RECOVERY - icclopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'icclopedia.org' will expire on Wed 09 Oct 2024 04:35:15 PM GMT +0000. [18:03:49] PROBLEM - allthetropes.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Sat 27 Jul 2024 05:53:43 PM GMT +0000). [18:05:08] PROBLEM - isv.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Sat 27 Jul 2024 05:53:43 PM GMT +0000). [18:06:42] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [18:07:20] PROBLEM - poserdazfreebies.orain.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'orain.org' expires in 15 day(s) (Sat 27 Jul 2024 05:53:43 PM GMT +0000). [18:08:43] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [33 system event log (SEL) entries present] [18:19:12] PROBLEM - iceria.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.iceria.org' expires in 15 day(s) (Sat 27 Jul 2024 06:17:08 PM GMT +0000). [18:29:03] RECOVERY - history.estill.org - LetsEncrypt on sslhost is OK: OK - Certificate 'history.estill.org' will expire on Wed 09 Oct 2024 05:00:37 PM GMT +0000. [18:29:26] RECOVERY - index.pdcommunity.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'index.pdcommunity.ir' will expire on Wed 09 Oct 2024 05:00:52 PM GMT +0000. [18:29:53] PROBLEM - wiki.mineland.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mineland.eu' expires in 15 day(s) (Sat 27 Jul 2024 06:01:27 PM GMT +0000). [18:30:09] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b928665fb633...ed6fdf602219 [18:30:12] [02ssl] 07WikiTideSSLBot 03ed6fdf6 - Bot: Update SSL cert for wiki.mineland.eu [18:33:54] PROBLEM - wkmr.liao.media - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wkmr.liao.media' expires in 15 day(s) (Sat 27 Jul 2024 06:09:40 PM GMT +0000). [18:34:05] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ed6fdf602219...124964805048 [18:34:07] [02ssl] 07WikiTideSSLBot 031249648 - Bot: Update SSL cert for wkmr.liao.media [18:40:11] PROBLEM - www.iceria.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.iceria.org' expires in 15 day(s) (Sat 27 Jul 2024 06:17:08 PM GMT +0000). [18:40:26] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/124964805048...993238da3d23 [18:40:27] [02ssl] 07WikiTideSSLBot 03993238d - Bot: Update SSL cert for www.iceria.org [18:44:51] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:48:04] PROBLEM - tl.awiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tl.awiki.org' expires in 15 day(s) (Sat 27 Jul 2024 06:36:04 PM GMT +0000). [18:51:06] PROBLEM - wiki.ssij.se - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ssij.se' expires in 15 day(s) (Sat 27 Jul 2024 06:30:41 PM GMT +0000). [18:51:19] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/993238da3d23...fe1c88fa4163 [18:51:20] [02ssl] 07WikiTideSSLBot 03fe1c88f - Bot: Update SSL cert for wiki.ssij.se [18:52:24] PROBLEM - fc.songcontests.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fc.songcontests.eu' expires in 15 day(s) (Sat 27 Jul 2024 06:37:40 PM GMT +0000). [18:52:35] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fe1c88fa4163...bb8e5ce921a0 [18:52:38] [02ssl] 07WikiTideSSLBot 03bb8e5ce - Bot: Update SSL cert for fc.songcontests.eu [18:59:17] RECOVERY - wiki.mineland.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mineland.eu' will expire on Wed 09 Oct 2024 05:30:02 PM GMT +0000. [19:02:48] RECOVERY - wkmr.liao.media - LetsEncrypt on sslhost is OK: OK - Certificate 'wkmr.liao.media' will expire on Wed 09 Oct 2024 05:33:59 PM GMT +0000. [19:02:50] PROBLEM - pubwiki.lab.co.pl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pubwiki.lab.co.pl' expires in 15 day(s) (Sat 27 Jul 2024 06:59:28 PM GMT +0000). [19:03:01] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bb8e5ce921a0...0cbc67016f2c [19:03:04] [02ssl] 07WikiTideSSLBot 030cbc670 - Bot: Update SSL cert for pubwiki.lab.co.pl [19:08:42] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [19:09:23] RECOVERY - www.iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Wed 09 Oct 2024 05:40:21 PM GMT +0000. [19:10:43] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [34 system event log (SEL) entries present] [19:14:14] PROBLEM - guia.esporo.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'guia.esporo.net' expires in 15 day(s) (Sat 27 Jul 2024 06:50:42 PM GMT +0000). [19:14:25] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0cbc67016f2c...e85b986124dd [19:14:27] [02ssl] 07WikiTideSSLBot 03e85b986 - Bot: Update SSL cert for guia.esporo.net [19:17:36] RECOVERY - iceria.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.iceria.org' will expire on Wed 09 Oct 2024 05:40:21 PM GMT +0000. [19:20:41] RECOVERY - wiki.ssij.se - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ssij.se' will expire on Wed 09 Oct 2024 05:51:12 PM GMT +0000. [19:22:16] RECOVERY - fc.songcontests.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'fc.songcontests.eu' will expire on Wed 09 Oct 2024 05:52:29 PM GMT +0000. [19:32:09] RECOVERY - pubwiki.lab.co.pl - LetsEncrypt on sslhost is OK: OK - Certificate 'pubwiki.lab.co.pl' will expire on Wed 09 Oct 2024 06:02:55 PM GMT +0000. [19:41:52] PROBLEM - mwtask171 APT on mwtask171 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (4 critical updates). [19:42:57] RECOVERY - guia.esporo.net - LetsEncrypt on sslhost is OK: OK - Certificate 'guia.esporo.net' will expire on Wed 09 Oct 2024 06:14:19 PM GMT +0000. [19:43:09] PROBLEM - prometheus151 APT on prometheus151 is CRITICAL: APT CRITICAL: 57 packages available for upgrade (4 critical updates). [19:45:11] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:49:34] PROBLEM - franchise.franchising.org.ua - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'franchise.franchising.org.ua' expires in 15 day(s) (Sat 27 Jul 2024 07:41:51 PM GMT +0000). [19:49:34] PROBLEM - baharna.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.baharna.org' expires in 15 day(s) (Sat 27 Jul 2024 07:29:48 PM GMT +0000). [19:49:59] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e85b986124dd...71a81814bc91 [19:50:02] [02ssl] 07WikiTideSSLBot 0371a8181 - Bot: Update SSL cert for franchise.franchising.org.ua [19:50:29] PROBLEM - wiki.lefrenchmelee.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.lefrenchmelee.fr' expires in 15 day(s) (Sat 27 Jul 2024 07:38:26 PM GMT +0000). [19:50:34] PROBLEM - www.baharna.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.baharna.org' expires in 15 day(s) (Sat 27 Jul 2024 07:29:48 PM GMT +0000). [19:50:41] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/71a81814bc91...f02a49a42894 [19:50:43] [02ssl] 07WikiTideSSLBot 03f02a49a - Bot: Update SSL cert for wiki.lefrenchmelee.fr [19:50:51] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/f02a49a42894...5eb3cf75fbc4 [19:50:54] [02ssl] 07WikiTideSSLBot 035eb3cf7 - Bot: Update SSL cert for www.baharna.org [19:51:10] PROBLEM - test151 APT on test151 is CRITICAL: APT CRITICAL: 80 packages available for upgrade (4 critical updates). [19:52:20] PROBLEM - mwtask181 APT on mwtask181 is CRITICAL: APT CRITICAL: 68 packages available for upgrade (4 critical updates). [19:53:50] PROBLEM - wiki.triplescripts.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.triplescripts.org' expires in 15 day(s) (Sat 27 Jul 2024 07:29:37 PM GMT +0000). [19:54:01] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5eb3cf75fbc4...96f75a2e57f2 [19:54:04] [02ssl] 07WikiTideSSLBot 0396f75a2 - Bot: Update SSL cert for wiki.triplescripts.org [19:55:43] PROBLEM - graphite151 APT on graphite151 is CRITICAL: APT CRITICAL: 45 packages available for upgrade (4 critical updates). [20:01:35] [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/96f75a2e57f2...dc1e3730520d [20:01:37] [02ssl] 07MacFan4000 03dc1e373 - add meijinversus.miraheze.org redirect [20:02:40] PROBLEM - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ricochetuniverse.com' expires in 15 day(s) (Sat 27 Jul 2024 07:45:28 PM GMT +0000). [20:02:52] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/dc1e3730520d...c05a41a109bd [20:02:54] [02ssl] 07WikiTideSSLBot 03c05a41a - Bot: Update SSL cert for wiki.ricochetuniverse.com [20:03:33] PROBLEM - wiki.warframestat.us - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.warframestat.us' expires in 15 day(s) (Sat 27 Jul 2024 07:36:40 PM GMT +0000). [20:03:45] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c05a41a109bd...df1abc6200eb [20:03:47] [02ssl] 07WikiTideSSLBot 03df1abc6 - Bot: Update SSL cert for wiki.warframestat.us [20:03:51] [02ssl] 07MacFan4000 reviewed pull request 03#788 commit - 13https://github.com/miraheze/ssl/pull/788#discussion_r1674625439 [20:05:26] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.31, 19.91, 18.31 [20:05:35] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/df1abc6200eb...9d9fedf99981 [20:05:37] [02ssl] 07WikiTideSSLBot 039d9fedf - Bot: Update SSL cert for thegreatwar.uk [20:06:43] PROBLEM - wiki.insideearth.info - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.insideearth.info' expires in 15 day(s) (Sat 27 Jul 2024 07:36:20 PM GMT +0000). [20:06:54] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9d9fedf99981...106accffd1e2 [20:06:55] [02ssl] 07WikiTideSSLBot 03106accf - Bot: Update SSL cert for wiki.insideearth.info [20:07:20] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 15.43, 18.37, 17.95 [20:09:58] PROBLEM - wiki.danvs.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.danvs.net' expires in 15 day(s) (Sat 27 Jul 2024 07:45:15 PM GMT +0000). [20:10:10] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/106accffd1e2...2ccd1f822323 [20:10:12] [02ssl] 07WikiTideSSLBot 032ccd1f8 - Bot: Update SSL cert for wiki.danvs.net [20:10:52] [02ssl] 07MacFan4000 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2ccd1f822323...dc11086b4180 [20:10:53] [02ssl] 07MacFan4000 03dc11086 - add urbanshade.miraheze.org redirect [20:15:01] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [20:19:08] RECOVERY - baharna.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.baharna.org' will expire on Wed 09 Oct 2024 06:50:45 PM GMT +0000. [20:19:08] RECOVERY - franchise.franchising.org.ua - LetsEncrypt on sslhost is OK: OK - Certificate 'franchise.franchising.org.ua' will expire on Wed 09 Oct 2024 06:49:43 PM GMT +0000. [20:19:28] RECOVERY - wiki.lefrenchmelee.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.lefrenchmelee.fr' will expire on Wed 09 Oct 2024 06:50:35 PM GMT +0000. [20:19:38] RECOVERY - www.baharna.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.baharna.org' will expire on Wed 09 Oct 2024 06:50:45 PM GMT +0000. [20:22:49] RECOVERY - www.thegreatwar.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thegreatwar.uk' will expire on Wed 09 Oct 2024 07:05:29 PM GMT +0000. [20:23:09] RECOVERY - wiki.triplescripts.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.triplescripts.org' will expire on Wed 09 Oct 2024 06:53:55 PM GMT +0000. [20:25:37] [02dns] 07MacFan4000 opened pull request 03#534: Add 3 zones - 13https://github.com/miraheze/dns/pull/534 [20:26:47] could somebody please merge https://github.com/miraheze/dns/pull/534 [20:27:18] [02dns] 07The-Voidwalker closed pull request 03#534: Add 3 zones - 13https://github.com/miraheze/dns/pull/534 [20:27:21] [02dns] 07The-Voidwalker pushed 031 commit to 03master [+3/-0/±0] 13https://github.com/miraheze/dns/compare/bd45398d25cd...be1b59a6e7e8 [20:27:23] [02dns] 07MacFan4000 03be1b59a - Add 3 zones (#534) [20:27:27] 👍 [20:27:58] PROBLEM - wiki.climatechange.ai - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.climatechange.ai' expires in 15 day(s) (Sat 27 Jul 2024 08:07:29 PM GMT +0000). [20:28:09] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/dc11086b4180...90ccbfdb4b9c [20:28:11] [02ssl] 07WikiTideSSLBot 0390ccbfd - Bot: Update SSL cert for wiki.climatechange.ai [20:28:57] RECOVERY - thegreatwar.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'thegreatwar.uk' will expire on Wed 09 Oct 2024 07:05:29 PM GMT +0000. [20:31:50] RECOVERY - wiki.ricochetuniverse.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ricochetuniverse.com' will expire on Wed 09 Oct 2024 07:02:45 PM GMT +0000. [20:31:51] PROBLEM - wiki.shaazzz.ir - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.shaazzz.ir' expires in 15 day(s) (Sat 27 Jul 2024 08:05:51 PM GMT +0000). [20:32:03] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/90ccbfdb4b9c...10c56e138d26 [20:32:04] [02ssl] 07WikiTideSSLBot 0310c56e1 - Bot: Update SSL cert for wiki.shaazzz.ir [20:32:07] RECOVERY - wiki.warframestat.us - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.warframestat.us' will expire on Wed 09 Oct 2024 07:03:39 PM GMT +0000. [20:32:42] PROBLEM - wiki.vinesh.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.vinesh.eu.org' expires in 15 day(s) (Sat 27 Jul 2024 08:28:24 PM GMT +0000). [20:32:56] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/10c56e138d26...34762c0bbb0a [20:32:58] [02ssl] 07WikiTideSSLBot 0334762c0 - Bot: Update SSL cert for wiki.vinesh.eu.org [20:33:27] PROBLEM - wiki.cdntennis.ca - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cdntennis.ca' expires in 15 day(s) (Sat 27 Jul 2024 08:13:06 PM GMT +0000). [20:33:38] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/34762c0bbb0a...b24ebe43db34 [20:33:41] [02ssl] 07WikiTideSSLBot 03b24ebe4 - Bot: Update SSL cert for wiki.cdntennis.ca [20:35:26] RECOVERY - wiki.insideearth.info - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.insideearth.info' will expire on Wed 09 Oct 2024 07:06:49 PM GMT +0000. [20:43:04] PROBLEM - n64brew.dev - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'n64brew.dev' expires in 15 day(s) (Sat 27 Jul 2024 08:16:03 PM GMT +0000). [20:43:15] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b24ebe43db34...3fb723a165b4 [20:43:17] [02ssl] 07WikiTideSSLBot 033fb723a - Bot: Update SSL cert for n64brew.dev [20:44:47] PROBLEM - hololive.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'hololive.wiki' expires in 15 day(s) (Sat 27 Jul 2024 08:25:47 PM GMT +0000). [20:44:59] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/3fb723a165b4...76d98408d570 [20:45:00] [02ssl] 07WikiTideSSLBot 0376d9840 - Bot: Update SSL cert for hololive.wiki [20:48:19] PROBLEM - aman.awiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'aman.awiki.org' expires in 15 day(s) (Sat 27 Jul 2024 08:30:19 PM GMT +0000). [20:48:31] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/76d98408d570...b8ef11111050 [20:48:32] [02ssl] 07WikiTideSSLBot 03b8ef111 - Bot: Update SSL cert for aman.awiki.org [20:48:56] PROBLEM - wiki.mobilityengineer.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mobilityengineer.com' expires in 15 day(s) (Sat 27 Jul 2024 08:26:46 PM GMT +0000). [20:49:23] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b8ef11111050...880d8e6fc876 [20:49:24] [02ssl] 07WikiTideSSLBot 03880d8e6 - Bot: Update SSL cert for wiki.mobilityengineer.com [20:57:02] RECOVERY - wiki.climatechange.ai - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.climatechange.ai' will expire on Wed 09 Oct 2024 07:28:03 PM GMT +0000. [21:00:35] RECOVERY - wiki.shaazzz.ir - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.shaazzz.ir' will expire on Wed 09 Oct 2024 07:31:57 PM GMT +0000. [21:02:15] RECOVERY - wiki.cdntennis.ca - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cdntennis.ca' will expire on Wed 09 Oct 2024 07:33:32 PM GMT +0000. [21:02:17] RECOVERY - wiki.vinesh.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.vinesh.eu.org' will expire on Wed 09 Oct 2024 07:32:50 PM GMT +0000. [21:03:18] PROBLEM - wiki.minkyu.kim - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.minkyu.kim' expires in 15 day(s) (Sat 27 Jul 2024 08:38:28 PM GMT +0000). [21:03:29] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/880d8e6fc876...bee1a0cdc89e [21:03:31] [02ssl] 07WikiTideSSLBot 03bee1a0c - Bot: Update SSL cert for wiki.minkyu.kim [21:08:49] RECOVERY - wiki.danvs.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.danvs.net' will expire on Wed 09 Oct 2024 07:10:04 PM GMT +0000. [21:12:11] RECOVERY - n64brew.dev - LetsEncrypt on sslhost is OK: OK - Certificate 'n64brew.dev' will expire on Wed 09 Oct 2024 07:43:10 PM GMT +0000. [21:14:39] RECOVERY - hololive.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'hololive.wiki' will expire on Wed 09 Oct 2024 07:44:53 PM GMT +0000. [21:14:41] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [21:16:41] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [38 system event log (SEL) entries present] [21:16:53] RECOVERY - aman.awiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'aman.awiki.org' will expire on Wed 09 Oct 2024 07:48:25 PM GMT +0000. [21:31:52] RECOVERY - wiki.minkyu.kim - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.minkyu.kim' will expire on Wed 09 Oct 2024 08:03:24 PM GMT +0000. [21:37:20] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [21:48:14] RECOVERY - wiki.mobilityengineer.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mobilityengineer.com' will expire on Wed 09 Oct 2024 07:49:02 PM GMT +0000. [21:49:30] PROBLEM - wiki.b3313.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.b3313.org' expires in 15 day(s) (Sat 27 Jul 2024 09:44:21 PM GMT +0000). [21:49:41] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bee1a0cdc89e...ed7338c0b770 [21:49:44] [02ssl] 07WikiTideSSLBot 03ed7338c - Bot: Update SSL cert for wiki.b3313.org [21:56:58] PROBLEM - archive.stellurgists.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'archive.stellurgists.wiki' expires in 15 day(s) (Sat 27 Jul 2024 09:51:40 PM GMT +0000). [21:57:10] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ed7338c0b770...4fab65f5ea7f [21:57:12] [02ssl] 07WikiTideSSLBot 034fab65f - Bot: Update SSL cert for archive.stellurgists.wiki [22:01:33] PROBLEM - www.bluepageswiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.bluepageswiki.org' expires in 15 day(s) (Sat 27 Jul 2024 09:48:17 PM GMT +0000). [22:01:45] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4fab65f5ea7f...06f756f2a51a [22:01:48] [02ssl] 07WikiTideSSLBot 0306f756f - Bot: Update SSL cert for www.bluepageswiki.org [22:04:55] PROBLEM - www.mcpk.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.mcpk.wiki' expires in 15 day(s) (Sat 27 Jul 2024 09:50:45 PM GMT +0000). [22:05:06] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/06f756f2a51a...ce3b35ec144e [22:05:09] [02ssl] 07WikiTideSSLBot 03ce3b35e - Bot: Update SSL cert for www.mcpk.wiki [22:09:23] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw181 [22:10:03] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw181 [22:10:50] PROBLEM - tep.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tep.wiki' expires in 15 day(s) (Sat 27 Jul 2024 10:06:54 PM GMT +0000). [22:11:02] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ce3b35ec144e...b8d6ab075360 [22:11:04] [02ssl] 07WikiTideSSLBot 03b8d6ab0 - Bot: Update SSL cert for tep.wiki [22:11:08] PROBLEM - wiki.openhatch.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.openhatch.org' expires in 15 day(s) (Sat 27 Jul 2024 10:00:39 PM GMT +0000). [22:11:19] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b8d6ab075360...b175afa05dce [22:11:22] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [22:11:22] [02ssl] 07WikiTideSSLBot 03b175afa - Bot: Update SSL cert for wiki.openhatch.org [22:12:01] PROBLEM - wiki.nowchess.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.nowchess.org' expires in 15 day(s) (Sat 27 Jul 2024 10:02:10 PM GMT +0000). [22:12:04] PROBLEM - data.nonbinary.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'data.nonbinary.wiki' expires in 15 day(s) (Sat 27 Jul 2024 10:04:02 PM GMT +0000). [22:12:04] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [22:12:13] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b175afa05dce...1fe785c9f1f7 [22:12:14] [02ssl] 07WikiTideSSLBot 031fe785c - Bot: Update SSL cert for wiki.nowchess.org [22:12:23] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1fe785c9f1f7...8eb016133800 [22:12:26] [02ssl] 07WikiTideSSLBot 038eb0161 - Bot: Update SSL cert for data.nonbinary.wiki [22:13:55] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.zymonic.com' expires in 15 day(s) (Sat 27 Jul 2024 09:59:27 PM GMT +0000). [22:14:06] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/8eb016133800...ba8071f2e140 [22:14:07] [02ssl] 07WikiTideSSLBot 03ba8071f - Bot: Update SSL cert for wiki.zymonic.com [22:15:37] PROBLEM - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mxlinuxusers.de' expires in 15 day(s) (Sat 27 Jul 2024 10:02:49 PM GMT +0000). [22:15:49] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ba8071f2e140...1ab5a37a2e22 [22:15:50] [02ssl] 07WikiTideSSLBot 031ab5a37 - Bot: Update SSL cert for wiki.mxlinuxusers.de [22:17:30] PROBLEM - wiki.counterculturelabs.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.counterculturelabs.org' expires in 15 day(s) (Sat 27 Jul 2024 09:47:05 PM GMT +0000). [22:17:39] PROBLEM - ie.unigon.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ie.unigon.net' expires in 15 day(s) (Sat 27 Jul 2024 10:14:51 PM GMT +0000). [22:17:41] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1ab5a37a2e22...da20ce953020 [22:17:43] [02ssl] 07WikiTideSSLBot 03da20ce9 - Bot: Update SSL cert for wiki.counterculturelabs.org [22:17:52] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/da20ce953020...900b547eca2d [22:17:53] [02ssl] 07WikiTideSSLBot 03900b547 - Bot: Update SSL cert for ie.unigon.net [22:18:57] PROBLEM - wiki.worlduniversityandschool.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.worlduniversityandschool.org' expires in 15 day(s) (Sat 27 Jul 2024 10:13:34 PM GMT +0000). [22:18:59] RECOVERY - wiki.b3313.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.b3313.org' will expire on Wed 09 Oct 2024 08:49:35 PM GMT +0000. [22:19:09] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/900b547eca2d...10ce2b14e612 [22:19:10] [02ssl] 07WikiTideSSLBot 0310ce2b1 - Bot: Update SSL cert for wiki.worlduniversityandschool.org [22:20:07] PROBLEM - wiki.villagecollaborative.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.villagecollaborative.net' expires in 15 day(s) (Sat 27 Jul 2024 09:53:28 PM GMT +0000). [22:20:19] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/10ce2b14e612...81f039d71059 [22:20:22] [02ssl] 07WikiTideSSLBot 0381f039d - Bot: Update SSL cert for wiki.villagecollaborative.net [22:20:30] PROBLEM - guia.cineastas.pt - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'guia.cineastas.pt' expires in 15 day(s) (Sat 27 Jul 2024 10:04:41 PM GMT +0000). [22:20:42] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/81f039d71059...c847ddcb2660 [22:20:45] [02ssl] 07WikiTideSSLBot 03c847ddc - Bot: Update SSL cert for guia.cineastas.pt [22:20:56] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [22:21:29] PROBLEM - wiki.kkstudio.eu.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.kkstudio.eu.org' expires in 15 day(s) (Sat 27 Jul 2024 10:15:55 PM GMT +0000). [22:21:43] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c847ddcb2660...fd0ca5d26f0e [22:21:45] [02ssl] 07WikiTideSSLBot 03fd0ca5d - Bot: Update SSL cert for wiki.kkstudio.eu.org [22:21:52] PROBLEM - cyberlaw.ccdcoe.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'cyberlaw.ccdcoe.org' expires in 15 day(s) (Sat 27 Jul 2024 10:07:47 PM GMT +0000). [22:22:04] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fd0ca5d26f0e...57b4a5422336 [22:22:05] [02ssl] 07WikiTideSSLBot 0357b4a54 - Bot: Update SSL cert for cyberlaw.ccdcoe.org [22:22:07] PROBLEM - wiki.valkyrienskies.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.valkyrienskies.org' expires in 15 day(s) (Sat 27 Jul 2024 09:57:04 PM GMT +0000). [22:22:19] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/57b4a5422336...e5fd09e934a7 [22:22:22] [02ssl] 07WikiTideSSLBot 03e5fd09e - Bot: Update SSL cert for wiki.valkyrienskies.org [22:22:56] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 40 system event log (SEL) entries present] [22:24:04] PROBLEM - wiki.macc.nyc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.macc.nyc' expires in 15 day(s) (Sat 27 Jul 2024 10:14:27 PM GMT +0000). [22:24:15] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e5fd09e934a7...ec01eacb8a7d [22:24:16] [02ssl] 07WikiTideSSLBot 03ec01eac - Bot: Update SSL cert for wiki.macc.nyc [22:24:33] PROBLEM - ao90.pinho.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ao90.pinho.org' expires in 15 day(s) (Sat 27 Jul 2024 10:06:05 PM GMT +0000). [22:24:44] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/ec01eacb8a7d...080b3ad3f2ae [22:24:46] [02ssl] 07WikiTideSSLBot 03080b3ad - Bot: Update SSL cert for ao90.pinho.org [22:25:08] PROBLEM - wiki.joust.ro - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.joust.ro' expires in 15 day(s) (Sat 27 Jul 2024 10:01:25 PM GMT +0000). [22:25:20] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/080b3ad3f2ae...a8519cf82a92 [22:25:22] [02ssl] 07WikiTideSSLBot 03a8519cf - Bot: Update SSL cert for wiki.joust.ro [22:25:53] RECOVERY - archive.stellurgists.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.stellurgists.wiki' will expire on Wed 09 Oct 2024 08:57:04 PM GMT +0000. [22:26:10] PROBLEM - wiki.grottocenter.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.grottocenter.org' expires in 15 day(s) (Sat 27 Jul 2024 10:12:50 PM GMT +0000). [22:26:23] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a8519cf82a92...e96b859036be [22:26:24] [02ssl] 07WikiTideSSLBot 03e96b859 - Bot: Update SSL cert for wiki.grottocenter.org [22:29:55] PROBLEM - wiki.graalmilitary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.graalmilitary.com' expires in 15 day(s) (Sat 27 Jul 2024 10:11:27 PM GMT +0000). [22:30:06] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e96b859036be...e34edec33b82 [22:30:09] [02ssl] 07WikiTideSSLBot 03e34edec - Bot: Update SSL cert for wiki.graalmilitary.com [22:30:34] RECOVERY - www.bluepageswiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'www.bluepageswiki.org' will expire on Wed 09 Oct 2024 09:01:39 PM GMT +0000. [22:31:23] PROBLEM - wiki.ciptamedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.ciptamedia.org' expires in 15 day(s) (Sat 27 Jul 2024 10:05:21 PM GMT +0000). [22:31:34] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e34edec33b82...96509435469c [22:31:35] [02ssl] 07WikiTideSSLBot 039650943 - Bot: Update SSL cert for wiki.ciptamedia.org [22:33:38] PROBLEM - wiki.s23.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.s23.org' expires in 15 day(s) (Sat 27 Jul 2024 10:08:59 PM GMT +0000). [22:33:49] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/96509435469c...604ba4674ad7 [22:33:51] [02ssl] 07WikiTideSSLBot 03604ba46 - Bot: Update SSL cert for wiki.s23.org [22:34:49] RECOVERY - www.mcpk.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'www.mcpk.wiki' will expire on Wed 09 Oct 2024 09:05:00 PM GMT +0000. [22:38:20] PROBLEM - wiki.mikrodev.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.mikrodev.com' expires in 15 day(s) (Sat 27 Jul 2024 10:10:35 PM GMT +0000). [22:38:30] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/604ba4674ad7...304d7336eca1 [22:38:33] [02ssl] 07WikiTideSSLBot 03304d733 - Bot: Update SSL cert for wiki.mikrodev.com [22:39:21] PROBLEM - wiki.scvo.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.scvo.org' expires in 15 day(s) (Sat 27 Jul 2024 10:09:55 PM GMT +0000). [22:39:33] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/304d7336eca1...a4117e58b7f7 [22:39:34] [02ssl] 07WikiTideSSLBot 03a4117e5 - Bot: Update SSL cert for wiki.scvo.org [22:47:20] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [22:50:26] RECOVERY - wiki.kkstudio.eu.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.kkstudio.eu.org' will expire on Wed 09 Oct 2024 09:21:37 PM GMT +0000. [22:51:14] RECOVERY - cyberlaw.ccdcoe.org - LetsEncrypt on sslhost is OK: OK - Certificate 'cyberlaw.ccdcoe.org' will expire on Wed 09 Oct 2024 09:21:58 PM GMT +0000. [22:52:03] RECOVERY - wiki.valkyrienskies.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.valkyrienskies.org' will expire on Wed 09 Oct 2024 09:22:13 PM GMT +0000. [22:52:37] RECOVERY - wiki.macc.nyc - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.macc.nyc' will expire on Wed 09 Oct 2024 09:24:09 PM GMT +0000. [22:54:03] RECOVERY - wiki.joust.ro - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.joust.ro' will expire on Wed 09 Oct 2024 09:25:14 PM GMT +0000. [22:54:13] RECOVERY - ao90.pinho.org - LetsEncrypt on sslhost is OK: OK - Certificate 'ao90.pinho.org' will expire on Wed 09 Oct 2024 09:24:38 PM GMT +0000. [22:54:40] RECOVERY - wiki.grottocenter.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.grottocenter.org' will expire on Wed 09 Oct 2024 09:26:15 PM GMT +0000. [22:59:50] RECOVERY - wiki.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.graalmilitary.com' will expire on Wed 09 Oct 2024 09:30:00 PM GMT +0000. [23:00:51] RECOVERY - wiki.ciptamedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.ciptamedia.org' will expire on Wed 09 Oct 2024 09:31:29 PM GMT +0000. [23:01:32] [02ssl] 07BlankEclair synchronize pull request 03#788: T12311: Redirect gimkit.miraheze.org to gimkit.wiki - 13https://github.com/miraheze/ssl/pull/788 [23:01:36] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#788: T12311: Redirect gimkit.miraheze.org to gimkit.wiki - 13https://github.com/miraheze/ssl/pull/788#issuecomment-2219879002 [23:03:22] RECOVERY - wiki.s23.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.s23.org' will expire on Wed 09 Oct 2024 09:33:43 PM GMT +0000. [23:03:39] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#788: T12311: Redirect gimkit.miraheze.org to gimkit.wiki - 13https://github.com/miraheze/ssl/pull/788#issuecomment-2219879002 [23:08:19] RECOVERY - wiki.mikrodev.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mikrodev.com' will expire on Wed 09 Oct 2024 09:38:25 PM GMT +0000. [23:08:35] RECOVERY - wiki.openhatch.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.openhatch.org' will expire on Wed 09 Oct 2024 09:11:14 PM GMT +0000. [23:08:58] RECOVERY - wiki.scvo.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.scvo.org' will expire on Wed 09 Oct 2024 09:39:26 PM GMT +0000. [23:10:11] RECOVERY - tep.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tep.wiki' will expire on Wed 09 Oct 2024 09:10:56 PM GMT +0000. [23:11:47] RECOVERY - wiki.nowchess.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nowchess.org' will expire on Wed 09 Oct 2024 09:12:07 PM GMT +0000. [23:11:55] RECOVERY - data.nonbinary.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'data.nonbinary.wiki' will expire on Wed 09 Oct 2024 09:12:18 PM GMT +0000. [23:13:27] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Wed 09 Oct 2024 09:14:00 PM GMT +0000. [23:13:41] RECOVERY - wiki.mxlinuxusers.de - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mxlinuxusers.de' will expire on Wed 09 Oct 2024 09:15:43 PM GMT +0000. [23:15:33] RECOVERY - ie.unigon.net - LetsEncrypt on sslhost is OK: OK - Certificate 'ie.unigon.net' will expire on Wed 09 Oct 2024 09:17:46 PM GMT +0000. [23:16:07] RECOVERY - wiki.counterculturelabs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.counterculturelabs.org' will expire on Wed 09 Oct 2024 09:17:36 PM GMT +0000. [23:16:57] RECOVERY - wiki.worlduniversityandschool.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.worlduniversityandschool.org' will expire on Wed 09 Oct 2024 09:19:02 PM GMT +0000. [23:17:09] RECOVERY - wiki.villagecollaborative.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.villagecollaborative.net' will expire on Wed 09 Oct 2024 09:20:12 PM GMT +0000. [23:18:36] RECOVERY - guia.cineastas.pt - LetsEncrypt on sslhost is OK: OK - Certificate 'guia.cineastas.pt' will expire on Wed 09 Oct 2024 09:20:36 PM GMT +0000. [23:49:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [23:54:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1