[00:00:09] PROBLEM - db181 Current Load on db181 is CRITICAL: LOAD CRITICAL - total load average: 19.81, 19.73, 8.72 [00:00:46] miraheze/python-functions - dependabot[bot] the build passed. [00:04:09] RECOVERY - db181 Current Load on db181 is OK: LOAD OK - total load average: 0.52, 8.94, 6.77 [00:21:32] [02mw-config] 07The-Voidwalker commented on pull request 03#5602: set $wgCdnServers when cloudflare not in use - 13https://github.com/miraheze/mw-config/pull/5602#issuecomment-2224175308 [00:27:15] PROBLEM - rothwell-leeds.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.rothwell-leeds.co.uk' expires in 15 day(s) (Sun 28 Jul 2024 12:18:44 AM GMT +0000). [00:31:23] PROBLEM - www.rothwell-leeds.co.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.rothwell-leeds.co.uk' expires in 15 day(s) (Sun 28 Jul 2024 12:18:44 AM GMT +0000). [00:31:37] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a4117e58b7f7...fb04c2c52acf [00:31:38] [02ssl] 07WikiTideSSLBot 03fb04c2c - Bot: Update SSL cert for www.rothwell-leeds.co.uk [00:35:43] PROBLEM - christipedia.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.christipedia.nl' expires in 15 day(s) (Sun 28 Jul 2024 12:21:09 AM GMT +0000). [00:35:45] PROBLEM - www.christipedia.nl - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.christipedia.nl' expires in 15 day(s) (Sun 28 Jul 2024 12:21:09 AM GMT +0000). [00:37:32] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [00:38:46] PROBLEM - wikien.wildterra2.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikien.wildterra2.com' expires in 15 day(s) (Sun 28 Jul 2024 12:33:22 AM GMT +0000). [00:38:57] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fb04c2c52acf...189f457fc691 [00:39:00] [02ssl] 07WikiTideSSLBot 03189f457 - Bot: Update SSL cert for wikien.wildterra2.com [00:39:33] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [43 system event log (SEL) entries present] [00:40:17] [02mw-config] 07The-Voidwalker closed pull request 03#5602: set $wgCdnServers when cloudflare not in use - 13https://github.com/miraheze/mw-config/pull/5602 [00:40:19] [02mw-config] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±3] 13https://github.com/miraheze/mw-config/compare/f9a26469b0fb...6a3affc67672 [00:40:21] [02mw-config] 07The-Voidwalker 036a3affc - set $wgCdnServers when cloudflare not in use (#5602) [00:40:23] [02mw-config] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 [00:40:24] [02mw-config] 07The-Voidwalker deleted branch 03The-Voidwalker-patch-1 - 13https://github.com/miraheze/mw-config [00:41:13] miraheze/mw-config - The-Voidwalker the build passed. [00:41:51] !log [@mwtask181] starting deploy of {'config': True} to all [00:42:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:42:05] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 13s [00:42:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:54:17] !log [@test151] starting deploy of {'config': True} to test151 [00:54:18] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [00:54:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:54:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:56:49] RECOVERY - rothwell-leeds.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rothwell-leeds.co.uk' will expire on Wed 09 Oct 2024 11:31:31 PM GMT +0000. [01:00:14] RECOVERY - www.rothwell-leeds.co.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.rothwell-leeds.co.uk' will expire on Wed 09 Oct 2024 11:31:31 PM GMT +0000. [01:01:41] PROBLEM - heavyironmodding.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'heavyironmodding.org' expires in 15 day(s) (Sun 28 Jul 2024 12:58:32 AM GMT +0000). [01:01:57] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/189f457fc691...28d15c96578a [01:01:58] [02ssl] 07WikiTideSSLBot 0328d15c9 - Bot: Update SSL cert for heavyironmodding.org [01:03:56] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.25, 19.27, 17.39 [01:05:54] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 20.15, 19.33, 17.64 [01:07:28] !log [@mwtask171] starting deploy of {'config': True} to all [01:07:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:07:38] PROBLEM - trollpasta.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'trollpasta.com' expires in 15 day(s) (Sun 28 Jul 2024 12:52:16 AM GMT +0000). [01:07:42] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 14s [01:07:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:08:12] RECOVERY - wikien.wildterra2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikien.wildterra2.com' will expire on Wed 09 Oct 2024 11:38:52 PM GMT +0000. [01:09:07] PROBLEM - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.hoolehistoryheritagesociety.org.uk' expires in 15 day(s) (Sun 28 Jul 2024 12:38:17 AM GMT +0000). [01:09:19] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/28d15c96578a...69445f92f604 [01:09:21] [02ssl] 07WikiTideSSLBot 0369445f9 - Bot: Update SSL cert for www.hoolehistoryheritagesociety.org.uk [01:10:19] PROBLEM - repository.archiopedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'repository.archiopedia.org' expires in 15 day(s) (Sun 28 Jul 2024 12:58:52 AM GMT +0000). [01:10:30] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/69445f92f604...d8791fb1487f [01:10:31] [02ssl] 07WikiTideSSLBot 03d8791fb - Bot: Update SSL cert for repository.archiopedia.org [01:13:10] PROBLEM - www.trollpasta.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'trollpasta.com' expires in 15 day(s) (Sun 28 Jul 2024 12:52:16 AM GMT +0000). [01:15:48] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.99, 19.19, 17.70 [01:17:47] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 15.73, 17.99, 17.46 [01:18:10] PROBLEM - wiki.cyberfurs.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.cyberfurs.org' expires in 15 day(s) (Sun 28 Jul 2024 01:08:23 AM GMT +0000). [01:18:22] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d8791fb1487f...02b1547144ad [01:18:24] [02ssl] 07WikiTideSSLBot 0302b1547 - Bot: Update SSL cert for wiki.cyberfurs.org [01:21:36] PROBLEM - wiki.funkey-project.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.funkey-project.com' expires in 15 day(s) (Sun 28 Jul 2024 12:56:52 AM GMT +0000). [01:21:47] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/02b1547144ad...a2891f1079a4 [01:21:49] [02ssl] 07WikiTideSSLBot 03a2891f1 - Bot: Update SSL cert for wiki.funkey-project.com [01:23:42] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.53, 22.14, 19.43 [01:25:41] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.31, 21.21, 19.44 [01:27:40] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.19, 21.35, 19.65 [01:30:10] RECOVERY - heavyironmodding.org - LetsEncrypt on sslhost is OK: OK - Certificate 'heavyironmodding.org' will expire on Thu 10 Oct 2024 12:01:50 AM GMT +0000. [01:31:38] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 18.56, 21.08, 19.98 [01:33:36] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.27, 23.00, 20.85 [01:35:35] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.55, 21.46, 20.52 [01:39:46] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [01:41:32] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 17.21, 19.19, 19.91 [01:41:47] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [45 system event log (SEL) entries present] [01:44:21] PROBLEM - wiki.astralprojections.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.astralprojections.org' expires in 15 day(s) (Sun 28 Jul 2024 01:28:18 AM GMT +0000). [01:44:33] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a2891f1079a4...4c34fa94a20a [01:44:35] [02ssl] 07WikiTideSSLBot 034c34fa9 - Bot: Update SSL cert for wiki.astralprojections.org [01:47:27] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.78, 22.34, 21.06 [01:50:25] RECOVERY - wiki.funkey-project.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.funkey-project.com' will expire on Thu 10 Oct 2024 12:21:42 AM GMT +0000. [01:53:25] PROBLEM - civwiki.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'civwiki.org' expires in 15 day(s) (Sun 28 Jul 2024 01:37:16 AM GMT +0000). [01:53:36] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4c34fa94a20a...717d57d5b79e [01:53:39] [02ssl] 07WikiTideSSLBot 03717d57d - Bot: Update SSL cert for civwiki.org [01:56:25] PROBLEM - www.fossilfighterswiki.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.fossilfighterswiki.com' expires in 15 day(s) (Sun 28 Jul 2024 01:53:35 AM GMT +0000). [01:56:36] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/717d57d5b79e...fae4e31ab914 [01:56:38] [02ssl] 07WikiTideSSLBot 03fae4e31 - Bot: Update SSL cert for www.fossilfighterswiki.com [01:57:22] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 15.02, 18.91, 20.18 [02:02:14] PROBLEM - celeste.ink - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'celeste.ink' expires in 15 day(s) (Sun 28 Jul 2024 01:45:11 AM GMT +0000). [02:02:24] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fae4e31ab914...9df7e7bbcc50 [02:02:25] [02ssl] 07WikiTideSSLBot 039df7e7b - Bot: Update SSL cert for celeste.ink [02:08:39] RECOVERY - www.hoolehistoryheritagesociety.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'www.hoolehistoryheritagesociety.org.uk' will expire on Thu 10 Oct 2024 12:09:13 AM GMT +0000. [02:08:55] RECOVERY - repository.archiopedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'repository.archiopedia.org' will expire on Thu 10 Oct 2024 12:10:24 AM GMT +0000. [02:11:12] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 24.95, 21.53, 20.47 [02:13:11] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 22.61, 21.50, 20.57 [02:13:12] RECOVERY - wiki.astralprojections.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.astralprojections.org' will expire on Thu 10 Oct 2024 12:44:27 AM GMT +0000. [02:17:12] RECOVERY - wiki.cyberfurs.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.cyberfurs.org' will expire on Thu 10 Oct 2024 12:18:16 AM GMT +0000. [02:22:12] PROBLEM - ecole.science - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ecole.science' expires in 15 day(s) (Sun 28 Jul 2024 01:56:10 AM GMT +0000). [02:22:21] RECOVERY - civwiki.org - LetsEncrypt on sslhost is OK: OK - Certificate 'civwiki.org' will expire on Thu 10 Oct 2024 12:53:31 AM GMT +0000. [02:22:24] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/9df7e7bbcc50...aacd137e54f9 [02:22:27] [02ssl] 07WikiTideSSLBot 03aacd137 - Bot: Update SSL cert for ecole.science [02:23:05] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 28.78, 23.35, 21.42 [02:25:04] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.83, 21.68, 21.04 [02:26:14] RECOVERY - www.fossilfighterswiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'www.fossilfighterswiki.com' will expire on Thu 10 Oct 2024 12:56:31 AM GMT +0000. [02:30:57] RECOVERY - celeste.ink - LetsEncrypt on sslhost is OK: OK - Certificate 'celeste.ink' will expire on Thu 10 Oct 2024 01:02:19 AM GMT +0000. [02:35:51] PROBLEM - wiki.meregos.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.meregos.com' expires in 15 day(s) (Sun 28 Jul 2024 02:05:25 AM GMT +0000). [02:36:03] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/aacd137e54f9...6a6ac888f681 [02:36:05] [02ssl] 07WikiTideSSLBot 036a6ac88 - Bot: Update SSL cert for wiki.meregos.com [02:38:56] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 16.97, 19.11, 20.10 [02:39:12] PROBLEM - www.winenjoy.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.winenjoy.net' expires in 15 day(s) (Sun 28 Jul 2024 02:25:11 AM GMT +0000). [02:39:22] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6a6ac888f681...b1a17d68ec0d [02:39:25] [02ssl] 07WikiTideSSLBot 03b1a17d6 - Bot: Update SSL cert for www.winenjoy.net [02:41:02] PROBLEM - wiki.yumeka.icu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.yumeka.icu' expires in 15 day(s) (Sun 28 Jul 2024 02:25:00 AM GMT +0000). [02:41:13] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b1a17d68ec0d...bbabd6a9db5a [02:41:15] [02ssl] 07WikiTideSSLBot 03bbabd6a - Bot: Update SSL cert for wiki.yumeka.icu [02:42:02] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [02:42:53] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.29, 20.58, 20.48 [02:44:02] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [46 system event log (SEL) entries present] [02:44:51] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.91, 21.50, 20.78 [02:46:34] PROBLEM - wiki.digitaldesignhq.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.digitaldesignhq.com' expires in 15 day(s) (Sun 28 Jul 2024 02:34:31 AM GMT +0000). [02:46:49] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/bbabd6a9db5a...1abe07dd2eb5 [02:46:50] [02ssl] 07WikiTideSSLBot 031abe07d - Bot: Update SSL cert for wiki.digitaldesignhq.com [02:46:50] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.82, 21.04, 20.68 [02:47:58] PROBLEM - wiki.seredrau.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.seredrau.xyz' expires in 15 day(s) (Sun 28 Jul 2024 02:17:32 AM GMT +0000). [02:48:10] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1abe07dd2eb5...d3f9141f7bac [02:48:12] [02ssl] 07WikiTideSSLBot 03d3f9141 - Bot: Update SSL cert for wiki.seredrau.xyz [02:48:49] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.20, 22.48, 21.20 [02:50:48] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 18.12, 21.34, 20.99 [02:51:00] RECOVERY - ecole.science - LetsEncrypt on sslhost is OK: OK - Certificate 'ecole.science' will expire on Thu 10 Oct 2024 01:22:18 AM GMT +0000. [02:52:47] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 12.45, 18.18, 19.87 [02:53:17] PROBLEM - wiki.aridia.space - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.aridia.space' expires in 15 day(s) (Sun 28 Jul 2024 02:47:35 AM GMT +0000). [02:53:27] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d3f9141f7bac...1ed4095d0891 [02:53:28] [02ssl] 07WikiTideSSLBot 031ed4095 - Bot: Update SSL cert for wiki.aridia.space [02:59:48] PROBLEM - wmworld.sktz.live - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wmworld.sktz.live' expires in 15 day(s) (Sun 28 Jul 2024 02:53:23 AM GMT +0000). [03:00:02] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/1ed4095d0891...04372687d3a3 [03:00:04] [02ssl] 07WikiTideSSLBot 030437268 - Bot: Update SSL cert for wmworld.sktz.live [03:00:15] RECOVERY - db151 Backups SQL on db151 is OK: FILE_AGE OK: /var/log/sql-backup.log is 14 seconds old and 0 bytes [03:04:33] PROBLEM - equestripedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'equestripedia.org' expires in 15 day(s) (Sun 28 Jul 2024 02:52:23 AM GMT +0000). [03:04:44] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/04372687d3a3...5a2a3684468c [03:04:46] [02ssl] 07WikiTideSSLBot 035a2a368 - Bot: Update SSL cert for equestripedia.org [03:05:13] RECOVERY - wiki.meregos.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.meregos.com' will expire on Thu 10 Oct 2024 01:35:57 AM GMT +0000. [03:06:24] PROBLEM - sarovia.graalmilitary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'sarovia.graalmilitary.com' expires in 15 day(s) (Sun 28 Jul 2024 03:02:35 AM GMT +0000). [03:06:35] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5a2a3684468c...e23e7dd8a45b [03:06:37] [02ssl] 07WikiTideSSLBot 03e23e7dd - Bot: Update SSL cert for sarovia.graalmilitary.com [03:08:11] PROBLEM - archive.a2b2.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'archive.a2b2.org' expires in 15 day(s) (Sun 28 Jul 2024 02:37:10 AM GMT +0000). [03:08:22] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e23e7dd8a45b...b099e57df4bb [03:08:23] [02ssl] 07WikiTideSSLBot 03b099e57 - Bot: Update SSL cert for archive.a2b2.org [03:09:48] PROBLEM - wiki.creaturathegame.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.creaturathegame.com' expires in 15 day(s) (Sun 28 Jul 2024 02:53:12 AM GMT +0000). [03:10:00] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b099e57df4bb...4720d2f7886d [03:10:01] [02ssl] 07WikiTideSSLBot 034720d2f - Bot: Update SSL cert for wiki.creaturathegame.com [03:12:35] PROBLEM - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.bikinibottomdiscord.org' expires in 15 day(s) (Sun 28 Jul 2024 02:56:26 AM GMT +0000). [03:12:46] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4720d2f7886d...d84fe43fadac [03:12:48] [02ssl] 07WikiTideSSLBot 03d84fe43 - Bot: Update SSL cert for wiki.bikinibottomdiscord.org [03:18:12] PROBLEM - wikiru.wildterra2.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wikiru.wildterra2.com' expires in 15 day(s) (Sun 28 Jul 2024 03:13:52 AM GMT +0000). [03:18:24] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d84fe43fadac...0bbda926ae66 [03:18:27] [02ssl] 07WikiTideSSLBot 030bbda92 - Bot: Update SSL cert for wikiru.wildterra2.com [03:20:00] PROBLEM - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.qadrishattari.xyz' expires in 15 day(s) (Sun 28 Jul 2024 02:52:54 AM GMT +0000). [03:20:12] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0bbda926ae66...b8c8d2a5b3a0 [03:20:13] [02ssl] 07WikiTideSSLBot 03b8c8d2a - Bot: Update SSL cert for wiki.qadrishattari.xyz [03:22:03] RECOVERY - wiki.aridia.space - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.aridia.space' will expire on Thu 10 Oct 2024 01:53:22 AM GMT +0000. [03:29:06] RECOVERY - wmworld.sktz.live - LetsEncrypt on sslhost is OK: OK - Certificate 'wmworld.sktz.live' will expire on Thu 10 Oct 2024 01:59:54 AM GMT +0000. [03:34:05] RECOVERY - equestripedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'equestripedia.org' will expire on Thu 10 Oct 2024 02:04:38 AM GMT +0000. [03:36:03] RECOVERY - sarovia.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'sarovia.graalmilitary.com' will expire on Thu 10 Oct 2024 02:06:30 AM GMT +0000. [03:36:52] RECOVERY - archive.a2b2.org - LetsEncrypt on sslhost is OK: OK - Certificate 'archive.a2b2.org' will expire on Thu 10 Oct 2024 02:08:17 AM GMT +0000. [03:37:53] RECOVERY - www.winenjoy.net - LetsEncrypt on sslhost is OK: OK - Certificate 'www.winenjoy.net' will expire on Thu 10 Oct 2024 01:39:17 AM GMT +0000. [03:39:00] RECOVERY - wiki.creaturathegame.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.creaturathegame.com' will expire on Thu 10 Oct 2024 02:09:54 AM GMT +0000. [03:40:38] RECOVERY - wiki.yumeka.icu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.yumeka.icu' will expire on Thu 10 Oct 2024 01:41:08 AM GMT +0000. [03:41:09] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw172 [03:41:10] RECOVERY - wiki.bikinibottomdiscord.org - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.bikinibottomdiscord.org' will expire on Thu 10 Oct 2024 02:12:41 AM GMT +0000. [03:43:08] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [03:45:41] RECOVERY - wiki.digitaldesignhq.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.digitaldesignhq.com' will expire on Thu 10 Oct 2024 01:46:40 AM GMT +0000. [03:46:03] PROBLEM - pt.graalmilitary.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pt.graalmilitary.com' expires in 15 day(s) (Sun 28 Jul 2024 03:22:12 AM GMT +0000). [03:46:16] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b8c8d2a5b3a0...2b8fbc979c5d [03:46:18] [02ssl] 07WikiTideSSLBot 032b8fbc9 - Bot: Update SSL cert for pt.graalmilitary.com [03:46:55] RECOVERY - wiki.seredrau.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.seredrau.xyz' will expire on Thu 10 Oct 2024 01:48:04 AM GMT +0000. [03:47:56] PROBLEM - wiki.nevillepedia.eu - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.nevillepedia.eu' expires in 15 day(s) (Sun 28 Jul 2024 03:43:14 AM GMT +0000). [03:48:01] RECOVERY - wikiru.wildterra2.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wikiru.wildterra2.com' will expire on Thu 10 Oct 2024 02:18:18 AM GMT +0000. [03:48:07] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2b8fbc979c5d...6478a7beffb2 [03:48:08] [02ssl] 07WikiTideSSLBot 036478a7b - Bot: Update SSL cert for wiki.nevillepedia.eu [03:48:18] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [03:50:18] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [47 system event log (SEL) entries present] [03:50:45] PROBLEM - wiki.globasa.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.globasa.net' expires in 15 day(s) (Sun 28 Jul 2024 03:38:42 AM GMT +0000). [03:50:56] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/6478a7beffb2...26e328c9340c [03:50:59] [02ssl] 07WikiTideSSLBot 0326e328c - Bot: Update SSL cert for wiki.globasa.net [03:54:30] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:05:54] PROBLEM - you.r-fit.cc - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'you.r-fit.cc' expires in 15 day(s) (Sun 28 Jul 2024 03:44:54 AM GMT +0000). [04:06:07] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/26e328c9340c...804042de7c78 [04:06:09] [02ssl] 07WikiTideSSLBot 03804042d - Bot: Update SSL cert for you.r-fit.cc [04:15:05] RECOVERY - pt.graalmilitary.com - LetsEncrypt on sslhost is OK: OK - Certificate 'pt.graalmilitary.com' will expire on Thu 10 Oct 2024 02:46:09 AM GMT +0000. [04:17:47] RECOVERY - wiki.nevillepedia.eu - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.nevillepedia.eu' will expire on Thu 10 Oct 2024 02:48:02 AM GMT +0000. [04:18:01] RECOVERY - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.qadrishattari.xyz' will expire on Thu 10 Oct 2024 02:20:07 AM GMT +0000. [04:19:34] RECOVERY - wiki.globasa.net - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.globasa.net' will expire on Thu 10 Oct 2024 02:50:50 AM GMT +0000. [04:34:37] RECOVERY - you.r-fit.cc - LetsEncrypt on sslhost is OK: OK - Certificate 'you.r-fit.cc' will expire on Thu 10 Oct 2024 03:06:00 AM GMT +0000. [04:46:45] [02ssl] 07BlankEclair synchronize pull request 03#788: T12311: Redirect gimkit.miraheze.org to gimkit.wiki - 13https://github.com/miraheze/ssl/pull/788 [04:46:50] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#788: T12311: Redirect gimkit.miraheze.org to gimkit.wiki - 13https://github.com/miraheze/ssl/pull/788#issuecomment-2219879002 [04:49:07] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw152 [04:49:23] [02ssl] 07coderabbitai[bot] edited a comment on pull request 03#788: T12311: Redirect gimkit.miraheze.org to gimkit.wiki - 13https://github.com/miraheze/ssl/pull/788#issuecomment-2219879002 [04:50:44] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [04:51:06] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [04:52:45] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [47 system event log (SEL) entries present] [05:14:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:24:23] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.53, 20.44, 19.05 [05:30:19] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.19, 19.97, 19.32 [05:54:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:01:59] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 23.25, 21.03, 19.61 [06:05:57] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 25.53, 22.38, 20.40 [06:07:56] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.91, 22.37, 20.66 [06:15:51] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 26.41, 24.50, 22.25 [06:19:49] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.35, 23.43, 22.43 [06:21:48] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 30.27, 25.28, 23.18 [06:24:30] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:29:43] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 21.56, 23.87, 23.44 [06:31:42] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 22.65, 24.15, 23.63 [06:33:41] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 19.89, 22.83, 23.22 [06:35:40] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 27.83, 24.07, 23.58 [06:37:39] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 20.91, 23.08, 23.31 [06:43:36] PROBLEM - mwtask181 Current Load on mwtask181 is CRITICAL: LOAD CRITICAL - total load average: 30.62, 25.53, 24.10 [06:59:27] PROBLEM - mwtask181 Current Load on mwtask181 is WARNING: LOAD WARNING - total load average: 18.44, 21.50, 23.45 [07:01:09] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.90, 19.87, 15.36 [07:01:43] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.11, 19.65, 14.41 [07:03:33] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw152 mw172 [07:03:34] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw172 [07:03:35] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 2 backends are down. mw152 mw172 [07:03:37] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw152 mw172 [07:03:39] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 2 backends are down. mw152 mw172 [07:03:43] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.87, 18.64, 14.71 [07:03:43] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 2 backends are down. mw152 mw172 [07:05:49] PROBLEM - mw161 HTTPS on mw161 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 502 [07:07:18] PROBLEM - mw161 MediaWiki Rendering on mw161 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:08:46] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 24.04, 19.59, 13.28 [07:10:46] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.55, 18.74, 13.75 [07:12:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:13:09] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.82, 23.42, 20.82 [07:13:30] RECOVERY - mw161 MediaWiki Rendering on mw161 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 4.794 second response time [07:13:49] RECOVERY - mw161 HTTPS on mw161 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 286 bytes in 0.067 second response time [07:15:06] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [07:15:31] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [07:15:38] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [07:15:53] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [07:17:08] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [07:17:09] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.27, 19.03, 19.61 [07:17:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:17:28] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [07:20:48] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [07:21:14] RECOVERY - mwtask181 Current Load on mwtask181 is OK: LOAD OK - total load average: 19.27, 19.03, 20.20 [07:27:31] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw162 [07:29:26] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [07:41:22] [02CreateWiki] 07waki285 opened pull request 03#542: Make Special:RequestWikiQueue respect the timezone set by the user - 13https://github.com/miraheze/CreateWiki/pull/542 [07:41:25] [02CreateWiki] 07coderabbitai[bot] commented on pull request 03#542: Make Special:RequestWikiQueue respect the timezone set by the user - 13https://github.com/miraheze/CreateWiki/pull/542#issuecomment-2225010036 [07:41:52] [02CreateWiki] 07waki285 commented on pull request 03#542: Make Special:RequestWikiQueue respect the timezone set by the user - 13https://github.com/miraheze/CreateWiki/pull/542#issuecomment-2225010655 [07:41:57] [02CreateWiki] 07coderabbitai[bot] commented on pull request 03#542: Make Special:RequestWikiQueue respect the timezone set by the user - 13https://github.com/miraheze/CreateWiki/pull/542#issuecomment-2225010789 [07:41:59] [02CreateWiki] 07coderabbitai[bot] edited a comment on pull request 03#542: Make Special:RequestWikiQueue respect the timezone set by the user - 13https://github.com/miraheze/CreateWiki/pull/542#issuecomment-2225010036 [07:42:48] [02CreateWiki] 07coderabbitai[bot] edited pull request 03#542: Make Special:RequestWikiQueue respect the timezone set by the user - 13https://github.com/miraheze/CreateWiki/pull/542 [07:43:00] [02CreateWiki] 07coderabbitai[bot] edited a comment on pull request 03#542: Make Special:RequestWikiQueue respect the timezone set by the user - 13https://github.com/miraheze/CreateWiki/pull/542#issuecomment-2225010036 [07:45:14] PROBLEM - ns2 Auth DNS on ns2 is CRITICAL: No response from DNS 2001:41d0:801:2000::4089 [07:45:56] PROBLEM - ping6 on cp41 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 174.58 ms [07:46:07] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [07:46:09] PROBLEM - ping6 on ns2 is CRITICAL: PING CRITICAL - Packet loss = 90%, RTA = 212.89 ms [07:46:16] PROBLEM - creativecommons.id - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:46:22] PROBLEM - cp51 HTTPS on cp51 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: cURL returned 28 - Connection timed out after 10004 milliseconds [07:46:22] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 7 backends are down. mw151 mw152 mw161 mw162 mw171 mw181 mw182 [07:46:22] PROBLEM - ping6 on cp51 is CRITICAL: PING CRITICAL - Packet loss = 80%, RTA = 210.83 ms [07:46:25] PROBLEM - ns1 GDNSD Datacenters on ns1 is CRITICAL: CRITICAL - 3 datacenters are down: 46.250.240.167/cpweb, 2407:3641:2161:9774::1/cpweb, 2400:d320:2161:9775::1/cpweb [07:47:15] RECOVERY - ns2 Auth DNS on ns2 is OK: DNS OK: 3.478 seconds response time. wikitide.net returns 2407:3641:2161:9774::1,46.250.240.167 [07:47:17] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:47:34] PROBLEM - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is CRITICAL: CRITICAL - NGINX Error Rate is 77% [07:47:38] PROBLEM - ns2 GDNSD Datacenters on ns2 is CRITICAL: CRITICAL - 4 datacenters are down: 38.46.223.205/cpweb, 38.46.223.206/cpweb, 2602:294:0:b13::110/cpweb, 2602:294:0:b23::112/cpweb [07:48:06] PROBLEM - mw171 Puppet on mw171 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[git_pull_3d2png] [07:48:22] RECOVERY - ns1 GDNSD Datacenters on ns1 is OK: OK - all datacenters are online [07:50:56] PROBLEM - ns2 APT on ns2 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [07:51:00] PROBLEM - polandballwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Socket timeout after 10 seconds [07:52:03] RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3784 bytes in 0.851 second response time [07:52:09] RECOVERY - ping6 on cp41 is OK: PING OK - Packet loss = 0%, RTA = 102.35 ms [07:52:19] RECOVERY - ping6 on ns2 is OK: PING OK - Packet loss = 0%, RTA = 142.56 ms [07:52:22] RECOVERY - cp51 HTTPS on cp51 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3783 bytes in 1.046 second response time [07:52:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:52:26] RECOVERY - cp51 HTTP 4xx/5xx ERROR Rate on cp51 is OK: OK - NGINX Error Rate is 11% [07:52:32] RECOVERY - ns2 GDNSD Datacenters on ns2 is OK: OK - all datacenters are online [07:52:34] RECOVERY - ping6 on cp51 is OK: PING OK - Packet loss = 0%, RTA = 162.13 ms [07:52:55] RECOVERY - ns2 APT on ns2 is OK: APT OK: 27 packages available for upgrade (0 critical updates). [07:53:46] miraheze/CreateWiki - waki285 the build passed. [07:54:00] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [07:55:09] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [07:55:22] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [07:57:22] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [48 system event log (SEL) entries present] [08:13:35] PROBLEM - cp41 HTTPS on cp41 is CRITICAL: HTTP CRITICAL - Invalid HTTP response received from host on port 443: HTTP/2 503 [08:15:01] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw161 [08:15:01] RECOVERY - creativecommons.id - LetsEncrypt on sslhost is OK: OK - Certificate 'creativecommons.id' will expire on Mon 29 Jul 2024 07:48:07 PM GMT +0000. [08:15:04] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw161 [08:15:04] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [08:15:30] RECOVERY - cp41 HTTPS on cp41 is OK: HTTP OK: HTTP/2 404 - Status line output matched "HTTP/2 404" - 3784 bytes in 0.692 second response time [08:16:14] RECOVERY - mw171 Puppet on mw171 is OK: OK: Puppet is currently enabled, last run 59 seconds ago with 0 failures [08:16:58] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [08:17:00] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [08:20:02] RECOVERY - polandballwiki.com - LetsEncrypt on sslhost is OK: OK - Certificate 'polandballwiki.com' will expire on Wed 25 Sep 2024 02:24:04 AM GMT +0000. [08:20:38] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw152 [08:22:33] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [08:25:57] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw181 [08:26:51] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw181 mw182 [08:27:56] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [08:28:19] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw171 [08:30:14] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [08:34:46] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [08:41:28] PROBLEM - correypedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'correypedia.org' expires in 15 day(s) (Sun 28 Jul 2024 08:29:30 AM GMT +0000). [08:41:41] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/804042de7c78...48fa5cfca540 [08:41:44] [02ssl] 07WikiTideSSLBot 0348fa5cf - Bot: Update SSL cert for correypedia.org [08:43:27] PROBLEM - wiki.wikimedia.cat - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.wikimedia.cat' expires in 15 day(s) (Sun 28 Jul 2024 08:30:13 AM GMT +0000). [08:43:42] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/48fa5cfca540...c846f2c6595b [08:43:44] [02ssl] 07WikiTideSSLBot 03c846f2c - Bot: Update SSL cert for wiki.wikimedia.cat [08:48:09] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 1 backends are down. mw182 [08:48:41] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw171 [08:50:04] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [08:51:13] PROBLEM - wiki.apico.buzz - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.apico.buzz' expires in 15 day(s) (Sun 28 Jul 2024 08:27:24 AM GMT +0000). [08:51:25] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/c846f2c6595b...14620f80e339 [08:51:28] [02ssl] 07WikiTideSSLBot 0314620f8 - Bot: Update SSL cert for wiki.apico.buzz [08:52:41] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [08:55:55] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 2 backends are down. mw161 mw172 [08:56:41] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 3 backends are down. mw161 mw172 mw182 [08:57:09] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.65, 19.07, 14.30 [08:57:55] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [08:58:41] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [08:59:09] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 11.97, 16.17, 13.84 [09:01:34] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [09:03:35] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [48 system event log (SEL) entries present] [09:08:34] PROBLEM - reviews.thejoshmeister.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'reviews.thejoshmeister.com' expires in 15 day(s) (Sun 28 Jul 2024 09:00:36 AM GMT +0000). [09:08:46] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/14620f80e339...31e0cb3d38b8 [09:08:47] [02ssl] 07WikiTideSSLBot 0331e0cb3 - Bot: Update SSL cert for reviews.thejoshmeister.com [09:11:15] RECOVERY - correypedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'correypedia.org' will expire on Thu 10 Oct 2024 07:41:34 AM GMT +0000. [09:12:54] RECOVERY - wiki.wikimedia.cat - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.wikimedia.cat' will expire on Thu 10 Oct 2024 07:43:36 AM GMT +0000. [09:17:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:20:08] PROBLEM - wiki.consid.vn - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.consid.vn' expires in 15 day(s) (Sun 28 Jul 2024 09:15:19 AM GMT +0000). [09:20:23] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/31e0cb3d38b8...e028907b87b1 [09:20:24] [02ssl] 07WikiTideSSLBot 03e028907 - Bot: Update SSL cert for wiki.consid.vn [09:20:50] RECOVERY - wiki.apico.buzz - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.apico.buzz' will expire on Thu 10 Oct 2024 07:51:19 AM GMT +0000. [09:24:39] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw161 [09:24:42] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw161 [09:24:44] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw161 [09:26:33] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [09:26:36] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [09:26:40] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [09:28:43] PROBLEM - mabi.myomi.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'mabi.myomi.net' expires in 15 day(s) (Sun 28 Jul 2024 09:16:48 AM GMT +0000). [09:28:54] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/e028907b87b1...a0973e16f781 [09:28:57] [02ssl] 07WikiTideSSLBot 03a0973e1 - Bot: Update SSL cert for mabi.myomi.net [09:34:00] PROBLEM - wiki.eversd.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.eversd.com' expires in 15 day(s) (Sun 28 Jul 2024 09:17:16 AM GMT +0000). [09:34:12] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a0973e16f781...a514f62a4bb2 [09:34:13] [02ssl] 07WikiTideSSLBot 03a514f62 - Bot: Update SSL cert for wiki.eversd.com [09:37:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:37:38] RECOVERY - reviews.thejoshmeister.com - LetsEncrypt on sslhost is OK: OK - Certificate 'reviews.thejoshmeister.com' will expire on Thu 10 Oct 2024 08:08:40 AM GMT +0000. [09:45:48] PROBLEM - robloxapi.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'robloxapi.wiki' expires in 15 day(s) (Sun 28 Jul 2024 09:32:48 AM GMT +0000). [09:46:00] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/a514f62a4bb2...990844ada725 [09:46:02] [02ssl] 07WikiTideSSLBot 03990844a - Bot: Update SSL cert for robloxapi.wiki [09:51:07] PROBLEM - pwiki.drydraytonvillagehall.org.uk - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'pwiki.drydraytonvillagehall.org.uk' expires in 15 day(s) (Sun 28 Jul 2024 09:48:47 AM GMT +0000). [09:51:17] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/990844ada725...7625ec3701f6 [09:51:20] [02ssl] 07WikiTideSSLBot 037625ec3 - Bot: Update SSL cert for pwiki.drydraytonvillagehall.org.uk [09:58:26] RECOVERY - mabi.myomi.net - LetsEncrypt on sslhost is OK: OK - Certificate 'mabi.myomi.net' will expire on Thu 10 Oct 2024 08:28:48 AM GMT +0000. [10:00:50] PROBLEM - patternarchive.online - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'patternarchive.online' expires in 15 day(s) (Sun 28 Jul 2024 09:50:16 AM GMT +0000). [10:01:47] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [10:03:01] RECOVERY - wiki.eversd.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.eversd.com' will expire on Thu 10 Oct 2024 08:34:06 AM GMT +0000. [10:03:47] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [48 system event log (SEL) entries present] [10:04:55] PROBLEM - journeytheword.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.journeytheword.wiki' expires in 15 day(s) (Sun 28 Jul 2024 09:51:43 AM GMT +0000). [10:12:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [10:13:17] PROBLEM - wiki.potabi.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.potabi.com' expires in 15 day(s) (Sun 28 Jul 2024 10:06:54 AM GMT +0000). [10:13:29] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/7625ec3701f6...fb10b8732ae9 [10:13:32] [02ssl] 07WikiTideSSLBot 03fb10b87 - Bot: Update SSL cert for wiki.potabi.com [10:17:50] PROBLEM - www.journeytheword.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'www.journeytheword.wiki' expires in 15 day(s) (Sun 28 Jul 2024 09:51:43 AM GMT +0000). [10:18:08] RECOVERY - wiki.consid.vn - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.consid.vn' will expire on Thu 10 Oct 2024 08:20:17 AM GMT +0000. [10:20:48] RECOVERY - pwiki.drydraytonvillagehall.org.uk - LetsEncrypt on sslhost is OK: OK - Certificate 'pwiki.drydraytonvillagehall.org.uk' will expire on Thu 10 Oct 2024 08:51:12 AM GMT +0000. [10:29:41] PROBLEM - wiki.worldbuilding.network - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.worldbuilding.network' expires in 15 day(s) (Sun 28 Jul 2024 10:17:38 AM GMT +0000). [10:29:52] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/fb10b8732ae9...b56892c83932 [10:29:55] [02ssl] 07WikiTideSSLBot 03b56892c - Bot: Update SSL cert for wiki.worldbuilding.network [10:33:55] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw161 [10:37:20] PROBLEM - worldsanskrit.net - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'worldsanskrit.net' expires in 15 day(s) (Sun 28 Jul 2024 10:06:23 AM GMT +0000). [10:37:33] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/b56892c83932...64ebc0c10042 [10:37:34] [02ssl] 07WikiTideSSLBot 0364ebc0c - Bot: Update SSL cert for worldsanskrit.net [10:37:55] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [10:41:44] PROBLEM - wiki.alathramc.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.alathramc.com' expires in 15 day(s) (Sun 28 Jul 2024 10:10:13 AM GMT +0000). [10:41:56] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/64ebc0c10042...c38351ff8414 [10:41:57] [02ssl] 07WikiTideSSLBot 03c38351f - Bot: Update SSL cert for wiki.alathramc.com [10:43:00] RECOVERY - wiki.potabi.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.potabi.com' will expire on Thu 10 Oct 2024 09:13:23 AM GMT +0000. [10:43:48] RECOVERY - robloxapi.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'robloxapi.wiki' will expire on Thu 10 Oct 2024 08:45:54 AM GMT +0000. [10:48:36] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o