[00:07:26] RECOVERY - cp37 Disk Space on cp37 is OK: DISK OK - free space: / 56855MiB (12% inode=99%); [00:08:11] RECOVERY - cp38 Disk Space on cp38 is OK: DISK OK - free space: / 57399MiB (12% inode=99%); [03:02:55] PROBLEM - mon181 Backups Grafana on mon181 is WARNING: FILE_AGE WARNING: /var/log/grafana-backup/grafana-backup/grafana-backup.log is 1382554 seconds old and 753 bytes [07:46:06] PROBLEM - with.cpt-ra.bid - Cloudflare on sslhost is CRITICAL: Name or service not knownHTTP CRITICAL - Unable to open TCP socket [08:15:13] RECOVERY - with.cpt-ra.bid - Cloudflare on sslhost is OK: OK - Certificate 'with.cpt-ra.bid' will expire on Sat 24 May 2025 07:05:41 PM GMT +0000. [08:55:26] PROBLEM - cp37 Disk Space on cp37 is WARNING: DISK WARNING - free space: / 49880MiB (10% inode=99%); [09:44:11] PROBLEM - cp38 Disk Space on cp38 is WARNING: DISK WARNING - free space: / 49929MiB (10% inode=99%); [11:27:00] !log [blankeclair@mwtask181] starting deploy of {'l10n': True, 'versions': '1.43', 'upgrade_extensions': 'PortableInfobox'} to all [11:27:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:27:08] !log [blankeclair@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [11:27:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:27:21] !log [blankeclair@mwtask181] starting deploy of {'pull': 'PortableInfobox', 'l10n': True, 'versions': '1.43', 'upgrade_extensions': 'PortableInfobox'} to all [11:27:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:27:23] !log [blankeclair@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [11:27:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:27:51] !log [blankeclair@mwtask181] starting deploy of {'l10n': True, 'versions': '1.43', 'upgrade_extensions': 'PortableInfobox'} to all [11:27:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:34:53] !log [blankeclair@mwtask181] finished deploy of {'l10n': True, 'versions': '1.43', 'upgrade_extensions': 'PortableInfobox'} to all - SUCCESS in 421s [11:34:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:57:42] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/760a2042d60a [11:57:42] 02puppet/03paladox-patch-1 07paladox 03760a204 cache::haproxy: Remove X-OpenStack-Request-ID header if X-Miraheze-Debug isn't present [11:57:54] [02puppet] 07paladox opened pull request #4318: cache::haproxy: Remove X-OpenStack-Request-ID header if X-Miraheze-De… (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4318 [11:57:59] [02puppet] 07coderabbitai[bot] commented on pull request #4318: --- […] 13https://github.com/miraheze/puppet/pull/4318#issuecomment-2792501532 [11:58:00] [02puppet] 07paladox retitled PR #4318: "cache::haproxy: Remove X-OpenStack-Request-ID header if X-Miraheze-De…" ➜ "cache::haproxy: Remove X-OpenStack-Request-ID header if X-Miraheze-Debug isn't present" 13https://github.com/miraheze/puppet/pull/4318 [11:59:10] [02puppet] 07paladox pushed 1 new commit to 03paladox-patch-1 13https://github.com/miraheze/puppet/commit/94dad4517f822974fd8349a513ad1195637b0b72 [11:59:10] 02puppet/03paladox-patch-1 07paladox 0394dad45 Update cache.yaml [11:59:18] [02puppet] 07paladox retitled PR #4318: "cache::haproxy: Remove X-OpenStack-Request-ID header if X-Miraheze-Debug isn't present" ➜ "cache::haproxy: Remove X-OpenStack-Request-ID header if X-Wikitide-Debug isn't present" 13https://github.com/miraheze/puppet/pull/4318 [11:59:45] [02puppet] 07paladox merged pull request #4318: cache::haproxy: Remove X-OpenStack-Request-ID header if X-Wikitide-Debug isn't present (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4318 [11:59:47] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/eeaa6cda1d632507e9f29be1552152d313aca04e [11:59:49] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 0394dad45 13https://api.github.com/repos/miraheze/puppet/commit/94dad45 [11:59:51] 02puppet/03main 07paladox 03eeaa6cd cache::haproxy: Remove X-OpenStack-Request-ID header if X-Wikitide-Debug isn't present (#4318)… [14:43:08] PROBLEM - encyclopediarobotica.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'encyclopediarobotica.org' expires in 14 day(s) (Fri 25 Apr 2025 02:16:23 PM GMT +0000). [14:43:20] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/c27e07cbe69fb53a3ef35369fb8242436e9140e8 [14:43:20] 02ssl/03main 07WikiTideSSLBot 03c27e07c Bot: Update SSL cert for encyclopediarobotica.org [14:45:47] [02ssl] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/7500b4ff4716e77c2cc41c9a84c230305c42881b [14:45:47] 02ssl/03main 07MacFan4000 037500b4f switch infectowiki.com to cloudflare. [14:45:49] [02dns] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/2770417da490827dd4e2eaf616844ebc33f66f5e [14:45:50] 02dns/03main 07MacFan4000 032770417 Remove zone for infectowiki.com [14:46:41] miraheze/dns - MacFan4000 the build passed. [15:12:16] RECOVERY - encyclopediarobotica.org - LetsEncrypt on sslhost is OK: OK - Certificate 'encyclopediarobotica.org' will expire on Wed 09 Jul 2025 01:44:44 PM GMT +0000. [16:15:03] PROBLEM - cp38 Puppet on cp38 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[haproxy] [16:15:05] PROBLEM - cp37 Puppet on cp37 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[haproxy] [16:19:03] RECOVERY - cp38 Puppet on cp38 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:19:05] RECOVERY - cp37 Puppet on cp37 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [17:08:29] PROBLEM - matomo151 Current Load on matomo151 is WARNING: LOAD WARNING - total load average: 7.56, 6.80, 6.26 [17:10:25] RECOVERY - matomo151 Current Load on matomo151 is OK: LOAD OK - total load average: 6.36, 6.69, 6.29 [17:49:32] !log [paladox@mwtask181] starting deploy of {'upgrade_world': True, 'versions': '1.43', 'upgrade_extensions': 'all', 'upgrade_skins': 'all'} to all [17:49:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:49:52] !log [paladox@test151] starting deploy of {'upgrade_world': True, 'versions': ['1.43', '1.44'], 'upgrade_extensions': 'all', 'upgrade_skins': 'all'} to test151 [17:49:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:55:17] !log [paladox@mwtask181] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [17:55:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:56:04] !log [paladox@mwtask181] starting deploy of {'upgrade_world': True, 'force_upgrade': True, 'versions': '1.43', 'upgrade_extensions': 'all', 'upgrade_skins': 'all'} to all [17:56:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:56:29] !log [paladox@mwtask181] starting deploy of {'upgrade_world': True, 'force_upgrade': True, 'versions': '1.43', 'upgrade_extensions': 'all', 'upgrade_skins': 'all'} to all [17:56:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:58:03] !log [paladox@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [17:58:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:59:28] !log [paladox@test151] starting deploy of {'upgrade_world': True, 'force_upgrade': True, 'versions': ['1.43', '1.44'], 'upgrade_extensions': 'all', 'upgrade_skins': 'all'} to test151 [17:59:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:16:40] PROBLEM - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is WARNING: WARNING - NGINX Error Rate is 43% [18:18:40] RECOVERY - cp36 HTTP 4xx/5xx ERROR Rate on cp36 is OK: OK - NGINX Error Rate is 11% [18:26:47] !log [paladox@mwtask181] finished deploy of {'upgrade_world': True, 'force_upgrade': True, 'versions': '1.43', 'upgrade_extensions': 'all', 'upgrade_skins': 'all'} to all - SUCCESS in 1818s [18:26:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:19:39] [02ssl] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/d8140a441e39406e0385356b892530f89f8b3091 [19:19:39] 02ssl/03main 07MacFan4000 03d8140a4 rm lostmediawikiru redirect [19:21:58] [02dns] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/f3879441ba3f125994ffe1e7f74d0f35161508d1 [19:21:58] 02dns/03main 07MacFan4000 03f387944 Delete zones/lostmediawiki.ru [19:22:56] miraheze/dns - MacFan4000 the build passed. [19:26:11] [02ssl] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/42b5eb9d1916fc0ee11633bf5dc0262471e3e0b7 [19:26:11] 02ssl/03main 07MacFan4000 0342b5eb9 Delete certificates/lostmediawiki.ru.crt [19:29:29] [02ssl] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/123cd67a402f66ea7ddd6a257fae5acbf5b48742 [19:29:29] 02ssl/03main 07MacFan4000 03123cd67 Update certs.yaml [19:39:03] !log [paladox@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [19:39:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:48:55] did we just loose dns [20:55:27] PROBLEM - osfirsttimer.wiki - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:07:05] [02dns] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/dns/commit/99477fe4ccbe1b08c0d2abcdd248cd1832670e1a [21:07:05] 02dns/03main 07MacFan4000 0399477fe remove unused domains… [21:07:25] [02ssl] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/ab5009e96ba9b1c024004fc04d27bbd76c4468ad [21:07:25] 02ssl/03main 07MacFan4000 03ab5009e remove unused domains… [21:09:16] miraheze/dns - MacFan4000 the build passed. [21:16:35] PROBLEM - binrayarchives.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'binrayarchives.com' expires in 14 day(s) (Fri 25 Apr 2025 09:14:14 PM GMT +0000). [21:16:49] [02ssl] 07WikiTideSSLBot pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/949312eb681afd8af1aaa311cec91f133e13a185 [21:16:50] 02ssl/03main 07WikiTideSSLBot 03949312e Bot: Update SSL cert for binrayarchives.com [21:36:50] [02ssl] 07MacFan4000 pushed 1 new commit to 03main 13https://github.com/miraheze/ssl/commit/34f963bf707df7b5359aa86c7e2b6f395f00b013 [21:36:50] 02ssl/03main 07MacFan4000 0334f963b Update redirects.yaml [21:37:59] PROBLEM - www.pyramidgames.wiki - Cloudflare on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [21:53:18] RECOVERY - osfirsttimer.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'osfirsttimer.wiki' will expire on Wed 07 May 2025 06:30:08 PM GMT +0000. [21:55:34] [02puppet] 07paladox created 03paladox-patch-1 (+1 new commit) 13https://github.com/miraheze/puppet/commit/a5a68f7f873d [21:55:34] 02puppet/03paladox-patch-1 07paladox 03a5a68f7 cache::haproxy: set http-reuse to always and set http-server-close for frontend [21:55:39] [02puppet] 07paladox opened pull request #4319: cache::haproxy: set http-reuse to always and set http-server-close fo… (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4319 [21:55:43] [02puppet] 07paladox retitled PR #4319: "cache::haproxy: set http-reuse to always and set http-server-close fo…" ➜ "cache::haproxy: set http-reuse to always and set http-server-close for frontend" 13https://github.com/miraheze/puppet/pull/4319 [21:55:45] [02puppet] 07coderabbitai[bot] commented on pull request #4319: --- […] 13https://github.com/miraheze/puppet/pull/4319#issuecomment-2795250398 [21:55:50] [02puppet] 07paladox pushed 1 new commit to 03main 13https://github.com/miraheze/puppet/commit/da69402a550c2a5f3f8b4bd718d654900fef0291 [21:55:51] 02puppet/03main 07paladox 03da69402 cache::haproxy: set http-reuse to always and set http-server-close for frontend (#4319)… [21:55:53] [02puppet] 07paladox merged pull request #4319: cache::haproxy: set http-reuse to always and set http-server-close for frontend (03main...03paladox-patch-1) 13https://github.com/miraheze/puppet/pull/4319 [21:55:55] [02puppet] 07paladox 04deleted 03paladox-patch-1 at 03a5a68f7 13https://api.github.com/repos/miraheze/puppet/commit/a5a68f7 [22:15:27] RECOVERY - binrayarchives.com - LetsEncrypt on sslhost is OK: OK - Certificate 'binrayarchives.com' will expire on Wed 09 Jul 2025 08:18:13 PM GMT +0000.