[00:07:01] [02python-functions] 07dependabot[bot] pushed 031 commit to 03dependabot/pip/dot-github/setuptools-70.2.0 [+0/-0/±1] 13https://github.com/miraheze/python-functions/commit/3b936ff33877 [00:07:02] [02python-functions] 07dependabot[bot] 033b936ff - Bump setuptools from 70.1.1 to 70.2.0 in /.github [00:07:05] [02python-functions] 07dependabot[bot] created branch 03dependabot/pip/dot-github/setuptools-70.2.0 - 13https://github.com/miraheze/python-functions [00:07:06] [02python-functions] 07dependabot[bot] opened pull request 03#35: Bump setuptools from 70.1.1 to 70.2.0 in /.github - 13https://github.com/miraheze/python-functions/pull/35 [00:07:09] [02python-functions] 07dependabot[bot] labeled pull request 03#35: Bump setuptools from 70.1.1 to 70.2.0 in /.github - 13https://github.com/miraheze/python-functions/pull/35 [00:07:12] [02python-functions] 07dependabot[bot] labeled pull request 03#35: Bump setuptools from 70.1.1 to 70.2.0 in /.github - 13https://github.com/miraheze/python-functions/pull/35 [00:07:15] [02python-functions] 07dependabot[bot] pushed 031 commit to 03dependabot/pip/dot-github/flake8-comprehensions-3.15.0 [+0/-0/±1] 13https://github.com/miraheze/python-functions/commit/57075d9d11aa [00:07:16] [02python-functions] 07dependabot[bot] 0357075d9 - Bump flake8-comprehensions from 3.14.0 to 3.15.0 in /.github [00:07:18] [02python-functions] 07dependabot[bot] created branch 03dependabot/pip/dot-github/flake8-comprehensions-3.15.0 - 13https://github.com/miraheze/python-functions [00:07:19] [02python-functions] 07dependabot[bot] opened pull request 03#36: Bump flake8-comprehensions from 3.14.0 to 3.15.0 in /.github - 13https://github.com/miraheze/python-functions/pull/36 [00:07:21] [02python-functions] 07dependabot[bot] labeled pull request 03#36: Bump flake8-comprehensions from 3.14.0 to 3.15.0 in /.github - 13https://github.com/miraheze/python-functions/pull/36 [00:07:23] [02python-functions] 07dependabot[bot] labeled pull request 03#36: Bump flake8-comprehensions from 3.14.0 to 3.15.0 in /.github - 13https://github.com/miraheze/python-functions/pull/36 [00:07:24] [02python-functions] 07coderabbitai[bot] commented on pull request 03#35: Bump setuptools from 70.1.1 to 70.2.0 in /.github - 13https://github.com/miraheze/python-functions/pull/35#issuecomment-2201494138 [00:07:28] [02python-functions] 07coderabbitai[bot] commented on pull request 03#36: Bump flake8-comprehensions from 3.14.0 to 3.15.0 in /.github - 13https://github.com/miraheze/python-functions/pull/36#issuecomment-2201494285 [00:11:14] miraheze/python-functions - dependabot[bot] the build passed. [00:12:29] miraheze/python-functions - dependabot[bot] the build passed. [00:44:03] RECOVERY - wiki.aclevo.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.aclevo.com' will expire on Sun 29 Sep 2024 10:46:45 PM GMT +0000. [01:06:06] miraheze/mw-config - waki285 the build passed. [01:06:07] [02mw-config] 07OAuthority edited pull request 03#5597: T12272: Rename global-sysop to global-admin - 13https://github.com/miraheze/mw-config/pull/5597 [01:06:36] [02mw-config] 07OAuthority closed pull request 03#5597: T12272: Rename global-sysop to global-admin - 13https://github.com/miraheze/mw-config/pull/5597 [01:06:38] [02mw-config] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/749ec986e608...7af0633e1252 [01:06:38] [02mw-config] 07waki285 037af0633 - T12272: Rename global-sysop to global-admin (#5597) [01:07:33] miraheze/mw-config - OAuthority the build passed. [01:07:37] !log [@mwtask171] starting deploy of {'config': True} to all [01:07:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:07:48] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 10s [01:07:51] [02MirahezeMagic] 07OAuthority edited pull request 03#495: T12272: Add global-admin translations, per RfC - 13https://github.com/miraheze/MirahezeMagic/pull/495 [01:07:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:08:14] [02MirahezeMagic] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/425be95acf25...90e8b7097b61 [01:08:16] [02MirahezeMagic] 07waki285 0390e8b70 - T12272: Add global-admin translations, per RfC (#495) [01:08:19] [02MirahezeMagic] 07OAuthority closed pull request 03#495: T12272: Add global-admin translations, per RfC - 13https://github.com/miraheze/MirahezeMagic/pull/495 [01:08:47] [02MirahezeMagic] 07OAuthority edited pull request 03#494: T12272: Add i18n for Wiki Mechanic - 13https://github.com/miraheze/MirahezeMagic/pull/494 [01:09:29] [02MirahezeMagic] 07OAuthority closed pull request 03#494: T12272: Add i18n for Wiki Mechanic - 13https://github.com/miraheze/MirahezeMagic/pull/494 [01:09:31] [02MirahezeMagic] 07OAuthority pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/MirahezeMagic/compare/90e8b7097b61...3e1554a43221 [01:09:33] [02MirahezeMagic] 07waki285 033e1554a - T12272: Add i18n for Wiki Mechanic (#494) [01:09:51] [02MirahezeMagic] 07OAuthority edited pull request 03#493: T12272: Add global-patroller i18n - 13https://github.com/miraheze/MirahezeMagic/pull/493 [01:10:28] [02MirahezeMagic] 07OAuthority closed pull request 03#493: T12272: Add global-patroller i18n - 13https://github.com/miraheze/MirahezeMagic/pull/493 [01:10:30] [02MirahezeMagic] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/3e1554a43221...ad9286d6ea5c [01:10:32] [02MirahezeMagic] 07waki285 03ad9286d - T12272: Add global-patroller i18n (#493) [01:12:05] !log [oa@mwtask181] starting deploy of {'upgrade_world': True, 'l10n': True, 'versions': '1.41', 'upgrade_extensions': 'all', 'upgrade_skins': 'all'} to all [01:12:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:13:00] oopsie didn't mean that one! [01:13:13] !log [oa@mwtask181] starting deploy of {'config': True} to all [01:13:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:13:27] !log [oa@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 13s [01:13:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:13:44] [02MirahezeMagic] 07brandon-wm opened pull request 03#497: Update en.json - 13https://github.com/miraheze/MirahezeMagic/pull/497 [01:13:55] [02MirahezeMagic] 07coderabbitai[bot] commented on pull request 03#497: Update en.json - 13https://github.com/miraheze/MirahezeMagic/pull/497#issuecomment-2201597801 [01:14:05] miraheze/MirahezeMagic - OAuthority the build has errored. [01:14:28] [02MirahezeMagic] 07OAuthority closed pull request 03#497: Update en.json - 13https://github.com/miraheze/MirahezeMagic/pull/497 [01:14:30] [02MirahezeMagic] 07OAuthority pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/ad9286d6ea5c...7ae49e67b886 [01:14:33] [02MirahezeMagic] 07brandon-wm 037ae49e6 - Update en.json (#497) [01:14:44] !log [oa@mwtask181] starting deploy of {'l10n': True, 'versions': '1.41', 'upgrade_extensions': 'MirahezeMagic'} to all [01:14:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:18:34] miraheze/MirahezeMagic - brandon-wm the build has errored. [01:18:39] miraheze/MirahezeMagic - OAuthority the build has errored. [01:21:41] !log [oa@mwtask181] finished deploy of {'l10n': True, 'versions': '1.41', 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 417s [01:21:46] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:23:50] !log [@test151] starting deploy of {'config': True} to test151 [01:23:51] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [01:23:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:03] !log [@test151] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 [01:24:04] !log [@test151] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [01:24:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:14] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [01:24:15] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [01:24:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:27] !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 [01:24:28] !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [01:24:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:24:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:26:55] !log [oa@mwtask181] sudo -u www-data php /srv/mediawiki/1.41/maintenance/run.php /srv/mediawiki/1.41/extensions/Translate/scripts/moveTranslatableBundle.php Global Sysops Global Administrators Maintenance script --reason [[phorge:T12272|T12272]]: Rename Global Sysops page on Meta --wiki=metawiki [01:26:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:32:41] !log [@mwtask181] starting deploy of {'config': True} to all [01:32:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:32:53] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 11s [01:32:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:33:16] !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [01:33:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:33:27] !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 10s [01:33:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:17] !log [@mwtask171] starting deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all [01:37:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:28] !log [@mwtask171] finished deploy of {'folders': '1.41/extensions/MirahezeMagic'} to all - SUCCESS in 10s [01:37:36] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:38] !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [01:37:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:37:50] !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 11s [01:37:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:35:09] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw162 [02:35:26] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw162 [02:35:36] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 2 backends are down. mw162 mw182 [02:37:09] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [02:37:26] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [02:37:36] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [03:01:01] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw162 [03:03:00] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [03:13:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [03:18:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:01:58] PROBLEM - es.countryhumans.polandball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'es.countryhumans.polandball.wiki' expires in 15 day(s) (Thu 18 Jul 2024 03:50:03 AM GMT +0000). [04:02:10] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/4d2031089065...43fc38b52a4c [04:02:11] [02ssl] 07WikiTideSSLBot 0343fc38b - Bot: Update SSL cert for es.countryhumans.polandball.wiki [04:31:56] RECOVERY - es.countryhumans.polandball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'es.countryhumans.polandball.wiki' will expire on Mon 30 Sep 2024 03:02:04 AM GMT +0000. [04:33:26] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw152 [04:35:26] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [04:46:36] PROBLEM - wiki.3point0.science - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki.3point0.science' expires in 7 day(s) (Wed 10 Jul 2024 04:22:18 AM GMT +0000). [04:48:36] PROBLEM - ru.countryhumans.polandball.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'ru.countryhumans.polandball.wiki' expires in 15 day(s) (Thu 18 Jul 2024 04:37:23 AM GMT +0000). [04:48:55] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/43fc38b52a4c...93e88f4d1cd8 [04:48:57] [02ssl] 07WikiTideSSLBot 0393e88f4 - Bot: Update SSL cert for ru.countryhumans.polandball.wiki [04:57:44] PROBLEM - atlas.starworld.zone - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'atlas.starworld.zone' expires in 15 day(s) (Thu 18 Jul 2024 04:38:48 AM GMT +0000). [04:57:57] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/93e88f4d1cd8...5c8fa38eba79 [04:58:00] [02ssl] 07WikiTideSSLBot 035c8fa38 - Bot: Update SSL cert for atlas.starworld.zone [05:12:17] [02python-functions] 07github-merge-queue[bot] created branch 03gh-readonly-queue/master/pr-36-4e4dfde82b37074458bdd4a299e9366ec50f3e55 - 13https://github.com/miraheze/python-functions [05:12:19] [02python-functions] 07github-merge-queue[bot] pushed 031 commit to 03gh-readonly-queue/master/pr-36-4e4dfde82b37074458bdd4a299e9366ec50f3e55 [+0/-0/±1] 13https://github.com/miraheze/python-functions/commit/4ef4bab4f6d4 [05:12:21] [02python-functions] 07dependabot[bot] 034ef4bab - Bump flake8-comprehensions from 3.14.0 to 3.15.0 in /.github (#36) [05:12:40] [02python-functions] 07github-merge-queue[bot] pushed 031 commit to 03gh-readonly-queue/master/pr-35-4ef4bab4f6d4a72e679270a4103ddd39aef41c10 [+0/-0/±1] 13https://github.com/miraheze/python-functions/commit/5ac5bcb4c9c4 [05:12:43] [02python-functions] 07dependabot[bot] 035ac5bcb - Bump setuptools from 70.1.1 to 70.2.0 in /.github (#35) [05:12:46] [02python-functions] 07github-merge-queue[bot] created branch 03gh-readonly-queue/master/pr-35-4ef4bab4f6d4a72e679270a4103ddd39aef41c10 - 13https://github.com/miraheze/python-functions [05:18:33] [02python-functions] 07github-merge-queue[bot] pushed 032 commits to 03master [+0/-0/±2] 13https://github.com/miraheze/python-functions/compare/4e4dfde82b37...5ac5bcb4c9c4 [05:18:34] [02python-functions] 07github-merge-queue[bot] deleted branch 03gh-readonly-queue/master/pr-35-4ef4bab4f6d4a72e679270a4103ddd39aef41c10 [05:18:36] [02python-functions] 07RhinosF1 closed pull request 03#36: Bump flake8-comprehensions from 3.14.0 to 3.15.0 in /.github - 13https://github.com/miraheze/python-functions/pull/36 [05:18:39] [02python-functions] 07RhinosF1 closed pull request 03#35: Bump setuptools from 70.1.1 to 70.2.0 in /.github - 13https://github.com/miraheze/python-functions/pull/35 [05:18:42] [02python-functions] 07github-merge-queue[bot] deleted branch 03gh-readonly-queue/master/pr-36-4e4dfde82b37074458bdd4a299e9366ec50f3e55 [05:18:43] [02python-functions] 07github-merge-queue[bot] deleted branch 03gh-readonly-queue/master/pr-36-4e4dfde82b37074458bdd4a299e9366ec50f3e55 - 13https://github.com/miraheze/python-functions [05:18:44] [02python-functions] 07github-merge-queue[bot] deleted branch 03gh-readonly-queue/master/pr-35-4ef4bab4f6d4a72e679270a4103ddd39aef41c10 - 13https://github.com/miraheze/python-functions [05:18:46] [02python-functions] 07RhinosF1 deleted branch 03dependabot/pip/dot-github/setuptools-70.2.0 [05:18:49] [02python-functions] 07RhinosF1 deleted branch 03dependabot/pip/dot-github/setuptools-70.2.0 - 13https://github.com/miraheze/python-functions [05:18:52] [02python-functions] 07RhinosF1 deleted branch 03dependabot/pip/dot-github/flake8-comprehensions-3.15.0 [05:18:53] [02python-functions] 07RhinosF1 deleted branch 03dependabot/pip/dot-github/flake8-comprehensions-3.15.0 - 13https://github.com/miraheze/python-functions [05:23:10] miraheze/python-functions - github-merge-queue[bot] the build passed. [05:26:28] RECOVERY - atlas.starworld.zone - LetsEncrypt on sslhost is OK: OK - Certificate 'atlas.starworld.zone' will expire on Mon 30 Sep 2024 03:57:52 AM GMT +0000. [05:46:36] RECOVERY - ru.countryhumans.polandball.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'ru.countryhumans.polandball.wiki' will expire on Mon 30 Sep 2024 03:48:48 AM GMT +0000. [05:57:00] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:02:00] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:21:03] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.33, 20.25, 16.03 [06:23:03] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 19.54, 20.08, 16.50 [06:31:30] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [06:41:30] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:03:38] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zymonic.com All nameservers failed to answer the query. [07:09:17] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.99, 20.94, 18.70 [07:13:11] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.78, 19.54, 18.75 [07:22:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:22:50] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [08:52:13] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Sat 27 Jul 2024 09:59:27 PM GMT +0000. [09:02:41] RECOVERY - wiki.zymonic.com - reverse DNS on sslhost is OK: SSL OK - wiki.zymonic.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [09:49:08] can infoboxes be imported from Microwiki? [09:56:20] please ask on #general or #tech [10:58:00] PROBLEM - tdr.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tdr.wiki' expires in 15 day(s) (Thu 18 Jul 2024 10:44:07 AM GMT +0000). [10:58:11] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/5c8fa38eba79...a1e41c0b0ab5 [10:58:12] [02ssl] 07WikiTideSSLBot 03a1e41c0 - Bot: Update SSL cert for tdr.wiki [11:22:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:27:49] RECOVERY - tdr.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'tdr.wiki' will expire on Mon 30 Sep 2024 09:58:04 AM GMT +0000. [11:47:25] [Grafana] !tech RESOLVED: High Job Queue Backlog https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [12:13:03] PROBLEM - db171 Backups SQL on db171 is CRITICAL: FILE_AGE CRITICAL: /var/log/sql-backup.log is 1209703 seconds old and 134735 bytes [13:25:58] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.68, 21.63, 16.95 [13:29:52] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 15.42, 18.68, 16.81 [14:21:30] PROBLEM - wiki.zymonic.com - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [14:31:46] PROBLEM - wiki.zymonic.com - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.zymonic.com All nameservers failed to answer the query. [15:01:29] RECOVERY - wiki.zymonic.com - reverse DNS on sslhost is OK: SSL OK - wiki.zymonic.com reverse DNS resolves to cp36.wikitide.net - CNAME OK [15:21:03] RECOVERY - wiki.zymonic.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.zymonic.com' will expire on Sat 27 Jul 2024 09:59:27 PM GMT +0000. [17:13:15] !log [alex@test151] starting deploy of {'world': True, 'versions': '1.42'} to test151 [17:13:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:13:38] !log [alex@test151] finished deploy of {'world': True, 'versions': '1.42'} to test151 - SUCCESS in 22s [17:13:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:15:06] PROBLEM - test151 Puppet on test151 is WARNING: WARNING: Puppet is currently disabled, message: bisecting mediawiki core, disabling puppet to prevent it from interfering, last run 20 minutes ago with 0 failures [17:53:06] RECOVERY - test151 Puppet on test151 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [17:57:10] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [17:57:11] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [17:57:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:57:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:01:31] !log [alex@test151] starting deploy of {'upgrade_world': True, 'versions': '1.42', 'upgrade_extensions': 'all', 'upgrade_skins': 'all'} to test151 [18:01:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:04:39] !log [alex@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [18:04:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:07:24] PROBLEM - test151 MediaWiki Rendering on test151 is UNKNOWN: HTTP UNKNOWN: Failed to unchunk message body [18:08:36] [02mediawiki-repos] 07redbluegreenhat pushed 031 commit to 03ext-use-branch [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/commit/2ef1b8c15afa [18:08:39] [02mediawiki-repos] 07redbluegreenhat 032ef1b8c - Use _branch_ for some extensions [18:08:40] [02mediawiki-repos] 07redbluegreenhat created branch 03ext-use-branch - 13https://github.com/miraheze/mediawiki-repos [18:08:44] [02mediawiki-repos] 07redbluegreenhat opened pull request 03#23: Use _branch_ for some extensions - 13https://github.com/miraheze/mediawiki-repos/pull/23 [18:08:49] [02mediawiki-repos] 07coderabbitai[bot] commented on pull request 03#23: Use _branch_ for some extensions - 13https://github.com/miraheze/mediawiki-repos/pull/23#issuecomment-2203993036 [18:09:23] [02mediawiki-repos] 07redbluegreenhat closed pull request 03#23: Use _branch_ for some extensions - 13https://github.com/miraheze/mediawiki-repos/pull/23 [18:09:25] [02mediawiki-repos] 07redbluegreenhat deleted branch 03ext-use-branch [18:09:28] [02mediawiki-repos] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/13d742a89479...e0fda9ac1f75 [18:09:30] [02mediawiki-repos] 07redbluegreenhat 03e0fda9a - Use _branch_ for some extensions (#23) [18:09:31] [02mediawiki-repos] 07redbluegreenhat deleted branch 03ext-use-branch - 13https://github.com/miraheze/mediawiki-repos [18:09:40] !log [alex@test151] starting deploy of {'extension_list': True, 'versions': '1.42'} to test151 [18:09:41] !log [alex@test151] finished deploy of {'extension_list': True, 'versions': '1.42'} to test151 - SUCCESS in 1s [18:09:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:09:47] !log [alex@test151] starting deploy of {'upgrade_world': True, 'versions': '1.42', 'upgrade_extensions': 'all', 'upgrade_skins': 'all'} to test151 [18:09:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:09:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:12:44] !log [alex@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [18:13:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:14:38] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.63, 21.15, 18.92 [18:15:06] !log [alex@test151] starting deploy of {'upgrade_world': True, 'versions': '1.42', 'upgrade_extensions': 'all', 'upgrade_skins': 'all'} to test151 [18:15:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:18:07] !log [alex@test151] DEPLOY ABORTED: Non-Zero Exit Code in prep, see output. [18:18:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:22:24] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.26, 23.28, 20.70 [18:30:10] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 16.79, 21.77, 21.26 [18:36:54] PROBLEM - mwtask181 Puppet on mwtask181 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 3 minutes ago with 10 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_41 AJAXPoll],Exec[git_checkout_MediaWiki-REL1_41 Comments],Exec[git_checkout_MediaWiki-REL1_41 QuizGame],Exec[git_checkout_MediaWiki-REL1_41 VoteNY] [18:37:57] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.10, 18.77, 20.11 [18:40:14] PROBLEM - mwtask171 Puppet on mwtask171 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_41 AJAXPoll],Exec[git_checkout_MediaWiki-REL1_41 Comments],Exec[git_checkout_MediaWiki-REL1_41 QuizGame],Exec[git_checkout_MediaWiki-REL1_41 VoteNY] [18:57:06] PROBLEM - test151 Puppet on test151 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 2 minutes ago with 10 failures. Failed resources (up to 3 shown): Exec[git_checkout_MediaWiki-REL1_41 AJAXPoll],Exec[git_checkout_MediaWiki-REL1_41 Comments],Exec[git_checkout_MediaWiki-REL1_41 QuizGame],Exec[git_checkout_MediaWiki-REL1_41 VoteNY] [19:46:48] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [19:48:39] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o