[00:05:01] [1/2] Error 1264: Out of range value for column 'us_image_bits' at row 1 [00:05:01] [2/2] Function: UploadStash::stashFile [00:05:43] wa [00:15:01] PROBLEM - permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'permanentfuturelab.wiki' expires in 7 day(s) (Sat 27 Jul 2024 12:13:59 AM GMT +0000). [00:17:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.12, 21.98, 23.94 [00:23:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.50, 22.58, 23.40 [00:25:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 16.40, 20.51, 22.58 [00:25:43] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [00:25:45] PROBLEM - www.permanentfuturelab.wiki - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'permanentfuturelab.wiki' expires in 7 day(s) (Sat 27 Jul 2024 12:13:59 AM GMT +0000). [00:27:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [00:27:44] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [Inlet Temp = Critical, 192 system event log (SEL) entries present] [00:31:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.90, 22.11, 23.83 [00:32:30] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [00:37:50] @orduin someone from wmnyc told me bout https://phabricator.wikimedia.org/T370304 [00:39:56] Hmm? How is that related exactly? [00:40:37] it looked like the same erorr as before [00:40:55] didnt look too close but looked it since it also mentioned file uload [00:41:08] may be similer but different [00:42:43] No, not related, I don't think [00:47:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.22, 17.69, 19.91 [00:50:20] [02MirahezeMagic] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/MirahezeMagic/compare/cdab4bfbc31a...375fd6d3bc61 [00:50:21] [02MirahezeMagic] 07The-Voidwalker 03375fd6d - fix broken hook [00:50:55] !log [void@mwtask181] starting deploy of {'versions': '1.42', 'upgrade_extensions': 'MirahezeMagic'} to all [00:51:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:51:13] !log [void@mwtask181] finished deploy of {'versions': '1.42', 'upgrade_extensions': 'MirahezeMagic'} to all - SUCCESS in 17s [00:51:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:51:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.22, 21.03, 20.74 [00:53:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.75, 20.28, 20.49 [00:53:54] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [00:53:55] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [00:54:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:54:10] !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 [00:54:11] !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [00:54:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:54:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:54:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.88, 25.16, 22.72 [00:57:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.39, 23.66, 21.79 [01:03:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.32, 22.93, 22.35 [01:06:52] [02puppet] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/d413f0298669...fe0398388a58 [01:06:53] [02puppet] 07The-Voidwalker 03fe03983 - temporarily disable error reporting for E_DEPRECATED [01:07:47] !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [01:07:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:08:00] !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 12s [01:08:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:09:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.42, 23.26, 22.57 [01:11:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.81, 23.88, 23.56 [01:11:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.74, 22.34, 22.31 [01:12:40] [02puppet] 07The-Voidwalker created branch 03patch-revert - 13https://github.com/miraheze/puppet [01:12:42] [02puppet] 07The-Voidwalker pushed 031 commit to 03patch-revert [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/d4752507f49a [01:12:43] [02puppet] 07The-Voidwalker 03d475250 - Revert "temporarily disable error reporting for E_DEPRECATED" [01:13:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.95, 24.93, 24.02 [01:13:02] [02puppet] 07The-Voidwalker opened pull request 03#3886: Revert "temporarily disable error reporting for E_DEPRECATED" - 13https://github.com/miraheze/puppet/pull/3886 [01:13:06] [02puppet] 07coderabbitai[bot] commented on pull request 03#3886: Revert "temporarily disable error reporting for E_DEPRECATED" - 13https://github.com/miraheze/puppet/pull/3886#issuecomment-2237836066 [01:13:24] [02puppet] 07The-Voidwalker commented on pull request 03#3886: Revert "temporarily disable error reporting for E_DEPRECATED" - 13https://github.com/miraheze/puppet/pull/3886#issuecomment-2237836247 [01:17:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.99, 22.70, 22.29 [01:19:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.06, 21.93, 22.06 [01:21:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.23, 24.03, 22.79 [01:23:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.91, 22.76, 22.49 [01:25:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.78, 25.00, 23.36 [01:27:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.57, 22.71, 23.74 [01:30:39] [02python-functions] 07dependabot[bot] pushed 031 commit to 03dependabot/pip/dot-github/setuptools-71.0.3 [+0/-0/±1] 13https://github.com/miraheze/python-functions/commit/e4d3aa87575f [01:30:42] [02python-functions] 07dependabot[bot] 03e4d3aa8 - Bump setuptools from 71.0.1 to 71.0.3 in /.github [01:30:43] [02python-functions] 07dependabot[bot] created branch 03dependabot/pip/dot-github/setuptools-71.0.3 - 13https://github.com/miraheze/python-functions [01:30:45] [02python-functions] 07dependabot[bot] labeled pull request 03#44: Bump setuptools from 71.0.1 to 71.0.3 in /.github - 13https://github.com/miraheze/python-functions/pull/44 [01:30:47] [02python-functions] 07dependabot[bot] labeled pull request 03#44: Bump setuptools from 71.0.1 to 71.0.3 in /.github - 13https://github.com/miraheze/python-functions/pull/44 [01:30:50] [02python-functions] 07dependabot[bot] opened pull request 03#44: Bump setuptools from 71.0.1 to 71.0.3 in /.github - 13https://github.com/miraheze/python-functions/pull/44 [01:30:52] [02python-functions] 07coderabbitai[bot] commented on pull request 03#44: Bump setuptools from 71.0.1 to 71.0.3 in /.github - 13https://github.com/miraheze/python-functions/pull/44#issuecomment-2237849242 [01:31:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.39, 22.55, 22.82 [01:33:35] miraheze/MirahezeMagic - The-Voidwalker the build has errored. [01:35:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 30.33, 24.96, 23.87 [01:35:44] !log [void@graylog161] change rotation strategy from 1day, preserve last 4; to 20GiB, preserve last 16 [01:35:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:36:26] miraheze/python-functions - dependabot[bot] the build passed. [01:37:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.35, 23.73, 23.61 [01:43:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.27, 23.80, 23.74 [01:43:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.22, 22.13, 22.10 [01:45:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.48, 23.69, 23.75 [01:45:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.04, 22.00, 22.06 [01:49:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 29.35, 25.36, 24.24 [01:49:40] PROBLEM - os161 Disk Space on os161 is CRITICAL: DISK CRITICAL - free space: / 13104MiB (5% inode=99%); [01:49:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.73, 22.90, 22.39 [01:51:40] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 13722MiB (6% inode=99%); [01:51:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.14, 22.26, 22.22 [01:53:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.42, 23.51, 23.78 [01:53:58] PROBLEM - os151 Disk Space on os151 is CRITICAL: DISK CRITICAL - free space: / 13199MiB (5% inode=99%); [01:57:40] PROBLEM - os161 Disk Space on os161 is CRITICAL: DISK CRITICAL - free space: / 13256MiB (5% inode=99%); [01:57:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.39, 24.08, 22.88 [01:57:58] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 13410MiB (6% inode=99%); [01:59:40] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 13763MiB (6% inode=99%); [01:59:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.70, 22.21, 22.38 [01:59:58] PROBLEM - os151 Disk Space on os151 is CRITICAL: DISK CRITICAL - free space: / 12951MiB (5% inode=99%); [02:01:40] PROBLEM - os161 Disk Space on os161 is CRITICAL: DISK CRITICAL - free space: / 13086MiB (5% inode=99%); [02:01:58] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 13476MiB (6% inode=99%); [02:03:58] PROBLEM - os151 Disk Space on os151 is CRITICAL: DISK CRITICAL - free space: / 13096MiB (5% inode=99%); [02:05:58] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 13413MiB (6% inode=99%); [02:07:12] yes vbut you can also just do ?action=purge for a single page which will also purge varnish [02:07:58] PROBLEM - os151 Disk Space on os151 is CRITICAL: DISK CRITICAL - free space: / 13055MiB (5% inode=99%); [02:11:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.12, 24.86, 23.44 [02:13:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.22, 22.65, 22.31 [02:19:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.99, 23.70, 23.04 [02:21:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.68, 25.69, 23.85 [02:22:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:33:58] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 46265MiB (20% inode=99%); [02:35:40] RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 44915MiB (20% inode=99%); [02:43:58] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 22082MiB (9% inode=99%); [02:49:58] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 37307MiB (16% inode=99%); [02:51:40] PROBLEM - os161 Disk Space on os161 is CRITICAL: DISK CRITICAL - free space: / 12300MiB (5% inode=99%); [02:57:40] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 23053MiB (10% inode=99%); [02:59:40] PROBLEM - os161 Disk Space on os161 is CRITICAL: DISK CRITICAL - free space: / 5219MiB (2% inode=99%); [03:00:37] RECOVERY - db181 Backups SQL on db181 is OK: FILE_AGE OK: /var/log/sql-backup.log is 36 seconds old and 0 bytes [03:01:24] RECOVERY - db161 Backups SQL on db161 is OK: FILE_AGE OK: /var/log/sql-backup.log is 83 seconds old and 0 bytes [03:06:59] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.33, 21.93, 18.34 [03:08:20] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw152 [03:08:22] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.73, 18.40, 14.25 [03:10:19] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [03:10:22] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 20.26, 18.89, 14.93 [03:11:40] RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 28048MiB (12% inode=99%); [03:12:59] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.87, 23.65, 20.46 [03:14:43] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 5.84, 3.51, 2.48 [03:20:59] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.15, 19.75, 19.78 [03:24:43] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 2.99, 3.75, 3.33 [03:25:52] PROBLEM - www.publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'publictestwiki.com' expires in 7 day(s) (Sat 27 Jul 2024 03:16:29 AM GMT +0000). [03:26:51] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 22376MiB (10% inode=99%); [03:27:10] PROBLEM - publictestwiki.com - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'publictestwiki.com' expires in 7 day(s) (Sat 27 Jul 2024 03:16:29 AM GMT +0000). [03:28:43] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 2.52, 3.11, 3.16 [03:32:06] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw171 [03:32:12] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw171 [03:34:04] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [03:36:11] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [03:47:42] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importDump.php --wiki=chinafakewiki /home/void/chinafake.fandom.com-20240701-history.xml --username-prefix=wikia:chinafake (START) [03:47:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:51:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 11.67, 15.94, 22.66 [03:52:21] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 53401MiB (24% inode=99%); [03:55:35] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.28, 20.48, 19.43 [03:57:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.92, 19.89, 21.72 [03:57:33] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.88, 19.47, 19.19 [03:59:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.13, 21.19, 22.01 [04:01:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.80, 23.28, 22.65 [04:02:43] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 4.19, 3.12, 2.75 [04:04:42] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 2.72, 3.04, 2.77 [04:07:48] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importImages.php --wiki=chinafakewiki --search-resursively --skip-dupes --summary=[[phorge:T12281]] /home/void/images/ (START) [04:07:49] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importImages.php --wiki=chinafakewiki --search-resursively --skip-dupes --summary=[[phorge:T12281]] /home/void/images/ (END - exit=256) [04:07:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:08:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:08:15] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importImages.php --wiki=chinafakewiki /home/void/images --search-resursively --skip-dupes --summary=[[phorge:T12281]] (START) [04:08:16] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importImages.php --wiki=chinafakewiki /home/void/images --search-resursively --skip-dupes --summary=[[phorge:T12281]] (END - exit=256) [04:08:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:08:29] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importImages.php --wiki=chinafakewiki /home/void/images --search-resursively --skip-dupes --summary=[[phorge:T12281]] (START) [04:08:30] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importImages.php --wiki=chinafakewiki /home/void/images --search-resursively --skip-dupes --summary=[[phorge:T12281]] (END - exit=256) [04:08:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:08:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:09:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:09:22] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importImages.php --wiki=chinafakewiki /home/void/images --search-resursively --skip-dupes --summary=[[phorge:T12281]] (START) [04:09:22] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importImages.php --wiki=chinafakewiki /home/void/images --search-resursively --skip-dupes --summary=[[phorge:T12281]] (END - exit=256) [04:09:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:10:02] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importImages.php --wiki=chinafakewiki /home/void/images --search-recursively --skip-dupes --summary=[[phorge:T12281]] (START) [04:10:03] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importImages.php --wiki=chinafakewiki /home/void/images --search-recursively --skip-dupes --summary=[[phorge:T12281]] (END - exit=0) [04:10:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:10:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:10:27] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importImages.php --wiki=chinafakewiki /home/void/images --search-recursively --skip-dupes --summary=[[phorge:T12281]] (START) [04:10:45] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:10:55] Importing hard :( [04:11:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [04:13:19] o7 void [04:14:17] I also realized I forgot to do one of the imports in a screen. I'm now stuck here until it is done 🙃 [04:15:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 16.16, 21.17, 23.50 [04:16:10] This is why I use microsofts terminal app, lets me open a new tab and ssh back in when I do that [04:17:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.27, 24.17, 24.33 [04:21:58] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 23366MiB (10% inode=99%); [04:31:55] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 4.00, 3.64, 3.15 [04:33:49] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 2.62, 3.34, 3.10 [04:45:16] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.94, 3.77, 3.39 [04:49:43] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw152 [04:50:59] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 3.01, 3.29, 3.29 [04:51:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 11.36, 17.55, 23.84 [04:51:41] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [04:53:58] PROBLEM - os151 Disk Space on os151 is CRITICAL: DISK CRITICAL - free space: / 12739MiB (5% inode=99%); [04:54:48] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 4.12, 3.56, 3.39 [04:58:42] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 3.05, 3.85, 3.59 [04:59:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.95, 19.72, 21.67 [04:59:40] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 23838MiB (10% inode=99%); [05:00:43] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 2.12, 3.23, 3.39 [05:02:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:03:40] RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 24600MiB (11% inode=99%); [05:07:44] PROBLEM - prometheus151 Current Load on prometheus151 is CRITICAL: LOAD CRITICAL - total load average: 10.02, 4.97, 2.28 [05:09:40] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 1.61, 3.48, 2.05 [05:11:37] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.38, 2.42, 1.83 [05:12:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech RESOLVED: MediaWiki Exception Rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:13:58] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 48951MiB (22% inode=99%); [05:15:40] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 22564MiB (10% inode=99%); [05:17:40] RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 27467MiB (12% inode=99%); [05:18:43] PROBLEM - os151 Current Load on os151 is CRITICAL: LOAD CRITICAL - total load average: 6.52, 4.54, 3.56 [05:26:42] PROBLEM - os151 Current Load on os151 is WARNING: LOAD WARNING - total load average: 2.30, 3.72, 3.67 [05:30:42] RECOVERY - os151 Current Load on os151 is OK: LOAD OK - total load average: 1.49, 2.56, 3.20 [05:43:27] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importImages.php --wiki=chinafakewiki /home/void/images --search-recursively --skip-dupes --summary=[[phorge:T12281]] (END - exit=0) [05:43:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [06:33:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 11.78, 17.78, 23.22 [06:37:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 9.16, 13.75, 20.39 [06:41:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.65, 20.72, 21.67 [07:42:46] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 24302MiB (10% inode=99%); [07:44:44] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 25785MiB (11% inode=99%); [07:46:17] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/importDump.php --wiki=chinafakewiki /home/void/chinafake.fandom.com-20240701-history.xml --username-prefix=wikia:chinafake (END - exit=0) [07:46:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [07:56:25] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 23397MiB (10% inode=99%); [07:58:23] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 27067MiB (12% inode=99%); [07:59:00] [02mw-config] 07redbluegreenhat opened pull request 03#5617: Turn on debug logging for Math - 13https://github.com/miraheze/mw-config/pull/5617 [07:59:48] miraheze/mw-config - redbluegreenhat the build passed. [08:01:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.55, 22.21, 23.92 [08:01:42] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [08:03:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.45, 22.91, 23.96 [08:05:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.82, 22.54, 23.73 [08:05:03] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw152 [08:07:03] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [08:07:07] [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/513468340ed1...eacf64488455 [08:07:08] [02mw-config] 07redbluegreenhat 03eacf644 - Turn on debug logging for Math (#5617) [08:07:11] [02mw-config] 07redbluegreenhat closed pull request 03#5617: Turn on debug logging for Math - 13https://github.com/miraheze/mw-config/pull/5617 [08:07:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:07:43] !log [alex@mwtask181] starting deploy of {'pull': 'config', 'config': True} to all [08:07:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:08:01] miraheze/mw-config - redbluegreenhat the build passed. [08:08:05] !log [alex@mwtask181] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 22s [08:08:08] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=chinafakewiki (START) [08:08:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:08:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:08:49] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/maintenance/rebuildall.php --wiki=chinafakewiki (END - exit=256) [08:09:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:11:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.85, 24.06, 23.87 [08:16:59] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [08:19:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 17.45, 22.56, 23.71 [08:24:00] !log [@test151] starting deploy of {'config': True} to test151 [08:24:01] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [08:24:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:25:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.61, 22.51, 23.03 [08:25:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:27:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.44, 22.08, 22.79 [08:35:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 29.62, 24.06, 22.87 [08:36:53] !log [@mwtask171] starting deploy of {'config': True} to all [08:37:14] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 21s [08:37:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:38:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [08:39:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.95, 22.81, 22.70 [08:39:40] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 24050MiB (10% inode=99%); [08:42:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:43:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.18, 24.35, 23.24 [08:45:12] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures [08:51:40] RECOVERY - os161 Disk Space on os161 is OK: DISK OK - free space: / 24465MiB (11% inode=99%); [08:54:20] [02mw-config] 07redbluegreenhat pushed 031 commit to 03revert-5617-math-debug [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/f4d4427cd5bc [08:54:21] [02mw-config] 07redbluegreenhat 03f4d4427 - Revert "Turn on debug logging for Math (#5617)" [08:54:24] [02mw-config] 07redbluegreenhat created branch 03revert-5617-math-debug - 13https://github.com/miraheze/mw-config [08:54:37] [02mw-config] 07redbluegreenhat opened pull request 03#5618: Revert "Turn on debug logging for Math" - 13https://github.com/miraheze/mw-config/pull/5618 [08:54:51] [02mw-config] 07redbluegreenhat closed pull request 03#5618: Revert "Turn on debug logging for Math" - 13https://github.com/miraheze/mw-config/pull/5618 [08:54:54] [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/eacf64488455...5e50cf85d60a [08:54:55] [02mw-config] 07redbluegreenhat 035e50cf8 - Revert "Turn on debug logging for Math" (#5618) [08:54:58] [02mw-config] 07redbluegreenhat deleted branch 03revert-5617-math-debug [08:54:59] [02mw-config] 07redbluegreenhat deleted branch 03revert-5617-math-debug - 13https://github.com/miraheze/mw-config [08:55:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.87, 23.39, 23.93 [08:55:29] miraheze/mw-config - redbluegreenhat the build passed. [08:55:40] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 23396MiB (10% inode=99%); [08:55:48] miraheze/mw-config - redbluegreenhat the build passed. [08:57:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 23.64, 24.05, 24.15 [09:01:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.20, 23.56, 23.93 [09:02:12] !log [@mwtask181] starting deploy of {'config': True} to all [09:02:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:02:31] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 18s [09:02:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:05:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.70, 25.61, 24.63 [09:07:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 15.42, 22.44, 23.64 [09:07:11] !log [@mwtask171] starting deploy of {'config': True} to all [09:07:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:07:22] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 10s [09:07:29] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:07:34] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 23361MiB (10% inode=99%); [09:09:32] RECOVERY - os151 Disk Space on os151 is OK: DISK OK - free space: / 24626MiB (11% inode=99%); [09:13:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.64, 21.05, 22.15 [09:13:25] PROBLEM - os151 Disk Space on os151 is WARNING: DISK WARNING - free space: / 23900MiB (10% inode=99%); [09:23:31] !log [@test151] starting deploy of {'config': True} to test151 [09:23:32] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [09:23:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:23:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:30:27] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [09:35:52] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw152 [09:37:50] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [10:13:20] PROBLEM - wiki.gtnewhorizons.com - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.gtnewhorizons.com' expires in 15 day(s) (Sun 04 Aug 2024 10:03:23 AM GMT +0000). [10:13:33] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/d0b2532d2a05...0b0f71e81ddb [10:13:36] [02ssl] 07WikiTideSSLBot 030b0f71e - Bot: Update SSL cert for wiki.gtnewhorizons.com [10:15:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 16.56, 21.60, 23.57 [10:17:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.62, 23.00, 23.84 [10:23:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.96, 22.86, 23.77 [10:29:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 11.37, 15.38, 20.13 [10:29:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 13.69, 18.90, 23.41 [10:30:13] PROBLEM - www.dariawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'dariawiki.org' expires in 7 day(s) (Sat 27 Jul 2024 10:01:56 AM GMT +0000). [10:30:24] PROBLEM - dariawiki.org - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'dariawiki.org' expires in 7 day(s) (Sat 27 Jul 2024 10:01:56 AM GMT +0000). [10:33:10] PROBLEM - xomnipedia.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'xomnipedia.org' expires in 15 day(s) (Sun 04 Aug 2024 10:20:28 AM GMT +0000). [10:33:21] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0b0f71e81ddb...60defefa91b5 [10:33:24] [02ssl] 07WikiTideSSLBot 0360defef - Bot: Update SSL cert for xomnipedia.org [10:37:02] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 11.25, 14.45, 19.59 [10:43:24] PROBLEM - mw182 MediaWiki Rendering on mw182 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:43:56] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw182 [10:45:23] RECOVERY - mw182 MediaWiki Rendering on mw182 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 5.113 second response time [10:45:53] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [10:47:35] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [10:52:17] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.38, 17.92, 16.09 [10:54:15] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 18.96, 18.54, 16.56 [10:59:02] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 19.29, 21.78, 20.35 [11:01:00] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.94, 19.89, 19.82 [11:02:51] RECOVERY - xomnipedia.org - LetsEncrypt on sslhost is OK: OK - Certificate 'xomnipedia.org' will expire on Thu 17 Oct 2024 09:33:16 AM GMT +0000. [11:12:01] RECOVERY - wiki.gtnewhorizons.com - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.gtnewhorizons.com' will expire on Thu 17 Oct 2024 09:13:26 AM GMT +0000. [11:13:55] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 47 seconds ago with 0 failures [11:21:31] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 22.59, 20.25, 19.86 [11:25:26] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 18.16, 19.43, 19.65 [11:26:25] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [11:28:25] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [195 system event log (SEL) entries present] [11:47:40] PROBLEM - os161 Disk Space on os161 is CRITICAL: DISK CRITICAL - free space: / 12804MiB (5% inode=99%); [12:01:40] PROBLEM - os161 Disk Space on os161 is WARNING: DISK WARNING - free space: / 15661MiB (7% inode=99%); [12:42:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [14:09:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.92, 20.40, 23.89 [14:13:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.72, 21.39, 23.41 [14:15:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.25, 21.99, 23.46 [14:17:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.16, 22.53, 23.45 [14:19:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.41, 21.80, 23.07 [14:22:13] PROBLEM - wiki.qadrishattari.xyz - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.qadrishattari.xyz All nameservers failed to answer the query. [14:23:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.04, 22.03, 22.71 [14:27:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.70, 21.74, 22.64 [14:28:45] PROBLEM - wiki.qadrishattari.xyz - LetsEncrypt on sslhost is CRITICAL: Temporary failure in name resolutionHTTP CRITICAL - Unable to open TCP socket [14:31:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.45, 22.58, 22.73 [14:33:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 11.69, 19.12, 21.50 [14:35:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 11.11, 16.09, 20.07 [15:03:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.18, 21.56, 17.69 [15:05:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.31, 24.13, 20.61 [15:14:39] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.60, 20.98, 19.66 [15:15:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.88, 23.92, 21.90 [15:15:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.54, 23.84, 22.64 [15:15:55] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:17:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.15, 23.98, 22.14 [15:18:38] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.97, 19.43, 19.36 [15:19:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.08, 22.45, 21.81 [15:19:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.44, 23.89, 22.91 [15:21:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.11, 23.82, 22.38 [15:21:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.84, 23.08, 22.72 [15:23:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 29.45, 25.45, 23.63 [15:24:37] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.77, 22.20, 20.35 [15:26:36] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.39, 21.71, 20.45 [15:27:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.41, 23.73, 23.01 [15:28:35] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.32, 20.07, 20.02 [15:31:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.10, 23.98, 23.27 [15:33:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.18, 23.68, 23.26 [15:35:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.50, 25.38, 23.95 [15:37:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.86, 22.29, 23.54 [15:39:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.73, 23.60, 23.83 [15:41:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.39, 23.33, 23.71 [15:41:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 12.52, 19.24, 22.21 [15:43:31] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.70, 20.44, 20.02 [15:44:24] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [15:45:02] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 12.98, 15.04, 19.73 [15:45:30] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.71, 19.72, 19.81 [15:53:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.85, 21.52, 20.33 [15:55:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.75, 22.78, 22.70 [15:57:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.14, 23.46, 23.01 [15:59:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.52, 23.59, 23.09 [16:01:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.59, 22.78, 22.90 [16:03:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.83, 24.63, 23.54 [16:04:25] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.39, 20.65, 18.32 [16:07:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:08:24] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.92, 22.27, 19.54 [16:14:22] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.77, 22.07, 20.22 [16:16:22] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.69, 21.52, 20.26 [16:20:21] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.55, 24.49, 21.71 [16:26:18] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw152 [16:28:17] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [16:32:18] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.30, 23.73, 23.11 [16:36:17] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.76, 23.75, 23.20 [16:38:16] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.24, 22.36, 22.72 [16:40:16] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.64, 23.09, 22.93 [16:42:15] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.10, 22.21, 22.60 [16:46:14] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.96, 22.14, 22.41 [16:46:29] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [16:48:14] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.46, 22.22, 22.40 [16:50:13] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.55, 23.95, 23.00 [17:02:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: There has been a rise in the MediaWiki exception rate https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [17:03:04] Erm [17:03:08] Prometheus down? [17:03:33] PROBLEM - prometheus151 Current Load on prometheus151 is WARNING: LOAD WARNING - total load average: 3.56, 2.53, 1.10 [17:03:56] the greek gods got angry [17:04:01] they chained him to the rock [17:05:33] RECOVERY - prometheus151 Current Load on prometheus151 is OK: LOAD OK - total load average: 0.85, 1.86, 1.03 [17:07:25]