[00:00:48] [02python-functions] 07dependabot[bot] pushed 031 commit to 03dependabot/pip/dot-github/setuptools-71.0.0 [+0/-0/±1] 13https://github.com/miraheze/python-functions/commit/445f8f664c1c [00:00:50] [02python-functions] 07dependabot[bot] 03445f8f6 - Bump setuptools from 70.2.0 to 71.0.0 in /.github [00:00:53] [02python-functions] 07dependabot[bot] created branch 03dependabot/pip/dot-github/setuptools-71.0.0 - 13https://github.com/miraheze/python-functions [00:00:54] [02python-functions] 07dependabot[bot] labeled pull request 03#42: Bump setuptools from 70.2.0 to 71.0.0 in /.github - 13https://github.com/miraheze/python-functions/pull/42 [00:00:57] [02python-functions] 07dependabot[bot] labeled pull request 03#42: Bump setuptools from 70.2.0 to 71.0.0 in /.github - 13https://github.com/miraheze/python-functions/pull/42 [00:00:58] [02python-functions] 07dependabot[bot] opened pull request 03#42: Bump setuptools from 70.2.0 to 71.0.0 in /.github - 13https://github.com/miraheze/python-functions/pull/42 [00:00:59] [02python-functions] 07coderabbitai[bot] commented on pull request 03#42: Bump setuptools from 70.2.0 to 71.0.0 in /.github - 13https://github.com/miraheze/python-functions/pull/42#issuecomment-2234809484 [00:01:48] [02RemovePII] 07Universal-Omega closed pull request 03#88: Fix support for 1.42 - 13https://github.com/miraheze/RemovePII/pull/88 [00:01:50] [02RemovePII] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±2] 13https://github.com/miraheze/RemovePII/compare/78a9a2145109...18114922594f [00:01:51] [02RemovePII] 07Universal-Omega 031811492 - Fix support for 1.42 (#88) [00:01:54] [02RemovePII] 07Universal-Omega deleted branch 031.42 [00:01:56] [02RemovePII] 07Universal-Omega deleted branch 031.42 - 13https://github.com/miraheze/RemovePII [00:02:09] miraheze/RemovePII - Universal-Omega the build has errored. [00:02:13] !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [00:02:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:02:27] !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 14s [00:02:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:02:57] @bluemoon0332 FYI fixed fatals in RemovePII and VanishUser [00:03:49] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.54, 23.12, 22.10 [00:05:24] miraheze/python-functions - dependabot[bot] the build passed. [00:07:11] [02puppet] 07The-Voidwalker pushed 032 commits to 03patch-mwpatches [+2/-0/±3] 13https://github.com/miraheze/puppet/compare/d69348676234...b4b2162f923f [00:07:14] [02puppet] 07The-Voidwalker 038563360 - mwdeploy.py: add support for local patches [00:07:17] [02puppet] 07The-Voidwalker 03b4b2162 - quotes :( [00:07:18] [02puppet] 07The-Voidwalker synchronize pull request 03#3885: mwdeploy.py: add support for local patches - 13https://github.com/miraheze/puppet/pull/3885 [00:07:46] !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [00:07:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:08:06] !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 20s [00:08:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:08:30] miraheze/puppet - The-Voidwalker the build has errored. [00:09:23] miraheze/RemovePII - Universal-Omega the build has errored. [00:09:29] [02CreateWiki] 07Universal-Omega pushed 031 commit to 03ObjectCacheFactory [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/commit/6d1a49aeb1b4 [00:09:31] [02CreateWiki] 07Universal-Omega 036d1a49a - Use ObjectCacheFactory [00:09:33] [02CreateWiki] 07Universal-Omega created branch 03ObjectCacheFactory - 13https://github.com/miraheze/CreateWiki [00:09:34] [02CreateWiki] 07Universal-Omega opened pull request 03#543: Use ObjectCacheFactory - 13https://github.com/miraheze/CreateWiki/pull/543 [00:09:41] [02CreateWiki] 07coderabbitai[bot] commented on pull request 03#543: Use ObjectCacheFactory - 13https://github.com/miraheze/CreateWiki/pull/543#issuecomment-2234838227 [00:10:14] [02CreateWiki] 07Universal-Omega pushed 031 commit to 03ObjectCacheFactory [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/6d1a49aeb1b4...afe1b40e1842 [00:10:16] [02CreateWiki] 07Universal-Omega 03afe1b40 - Bump [00:10:19] [02CreateWiki] 07Universal-Omega synchronize pull request 03#543: Use ObjectCacheFactory - 13https://github.com/miraheze/CreateWiki/pull/543 [00:10:58] [02CreateWiki] 07Universal-Omega synchronize pull request 03#543: Use ObjectCacheFactory - 13https://github.com/miraheze/CreateWiki/pull/543 [00:11:00] [02CreateWiki] 07Universal-Omega pushed 031 commit to 03ObjectCacheFactory [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/afe1b40e1842...d3cee26bbc72 [00:11:01] [02CreateWiki] 07Universal-Omega 03d3cee26 - - [00:12:46] [02CreateWiki] 07Universal-Omega reviewed pull request 03#543 commit - 13https://github.com/miraheze/CreateWiki/pull/543#discussion_r1681915409 [00:15:16] miraheze/CreateWiki - Universal-Omega the build has errored. [00:17:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.17, 23.87, 23.81 [00:21:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.15, 21.90, 23.98 [00:23:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.46, 22.35, 23.86 [00:24:30] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [00:24:31] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [00:24:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:24:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:24:47] !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 [00:24:48] !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [00:24:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:24:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:27:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.77, 22.52, 23.60 [00:33:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.47, 20.95, 21.37 [00:35:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.84, 23.42, 23.30 [00:35:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.36, 20.68, 21.21 [00:37:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.26, 23.14, 23.25 [00:39:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 23.55, 24.08, 23.63 [00:42:16] !log [void@db171] backed up metawiki abuse filter tables [00:42:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:43:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.10, 23.37, 23.54 [00:43:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.70, 19.20, 20.34 [00:44:34] !log [void@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/AbuseFilter/maintenance/MigrateActorsAF.php --wiki=metawiki (END - exit=0) [00:44:41] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:47:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.78, 24.39, 23.80 [00:47:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.65, 21.94, 21.16 [00:51:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.13, 23.20, 23.50 [00:52:34] !log [void@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/AbuseFilter/maintenance/MigrateActorsAF.php (START) [00:52:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:55:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.79, 23.30, 23.39 [00:57:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.02, 22.18, 23.01 [01:01:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.32, 19.29, 20.19 [01:03:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.84, 22.39, 22.61 [01:04:44] !log [macfan@mwtask181] sudo -u www-data php /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/CreateWiki/maintenance/setContainersAccess.php --wiki=staffwiki (END - exit=0) [01:04:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:09:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.94, 23.89, 23.38 [01:11:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 29.40, 25.94, 24.19 [01:12:46] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.30, 21.03, 20.64 [01:18:45] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.31, 23.20, 21.65 [01:22:44] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.93, 23.09, 22.13 [01:24:43] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.32, 23.82, 22.51 [01:27:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 13.82, 20.76, 23.11 [01:31:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.26, 22.23, 22.94 [01:38:40] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.69, 23.57, 23.79 [01:41:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.54, 23.05, 23.73 [01:43:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.14, 23.26, 23.70 [01:45:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.79, 22.83, 23.50 [01:48:37] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.55, 21.44, 22.30 [01:49:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.92, 23.73, 23.71 [01:50:37] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.40, 20.63, 21.89 [01:51:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.17, 22.31, 23.17 [01:55:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.10, 21.97, 22.68 [01:56:35] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.38, 17.43, 20.03 [01:57:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.04, 21.14, 22.30 [02:05:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.98, 22.42, 21.81 [02:07:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.61, 22.68, 21.99 [02:07:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.60, 21.50, 23.67 [02:09:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.97, 23.32, 22.30 [02:11:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.51, 22.15, 21.98 [02:13:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.06, 22.96, 22.28 [02:13:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 25.39, 22.10, 23.00 [02:15:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.78, 22.35, 22.16 [02:17:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.44, 22.74, 22.31 [02:18:40] [02mw-config] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d471e2d95448...44e5e4c4993a [02:18:41] [02mw-config] 07The-Voidwalker 0344e5e4c - add wgTabberNeueEnableAnimation to ManageWikiSettings [02:19:36] miraheze/mw-config - The-Voidwalker the build has errored. [02:20:21] [02mw-config] 07The-Voidwalker pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/44e5e4c4993a...e80f5589aa20 [02:20:24] [02mw-config] 07The-Voidwalker 03e80f558 - add missing section property [02:21:15] miraheze/mw-config - The-Voidwalker the build passed. [02:21:28] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.79, 21.56, 19.96 [02:22:31] !log [@mwtask181] starting deploy of {'config': True} to all [02:22:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:22:47] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 16s [02:22:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:23:28] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.85, 22.10, 20.32 [02:23:36] !log [@test151] starting deploy of {'config': True} to test151 [02:23:37] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [02:23:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:24:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:27:27] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.60, 22.46, 20.97 [02:29:26] !log [void@puppet181] Upgraded packages linux-libc-dev on bots171 [02:29:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:29:39] !log [void@puppet181] Upgraded packages linux-libc-dev on eventgate181 [02:29:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:29:51] !log [void@puppet181] Upgraded packages linux-libc-dev on jobchron171 [02:29:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:30:03] !log [void@puppet181] Upgraded packages linux-libc-dev on db171 [02:30:04] RECOVERY - eventgate181 APT on eventgate181 is OK: APT OK: 45 packages available for upgrade (0 critical updates). [02:30:11] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:30:15] !log [void@puppet181] Upgraded packages linux-libc-dev on cloud16 [02:30:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:30:21] RECOVERY - bots171 APT on bots171 is OK: APT OK: 63 packages available for upgrade (0 critical updates). [02:30:27] !log [void@puppet181] Upgraded packages linux-libc-dev on cp37 [02:30:29] RECOVERY - db171 APT on db171 is OK: APT OK: 59 packages available for upgrade (0 critical updates). [02:30:35] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:30:38] RECOVERY - cp37 APT on cp37 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [02:30:39] !log [void@puppet181] Upgraded packages linux-libc-dev on cloud17 [02:30:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:30:50] !log [void@puppet181] Upgraded packages linux-libc-dev on cp36 [02:30:56] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:31:02] !log [void@puppet181] Upgraded packages linux-libc-dev on db161 [02:31:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:31:10] RECOVERY - jobchron171 APT on jobchron171 is OK: APT OK: 62 packages available for upgrade (0 critical updates). [02:31:14] !log [void@puppet181] Upgraded packages linux-libc-dev on graylog161 [02:31:22] RECOVERY - cloud17 APT on cloud17 is OK: APT OK: 100 packages available for upgrade (0 critical updates). [02:31:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:31:27] RECOVERY - cloud16 APT on cloud16 is OK: APT OK: 100 packages available for upgrade (0 critical updates). [02:31:27] !log [void@puppet181] Upgraded packages linux-libc-dev on ldap171 [02:31:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:31:40] !log [void@puppet181] Upgraded packages linux-libc-dev on db182 [02:31:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:31:52] !log [void@puppet181] Upgraded packages linux-libc-dev on cloud18 [02:31:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:32:05] !log [void@puppet181] Upgraded packages linux-libc-dev on db181 [02:32:06] RECOVERY - cloud18 APT on cloud18 is OK: APT OK: 101 packages available for upgrade (0 critical updates). [02:32:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:32:18] !log [void@puppet181] Upgraded packages linux-libc-dev on kafka181 [02:32:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:32:26] RECOVERY - cp36 APT on cp36 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [02:32:32] !log [void@puppet181] Upgraded packages linux-libc-dev on matomo151 [02:32:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:32:46] !log [void@puppet181] Upgraded packages linux-libc-dev on graphite151 [02:32:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:32:58] RECOVERY - db181 APT on db181 is OK: APT OK: 60 packages available for upgrade (0 critical updates). [02:33:00] RECOVERY - db161 APT on db161 is OK: APT OK: 59 packages available for upgrade (0 critical updates). [02:33:00] !log [void@puppet181] Upgraded packages linux-libc-dev on db151 [02:33:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:33:11] RECOVERY - db182 APT on db182 is OK: APT OK: 60 packages available for upgrade (0 critical updates). [02:33:15] RECOVERY - graylog161 APT on graylog161 is OK: APT OK: 56 packages available for upgrade (0 critical updates). [02:33:16] !log [void@puppet181] Upgraded packages linux-libc-dev on cloud15 [02:33:21] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:33:22] RECOVERY - ldap171 APT on ldap171 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [02:33:24] RECOVERY - matomo151 APT on matomo151 is OK: APT OK: 63 packages available for upgrade (0 critical updates). [02:33:27] !log [void@puppet181] Upgraded packages linux-libc-dev on bast161 [02:33:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:33:35] RECOVERY - kafka181 APT on kafka181 is OK: APT OK: 38 packages available for upgrade (0 critical updates). [02:33:42] !log [void@puppet181] Upgraded packages linux-libc-dev on cp26 [02:33:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:33:56] !log [void@puppet181] Upgraded packages linux-libc-dev on bast181 [02:34:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:34:08] !log [void@puppet181] Upgraded packages linux-libc-dev on mem161 [02:34:13] RECOVERY - mem161 APT on mem161 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [02:34:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:34:17] RECOVERY - graphite151 APT on graphite151 is OK: APT OK: 40 packages available for upgrade (0 critical updates). [02:34:21] !log [void@puppet181] Upgraded packages linux-libc-dev on mon181 [02:34:29] RECOVERY - bast181 APT on bast181 is OK: APT OK: 53 packages available for upgrade (0 critical updates). [02:34:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:34:32] RECOVERY - cloud15 APT on cloud15 is OK: APT OK: 108 packages available for upgrade (0 critical updates). [02:34:35] !log [void@puppet181] Upgraded packages linux-libc-dev on cp41 [02:34:38] RECOVERY - db151 APT on db151 is OK: APT OK: 60 packages available for upgrade (0 critical updates). [02:34:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:34:50] !log [void@puppet181] Upgraded packages linux-libc-dev on mem151 [02:34:54] RECOVERY - cp26 APT on cp26 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [02:34:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:35:02] !log [void@puppet181] Upgraded packages linux-libc-dev on mw161 [02:35:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:35:15] !log [void@puppet181] Upgraded packages linux-libc-dev on mw172 [02:35:17] RECOVERY - bast161 APT on bast161 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [02:35:21] RECOVERY - mem151 APT on mem151 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [02:35:22] RECOVERY - mw161 APT on mw161 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [02:35:23] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:35:28] !log [void@puppet181] Upgraded packages linux-libc-dev on mwtask171 [02:35:32] RECOVERY - mwtask171 APT on mwtask171 is OK: APT OK: 63 packages available for upgrade (0 critical updates). [02:35:33] RECOVERY - mw172 APT on mw172 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [02:35:33] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:35:41] !log [void@puppet181] Upgraded packages linux-libc-dev on os162 [02:35:49] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:35:54] !log [void@puppet181] Upgraded packages linux-libc-dev on ns1 [02:36:01] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:36:04] RECOVERY - mon181 APT on mon181 is OK: APT OK: 58 packages available for upgrade (0 critical updates). [02:36:07] !log [void@puppet181] Upgraded packages linux-libc-dev on mw162 [02:36:08] RECOVERY - cp41 APT on cp41 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [02:36:14] RECOVERY - ns1 APT on ns1 is OK: APT OK: 51 packages available for upgrade (0 critical updates). [02:36:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:36:20] !log [void@puppet181] Upgraded packages linux-libc-dev on mw171 [02:36:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:36:33] !log [void@puppet181] Upgraded packages linux-libc-dev on phorge171 [02:36:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:36:45] !log [void@puppet181] Upgraded packages linux-libc-dev on os161 [02:36:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:00] !log [void@puppet181] Upgraded packages linux-libc-dev on mwtask181 [02:37:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:16] RECOVERY - mwtask181 APT on mwtask181 is OK: APT OK: 63 packages available for upgrade (0 critical updates). [02:37:16] !log [void@puppet181] Upgraded packages linux-libc-dev on mw152 [02:37:21] RECOVERY - mw171 APT on mw171 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [02:37:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [02:37:27] RECOVERY - os162 APT on os162 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [02:37:28] !log [@mwtask171] starting deploy of {'config': True} to all [02:37:30] !log [void@puppet181] Upgraded packages linux-libc-dev on mw182 [02:37:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:37] void be cooking rn [02:37:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:40] RECOVERY - os161 APT on os161 is OK: APT OK: 53 packages available for upgrade (0 critical updates). [02:37:41] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 12s [02:37:46] !log [void@puppet181] Upgraded packages linux-libc-dev on os151 [02:37:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:50] RECOVERY - mw162 APT on mw162 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [02:37:52] RECOVERY - phorge171 APT on phorge171 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [02:37:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:37:56] security updates be like [02:38:01] !log [void@puppet181] Upgraded packages linux-libc-dev on mw151 [02:38:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:38:18] !log [void@puppet181] Upgraded packages linux-libc-dev on mw181 [02:38:20] RECOVERY - mw152 APT on mw152 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [02:38:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:38:33] !log [void@puppet181] Upgraded packages linux-libc-dev on cp51 [02:38:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:38:49] !log [void@puppet181] Upgraded packages linux-libc-dev on prometheus151 [02:38:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:38:57] RECOVERY - os151 APT on os151 is OK: APT OK: 54 packages available for upgrade (0 critical updates). [02:39:02] !log [void@puppet181] Upgraded packages linux-libc-dev on puppet181 [02:39:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.24, 22.60, 23.96 [02:39:03] RECOVERY - puppet181 APT on puppet181 is OK: APT OK: 61 packages available for upgrade (0 critical updates). [02:39:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:39:14] !log [void@puppet181] Upgraded packages linux-libc-dev on reports171 [02:39:14] RECOVERY - mw182 APT on mw182 is OK: APT OK: 66 packages available for upgrade (0 critical updates). [02:39:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:39:24] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 20.01, 20.08, 20.39 [02:39:28] !log [void@puppet181] Upgraded packages linux-libc-dev on rdb151 [02:39:36] RECOVERY - mw181 APT on mw181 is OK: APT OK: 66 packages available for upgrade (0 critical updates). [02:39:37] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:40:00] !log [void@puppet181] Upgraded packages linux-libc-dev on cp27 [02:40:03] RECOVERY - mw151 APT on mw151 is OK: APT OK: 65 packages available for upgrade (0 critical updates). [02:40:04] RECOVERY - cp51 APT on cp51 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [02:40:06] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:40:14] !log [void@puppet181] Upgraded packages linux-libc-dev on swiftobject161 [02:40:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:40:22] RECOVERY - rdb151 APT on rdb151 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [02:40:31] !log [void@puppet181] Upgraded packages linux-libc-dev on swiftac171 [02:40:39] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:40:40] RECOVERY - prometheus151 APT on prometheus151 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [02:40:44] !log [void@puppet181] Upgraded packages linux-libc-dev on swiftproxy161 [02:40:46] RECOVERY - reports171 APT on reports171 is OK: APT OK: 64 packages available for upgrade (0 critical updates). [02:40:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:40:58] RECOVERY - swiftac171 APT on swiftac171 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [02:40:58] !log [void@puppet181] Upgraded packages linux-libc-dev on swiftobject171 [02:41:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.32, 24.10, 24.32 [02:41:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:41:13] !log [void@puppet181] Upgraded packages linux-libc-dev on swiftobject181 [02:41:16] RECOVERY - cp27 APT on cp27 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [02:41:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:41:30] RECOVERY - swiftobject181 APT on swiftobject181 is OK: APT OK: 53 packages available for upgrade (0 critical updates). [02:41:31] !log [void@puppet181] Upgraded packages linux-libc-dev on swiftobject151 [02:41:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:41:47] !log [void@puppet181] Upgraded packages linux-libc-dev on test151 [02:41:49] !log [void@puppet181] varnishadm ban "obj.http.xkey ~ ^islegoblin.wiki" on all cp [02:41:53] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:41:53] RECOVERY - swiftobject161 APT on swiftobject161 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [02:41:59] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:42:03] !log [void@puppet181] Upgraded packages linux-libc-dev on swiftproxy171 [02:42:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [02:42:28] RECOVERY - swiftobject151 APT on swiftobject151 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [02:42:30] RECOVERY - swiftproxy161 APT on swiftproxy161 is OK: APT OK: 41 packages available for upgrade (0 critical updates). [02:42:46] RECOVERY - swiftobject171 APT on swiftobject171 is OK: APT OK: 52 packages available for upgrade (0 critical updates). [02:43:35] RECOVERY - swiftproxy171 APT on swiftproxy171 is OK: APT OK: 55 packages available for upgrade (0 critical updates). [02:43:42] RECOVERY - test151 APT on test151 is OK: APT OK: 75 packages available for upgrade (0 critical updates). [02:59:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.38, 23.12, 23.75 [03:01:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.47, 23.34, 23.71 [03:01:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.77, 20.82, 23.31 [03:01:10] RECOVERY - db171 Backups SQL on db171 is OK: FILE_AGE OK: /var/log/sql-backup.log is 69 seconds old and 0 bytes [03:03:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.56, 23.13, 23.58 [03:04:31] !log [void@mwtask181] sudo -u www-data /usr/local/bin/foreachwikiindblist /srv/mediawiki/cache/databases.json /srv/mediawiki/1.42/maintenance/run.php /srv/mediawiki/1.42/extensions/AbuseFilter/maintenance/MigrateActorsAF.php (END - exit=0) [03:05:30] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:06:34] PROBLEM - cloud15 Puppet on cloud15 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 2 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[ulogd2] [03:13:02] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 12.82, 16.00, 19.74 [03:19:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.16, 21.95, 21.58 [03:20:11] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.41, 20.05, 19.14 [03:21:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.67, 21.63, 21.47 [03:22:20] !log [void@db161] manually set af_actor in abuse_filter table due to missing actor information [03:22:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:22:52] Ahh, af_actor set for allthetropeswiki in ^ log [03:23:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.31, 22.27, 21.68 [03:23:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.22, 21.02, 20.40 [03:24:10] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.46, 21.48, 19.91 [03:25:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 17.53, 20.68, 21.19 [03:26:10] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.53, 22.38, 20.45 [03:27:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.65, 22.30, 21.20 [03:33:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.21, 18.67, 20.11 [03:34:39] RECOVERY - cloud15 Puppet on cloud15 is OK: OK: Puppet is currently enabled, last run 51 seconds ago with 0 failures [03:37:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.51, 22.13, 21.28 [03:40:06] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 19.39, 20.14, 20.38 [03:43:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.81, 23.14, 21.92 [03:45:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.18, 23.03, 22.03 [03:47:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.74, 22.98, 22.08 [03:47:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.10, 22.52, 21.77 [03:49:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.66, 22.17, 21.91 [03:49:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.20, 22.29, 21.79 [03:53:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.10, 23.76, 22.45 [03:56:01] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.56, 20.51, 20.05 [03:58:01] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.45, 19.99, 19.93 [03:59:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.71, 22.66, 22.60 [04:03:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 17.54, 18.77, 20.26 [04:07:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.35, 20.31, 20.69 [04:09:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 15.29, 18.98, 20.19 [04:13:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.72, 23.47, 22.55 [04:15:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.43, 23.44, 22.66 [04:17:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.92, 24.40, 23.09 [04:19:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 21.39, 23.44, 22.92 [04:22:54] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.61, 20.20, 19.21 [04:23:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 28.82, 24.98, 23.55 [04:24:53] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.14, 22.98, 20.35 [04:35:12] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.21, 22.95, 20.84 [04:36:50] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.08, 23.23, 22.45 [04:42:49] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.45, 22.89, 22.56 [04:43:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.30, 23.62, 22.01 [04:44:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.06, 22.01, 22.29 [04:45:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 29.63, 25.87, 23.03 [04:51:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.63, 22.86, 22.71 [04:52:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [04:53:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.26, 24.07, 23.15 [04:56:45] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.78, 22.17, 21.71 [04:57:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.35, 22.24, 22.74 [04:59:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.22, 23.61, 23.19 [05:00:44] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.03, 22.66, 22.13 [05:01:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 15.90, 20.95, 22.29 [05:02:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [05:07:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 11.56, 17.22, 22.73 [05:07:39] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [05:08:19] [02python-functions] 07github-merge-queue[bot] pushed 031 commit to 03gh-readonly-queue/master/pr-42-a50b31def39ecfcc0c3746bec9cb87bd58631186 [+0/-0/±1] 13https://github.com/miraheze/python-functions/commit/ab2eb343f9c3 [05:08:22] [02python-functions] 07dependabot[bot] 03ab2eb34 - Bump setuptools from 70.2.0 to 71.0.0 in /.github (#42) [05:08:24] [02python-functions] 07github-merge-queue[bot] created branch 03gh-readonly-queue/master/pr-42-a50b31def39ecfcc0c3746bec9cb87bd58631186 - 13https://github.com/miraheze/python-functions [05:09:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.23, 16.26, 19.66 [05:09:39] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [184 system event log (SEL) entries present] [05:12:41] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.69, 18.36, 20.24 [05:13:57] [02python-functions] 07github-merge-queue[bot] pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/python-functions/compare/a50b31def39e...ab2eb343f9c3 [05:13:58] [02python-functions] 07RhinosF1 closed pull request 03#42: Bump setuptools from 70.2.0 to 71.0.0 in /.github - 13https://github.com/miraheze/python-functions/pull/42 [05:14:00] [02python-functions] 07github-merge-queue[bot] deleted branch 03gh-readonly-queue/master/pr-42-a50b31def39ecfcc0c3746bec9cb87bd58631186 [05:14:01] [02python-functions] 07github-merge-queue[bot] deleted branch 03gh-readonly-queue/master/pr-42-a50b31def39ecfcc0c3746bec9cb87bd58631186 - 13https://github.com/miraheze/python-functions [05:14:04] [02python-functions] 07RhinosF1 deleted branch 03dependabot/pip/dot-github/setuptools-71.0.0 [05:14:06] [02python-functions] 07RhinosF1 deleted branch 03dependabot/pip/dot-github/setuptools-71.0.0 - 13https://github.com/miraheze/python-functions [05:14:07] [02python-functions] 07dependabot[bot] edited pull request 03#37: Bump setuptools from 70.2.0 to 70.3.0 in /.github - 13https://github.com/miraheze/python-functions/pull/37 [05:14:46] [02python-functions] 07dependabot[bot] edited pull request 03#37: Bump setuptools from 70.2.0 to 70.3.0 in /.github - 13https://github.com/miraheze/python-functions/pull/37 [05:14:47] [02python-functions] 07dependabot[bot] pushed 031 commit to 03dependabot/pip/dot-github/setuptools-71.0.1 [+0/-0/±1] 13https://github.com/miraheze/python-functions/commit/d35c2083707d [05:14:48] [02python-functions] 07dependabot[bot] 03d35c208 - Bump setuptools from 71.0.0 to 71.0.1 in /.github [05:14:49] [02python-functions] 07dependabot[bot] created branch 03dependabot/pip/dot-github/setuptools-71.0.1 - 13https://github.com/miraheze/python-functions [05:14:52] [02python-functions] 07dependabot[bot] opened pull request 03#43: Bump setuptools from 71.0.0 to 71.0.1 in /.github - 13https://github.com/miraheze/python-functions/pull/43 [05:14:53] [02python-functions] 07dependabot[bot] labeled pull request 03#43: Bump setuptools from 71.0.0 to 71.0.1 in /.github - 13https://github.com/miraheze/python-functions/pull/43 [05:14:54] [02python-functions] 07dependabot[bot] labeled pull request 03#43: Bump setuptools from 71.0.0 to 71.0.1 in /.github - 13https://github.com/miraheze/python-functions/pull/43 [05:14:56] [02python-functions] 07coderabbitai[bot] commented on pull request 03#43: Bump setuptools from 71.0.0 to 71.0.1 in /.github - 13https://github.com/miraheze/python-functions/pull/43#issuecomment-2235370672 [05:15:02] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 15.65, 16.42, 20.04 [05:19:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.54, 21.31, 21.30 [05:19:30] miraheze/python-functions - dependabot[bot] the build passed. [05:19:51] miraheze/python-functions - github-merge-queue[bot] the build passed. [05:21:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.18, 20.57, 19.83 [05:23:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.94, 22.46, 21.63 [05:24:37] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.93, 21.39, 20.74 [05:24:47] [02python-functions] 07github-merge-queue[bot] pushed 031 commit to 03gh-readonly-queue/master/pr-43-ab2eb343f9c3f3e91096ab3fba77e4a2862f35a6 [+0/-0/±1] 13https://github.com/miraheze/python-functions/commit/23766f717f48 [05:24:50] [02python-functions] 07dependabot[bot] 0323766f7 - Bump setuptools from 71.0.0 to 71.0.1 in /.github (#43) [05:24:51] [02python-functions] 07github-merge-queue[bot] created branch 03gh-readonly-queue/master/pr-43-ab2eb343f9c3f3e91096ab3fba77e4a2862f35a6 - 13https://github.com/miraheze/python-functions [05:25:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.34, 21.60, 20.41 [05:25:56] [02python-functions] 07RhinosF1 closed pull request 03#37: Bump setuptools from 70.2.0 to 70.3.0 in /.github - 13https://github.com/miraheze/python-functions/pull/37 [05:25:58] [02python-functions] 07dependabot[bot] commented on pull request 03#37: Bump setuptools from 70.2.0 to 70.3.0 in /.github - 13https://github.com/miraheze/python-functions/pull/37#issuecomment-2235424929 [05:26:02] [02python-functions] 07RhinosF1 deleted branch 03dependabot/pip/dot-github/setuptools-70.3.0 [05:26:04] [02python-functions] 07RhinosF1 deleted branch 03dependabot/pip/dot-github/setuptools-70.3.0 - 13https://github.com/miraheze/python-functions [05:26:37] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.74, 21.51, 20.88 [05:27:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.30, 20.15, 20.02 [05:28:44] [02python-functions] 07github-merge-queue[bot] pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/python-functions/compare/ab2eb343f9c3...23766f717f48 [05:28:46] [02python-functions] 07dependabot[bot] 0323766f7 - Bump setuptools from 71.0.0 to 71.0.1 in /.github (#43) [05:28:47] [02python-functions] 07github-merge-queue[bot] deleted branch 03gh-readonly-queue/master/pr-43-ab2eb343f9c3f3e91096ab3fba77e4a2862f35a6 [05:28:48] [02python-functions] 07RhinosF1 closed pull request 03#43: Bump setuptools from 71.0.0 to 71.0.1 in /.github - 13https://github.com/miraheze/python-functions/pull/43 [05:28:51] [02python-functions] 07github-merge-queue[bot] deleted branch 03gh-readonly-queue/master/pr-43-ab2eb343f9c3f3e91096ab3fba77e4a2862f35a6 - 13https://github.com/miraheze/python-functions [05:28:53] [02python-functions] 07RhinosF1 deleted branch 03dependabot/pip/dot-github/setuptools-71.0.1 [05:28:54] [02python-functions] 07RhinosF1 deleted branch 03dependabot/pip/dot-github/setuptools-71.0.1 - 13https://github.com/miraheze/python-functions [05:29:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 17.74, 21.81, 21.89 [05:34:35] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.44, 18.23, 19.86 [05:34:54] miraheze/python-functions - github-merge-queue[bot] the build passed. [05:39:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.15, 21.90, 21.53 [05:39:55] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.48, 20.03, 19.64 [05:41:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.13, 21.73, 21.56 [05:41:50] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 18.35, 19.77, 19.61 [05:55:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.31, 23.15, 21.54 [05:57:17] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.90, 18.98, 18.31 [05:59:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.19, 23.99, 22.35 [06:01:08] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.49, 18.86, 18.52 [06:02:38] [02mw-config] 07BlankEclair opened pull request 03#5616: T12347: Fix fatal exception of type "TypeError" - 13https://github.com/miraheze/mw-config/pull/5616 [06:03:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.40, 22.74, 22.10 [06:03:33] miraheze/mw-config - BlankEclair the build passed. [06:07:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.73, 21.89, 20.12 [06:07:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 19.67, 22.26, 22.13 [06:09:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 24.47, 23.27, 20.86 [06:09:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 27.91, 24.42, 22.93 [06:09:25] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.00, 19.51, 18.37 [06:11:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 17.37, 20.87, 20.27 [06:11:24] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.74, 21.69, 19.29 [06:13:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 16.18, 19.32, 19.79 [06:13:24] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.28, 21.92, 19.71 [06:21:22] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.14, 22.08, 20.51 [06:23:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.23, 20.68, 20.29 [06:23:21] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.41, 20.82, 20.23 [06:27:20] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.11, 20.06, 20.24 [06:31:19] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.74, 21.31, 20.72 [06:35:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.24, 23.33, 21.52 [06:37:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.07, 22.35, 21.40 [06:41:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.74, 24.58, 22.50 [06:45:15] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 28.06, 25.11, 23.09 [06:57:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.16, 23.17, 23.55 [06:59:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.77, 24.48, 24.00 [07:07:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:13:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.92, 23.66, 24.00 [07:15:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 29.18, 25.62, 24.67 [07:45:00] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.86, 21.99, 23.53 [07:50:59] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.44, 23.14, 23.40 [07:52:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [07:52:58] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.55, 23.38, 23.47 [07:54:58] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.98, 24.46, 23.86 [07:56:57] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.61, 23.76, 23.66 [08:01:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.70, 22.54, 23.72 [08:02:56] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.54, 23.35, 23.24 [08:04:10] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [08:05:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.44, 25.12, 24.48 [08:06:55] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.73, 21.11, 22.51 [08:14:53] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.82, 21.85, 22.05 [08:18:52] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.24, 23.52, 22.85 [08:26:50] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.43, 24.51, 23.27 [08:28:50] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.28, 23.25, 23.01 [08:32:49] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.60, 25.15, 23.76 [08:34:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.82, 23.99, 23.49 [08:36:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 23.73, 24.01, 23.56 [08:44:46] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.99, 23.34, 23.75 [08:47:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:56:43] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.38, 23.82, 23.26 [08:57:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [08:58:42] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.56, 23.28, 23.16 [09:02:41] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.05, 24.75, 23.76 [09:03:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.43, 21.14, 23.93 [09:07:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 27.25, 23.51, 24.13 [09:08:39] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 20.57, 23.34, 23.58 [09:13:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 13.77, 20.67, 23.03 [09:14:38] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.38, 23.73, 23.58 [09:15:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 10.86, 18.02, 23.05 [09:19:02] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 12.23, 14.68, 19.48 [09:20:08] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.52, 20.47, 17.32 [09:21:02] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 24.79, 20.36, 22.27 [09:22:12] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.06, 19.25, 14.87 [09:24:05] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 23.20, 23.01, 19.18 [09:26:12] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.77, 22.04, 17.00 [09:27:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 20.80, 22.53, 22.76 [09:27:10] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 21.59, 18.98, 15.28 [09:28:02] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.52, 24.33, 20.49 [09:29:11] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 25.52, 20.96, 16.44 [09:30:01] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.65, 23.54, 20.68 [09:31:10] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.38, 21.54, 17.22 [09:32:10] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [09:32:22] PROBLEM - mw162 Current Load on mw162 is CRITICAL: LOAD CRITICAL - total load average: 25.46, 22.79, 18.08 [09:36:22] PROBLEM - mw162 Current Load on mw162 is WARNING: LOAD WARNING - total load average: 21.41, 23.16, 19.37 [09:38:12] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.66, 23.74, 20.98 [09:40:22] RECOVERY - mw162 Current Load on mw162 is OK: LOAD OK - total load average: 16.45, 19.91, 18.94 [09:42:12] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 26.42, 24.33, 21.83 [09:43:02] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 20.12, 19.04, 20.39 [09:45:10] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.95, 21.47, 19.57 [09:45:50] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 17.50, 19.55, 20.34 [09:47:10] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.11, 21.87, 19.97 [09:52:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [09:56:12] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 22.73, 23.77, 23.26 [09:57:02] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.50, 21.10, 20.46 [09:57:10] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 16.13, 19.55, 20.04 [10:00:12] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 25.80, 24.10, 23.46 [10:01:02] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 14.70, 18.95, 19.86 [10:01:10] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 22.78, 21.21, 20.57 [10:02:12] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.37, 23.77, 23.42 [10:05:10] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 19.61, 20.37, 20.35 [10:08:12] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 26.13, 24.94, 23.95 [10:09:10] PROBLEM - mw171 Current Load on mw171 is CRITICAL: LOAD CRITICAL - total load average: 24.68, 21.99, 20.97 [10:11:10] PROBLEM - mw171 Current Load on mw171 is WARNING: LOAD WARNING - total load average: 19.88, 20.56, 20.55 [10:12:12] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 21.31, 23.91, 23.82 [10:14:26] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 20.52, 19.28, 18.28 [10:16:12] PROBLEM - mw172 Current Load on mw172 is CRITICAL: LOAD CRITICAL - total load average: 24.57, 23.57, 23.61 [10:16:21] RECOVERY - mw151 Current Load on mw151 is OK: LOAD OK - total load average: 14.66, 17.38, 17.70 [10:18:12] PROBLEM - mw172 Current Load on mw172 is WARNING: LOAD WARNING - total load average: 23.37, 23.49, 23.58 [10:22:27] PROBLEM - mw161 Current Load on mw161 is WARNING: LOAD WARNING - total load average: 20.84, 19.47, 17.19 [10:24:21] RECOVERY - mw161 Current Load on mw161 is OK: LOAD OK - total load average: 17.67, 18.65, 17.16 [10:29:10] RECOVERY - mw171 Current Load on mw171 is OK: LOAD OK - total load average: 19.47, 19.66, 20.36 [10:32:12] RECOVERY - mw172 Current Load on mw172 is OK: LOAD OK - total load average: 17.19, 17.65, 20.28 [10:36:42] PROBLEM - mw181 MediaWiki Rendering on mw181 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [10:40:05] PROBLEM - tintinpedia.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tintinpedia.fr' expires in 15 day(s) (Sat 03 Aug 2024 10:32:55 AM GMT +0000). [10:40:33] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/530ff2ba3daa...06c12dd236f3 [10:40:35] [02ssl] 07WikiTideSSLBot 0306c12dd - Bot: Update SSL cert for tintinpedia.fr [10:43:08] RECOVERY - mw181 MediaWiki Rendering on mw181 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 9.103 second response time [10:43:43] PROBLEM - www.tintinpedia.fr - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'tintinpedia.fr' expires in 15 day(s) (Sat 03 Aug 2024 10:32:55 AM GMT +0000). [10:50:25] [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/e80f5589aa20...b7cc746f173f [10:50:28] [02mw-config] 07Universal-Omega 03b7cc746 - Update SQL path (#5615) [10:50:29] [02mw-config] 07redbluegreenhat closed pull request 03#5615: Update SQL path - 13https://github.com/miraheze/mw-config/pull/5615 [10:50:30] [02mw-config] 07redbluegreenhat deleted branch 03Universal-Omega-patch-4 [10:50:33] [02mw-config] 07redbluegreenhat deleted branch 03Universal-Omega-patch-4 - 13https://github.com/miraheze/mw-config [10:50:46] [02mw-config] 07redbluegreenhat closed pull request 03#5614: Re-add TitleKey logging per comment - 13https://github.com/miraheze/mw-config/pull/5614 [10:50:48] [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b7cc746f173f...fd6e14ae8cf6 [10:50:49] [02mw-config] 07Universal-Omega 03fd6e14a - Re-add TitleKey logging per comment (#5614) [10:50:52] [02mw-config] 07redbluegreenhat deleted branch 03Universal-Omega-patch-3 [10:50:54] [02mw-config] 07redbluegreenhat deleted branch 03Universal-Omega-patch-3 - 13https://github.com/miraheze/mw-config [10:51:24] miraheze/mw-config - redbluegreenhat the build passed. [10:51:34] miraheze/mw-config - redbluegreenhat the build passed. [10:51:49] [02mw-config] 07redbluegreenhat closed pull request 03#5616: T12347: Fix fatal exception of type "TypeError" - 13https://github.com/miraheze/mw-config/pull/5616 [10:51:50] [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/fd6e14ae8cf6...d0f1ed242313 [10:51:52] [02mw-config] 07BlankEclair 03d0f1ed2 - T12347: Fix fatal exception of type "TypeError" (#5616) [10:52:39] miraheze/mw-config - redbluegreenhat the build passed. [10:53:28] !log [@test151] starting deploy of {'config': True} to test151 [10:53:29] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [10:54:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [10:54:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:02:04] !log [@mwtask181] starting deploy of {'config': True} to all [11:02:25] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 20s [11:02:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:03:28] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:04:09] PROBLEM - wiki.gab.pt.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.gab.pt.eu.org All nameservers failed to answer the query. [11:06:26] [02mw-config] 07Reception123 closed pull request 03#5608: T12340: Replace : in namespaces with _ - 13https://github.com/miraheze/mw-config/pull/5608 [11:06:27] [02mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/d0f1ed242313...ce9e54d4646f [11:06:29] [02mw-config] 07BlankEclair 03ce9e54d - T12340: Replace : in namespaces with _ (#5608) [11:07:29] !log [@mwtask171] starting deploy of {'config': True} to all [11:07:29] miraheze/mw-config - Reception123 the build passed. [11:07:45] [02mw-config] 07Reception123 commented on pull request 03#5606: $wgRSSAllowLinkTag: Escape HTML - 13https://github.com/miraheze/mw-config/pull/5606#issuecomment-2236233686 [11:07:48] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 20s [11:07:49] [02mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/ce9e54d4646f...b645a0794973 [11:07:51] [02mw-config] 07BlankEclair 03b645a07 - $wgRSSAllowLinkTag: Escape HTML (#5606) [11:07:54] [02mw-config] 07Reception123 closed pull request 03#5606: $wgRSSAllowLinkTag: Escape HTML - 13https://github.com/miraheze/mw-config/pull/5606 [11:08:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:08:09] [02mw-config] 07Reception123 closed pull request 03#5604: T12308: Add *.sm2d, *.smis, and *.smms - 13https://github.com/miraheze/mw-config/pull/5604 [11:08:12] [02mw-config] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/b645a0794973...f2db6fe5089e [11:08:13] [02mw-config] 07BlankEclair 03f2db6fe - T12308: Add *.sm2d, *.smis, and *.smms (#5604) [11:08:42] [02mw-config] 07redbluegreenhat closed pull request 03#5601: Update justinrainbow/json-schema requirement from 5.2.13 to 5.3.0 - 13https://github.com/miraheze/mw-config/pull/5601 [11:08:43] miraheze/mw-config - Reception123 the build passed. [11:08:45] [02mw-config] 07redbluegreenhat pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/f2db6fe5089e...513468340ed1 [11:08:47] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:08:47] [02mw-config] 07dependabot[bot] 035134683 - Update justinrainbow/json-schema requirement from 5.2.13 to 5.3.0 (#5601) [11:08:49] [02mw-config] 07redbluegreenhat deleted branch 03dependabot/composer/justinrainbow/json-schema-5.3.0 [11:08:50] [02mw-config] 07redbluegreenhat deleted branch 03dependabot/composer/justinrainbow/json-schema-5.3.0 - 13https://github.com/miraheze/mw-config [11:08:58] [02mediawiki-repos] 07Reception123 closed pull request 03#25: T12262: Add TimezoneConverter - 13https://github.com/miraheze/mediawiki-repos/pull/25 [11:09:01] [02mediawiki-repos] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mediawiki-repos/compare/9aeb383d709e...b75155552260 [11:09:02] [02mediawiki-repos] 07BlankEclair 03b751555 - T12262: Add TimezoneConverter (#25) [11:09:15] miraheze/mw-config - Reception123 the build passed. [11:09:39] miraheze/mw-config - redbluegreenhat the build passed. [11:10:45] [02dns] 07Reception123 closed pull request 03#520: T12156: Add requested subdomains for rainverse.wiki - 13https://github.com/miraheze/dns/pull/520 [11:10:47] [02dns] 07Reception123 pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/dns/compare/b526b73c69da...62562ac344af [11:10:50] [02dns] 07BlankEclair 0362562ac - T12156: Add requested subdomains for rainverse.wiki (#520) [11:19:25] PROBLEM - www.burnout.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'burnout.wiki' expires in 15 day(s) (Sat 03 Aug 2024 10:57:09 AM GMT +0000). [11:19:35] PROBLEM - burnout.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'burnout.wiki' expires in 15 day(s) (Sat 03 Aug 2024 10:57:09 AM GMT +0000). [11:23:37] !log [@test151] starting deploy of {'config': True} to test151 [11:23:38] !log [@test151] finished deploy of {'config': True} to test151 - SUCCESS in 0s [11:24:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:24:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:24:59] PROBLEM - kingdomdeath.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'kingdomdeath.wiki' expires in 15 day(s) (Sat 03 Aug 2024 10:56:52 AM GMT +0000). [11:25:14] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/06c12dd236f3...0edd630e900c [11:25:16] [02ssl] 07WikiTideSSLBot 030edd630 - Bot: Update SSL cert for kingdomdeath.wiki [11:31:24] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 1 backends are down. mw181 [11:32:27] !log [@mwtask181] starting deploy of {'config': True} to all [11:32:43] !log [@mwtask181] finished deploy of {'config': True} to all - SUCCESS in 16s [11:32:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:33:21] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [11:33:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:34:17] PROBLEM - wiki.corgicam.tv - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.corgicam.tv' expires in 15 day(s) (Sat 03 Aug 2024 11:24:17 AM GMT +0000). [11:34:30] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/0edd630e900c...387182613581 [11:34:31] [02ssl] 07WikiTideSSLBot 033871826 - Bot: Update SSL cert for wiki.corgicam.tv [11:37:35] !log [@mwtask171] starting deploy of {'config': True} to all [11:37:55] !log [@mwtask171] finished deploy of {'config': True} to all - SUCCESS in 19s [11:38:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:38:15] RECOVERY - tintinpedia.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'tintinpedia.fr' will expire on Wed 16 Oct 2024 09:40:27 AM GMT +0000. [11:38:40] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [11:43:10] RECOVERY - www.tintinpedia.fr - LetsEncrypt on sslhost is OK: OK - Certificate 'tintinpedia.fr' will expire on Wed 16 Oct 2024 09:40:27 AM GMT +0000. [11:52:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [11:54:51] RECOVERY - kingdomdeath.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'kingdomdeath.wiki' will expire on Wed 16 Oct 2024 10:25:08 AM GMT +0000. [12:02:29] RECOVERY - wiki.gab.pt.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.gab.pt.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [12:04:03] RECOVERY - wiki.corgicam.tv - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.corgicam.tv' will expire on Wed 16 Oct 2024 10:34:23 AM GMT +0000. [12:11:02] [02CreateWiki] 07translatewiki pushed 031 commit to 03master [+1/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/6228eb9767e8...2deb4261fb64 [12:11:05] [02CreateWiki] 07translatewiki 032deb426 - Localisation updates from https://translatewiki.net. [12:11:05] PROBLEM - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is CRITICAL: rDNS CRITICAL - wiki.andreijiroh.uk.eu.org All nameservers failed to answer the query. [12:11:06] [02ImportDump] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ImportDump/compare/fd9ab0e69be8...41a5c6de5575 [12:11:09] [02ImportDump] 07translatewiki 0341a5c6d - Localisation updates from https://translatewiki.net. [12:11:10] [02SpriteSheet] 07translatewiki pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/SpriteSheet/compare/45fb0c3fefc5...90e10b30d752 [12:11:11] [02SpriteSheet] 07translatewiki 0390e10b3 - Localisation updates from https://translatewiki.net. [12:11:14] [02MirahezeMagic] 07translatewiki pushed 031 commit to 03master [+0/-0/±4] 13https://github.com/miraheze/MirahezeMagic/compare/3a8bd852e5b8...cdab4bfbc31a [12:11:16] [02MirahezeMagic] 07translatewiki 03cdab4bf - Localisation updates from https://translatewiki.net. [12:14:10] miraheze/SpriteSheet - translatewiki the build has errored. [12:14:35] miraheze/CreateWiki - translatewiki the build has errored. [12:15:39] miraheze/MirahezeMagic - translatewiki the build has errored. [12:20:25] miraheze/ImportDump - translatewiki the build passed. [12:24:15] !log [@test151] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 [12:24:16] !log [@test151] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:27] !log [@test151] starting deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 [12:24:28] !log [@test151] finished deploy of {'folders': '1.43/extensions/MirahezeMagic'} to test151 - SUCCESS in 0s [12:24:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:24:48] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:25:22] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:25:57] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:32:53] !log [@mwtask181] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:33:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:33:08] !log [@mwtask181] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 15s [12:33:16] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:37:44] !log [@mwtask171] starting deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all [12:37:54] !log [@mwtask171] finished deploy of {'folders': '1.42/extensions/MirahezeMagic'} to all - SUCCESS in 10s [12:37:55] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:38:00] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [12:40:40] RECOVERY - wiki.andreijiroh.uk.eu.org - reverse DNS on sslhost is OK: SSL OK - wiki.andreijiroh.uk.eu.org reverse DNS resolves to cp36.wikitide.net - CNAME OK [13:23:15] PROBLEM - cloud15 IPMI Sensors on cloud15 is UNKNOWN: ipmi_sdr_cache_open: /root/.freeipmi/sdr-cache/sdr-cache-cloud15.localhost: internal IPMI error-> Execution of /usr/sbin/ipmi-sel failed with return code 1.-> /usr/sbin/ipmi-sel was executed with the following parameters: sudo /usr/sbin/ipmi-sel --output-event-state --interpret-oem-data --entity-sensor-names --sensor-types=all [13:25:15] PROBLEM - cloud15 IPMI Sensors on cloud15 is CRITICAL: IPMI Status: Critical [186 system event log (SEL) entries present] [14:07:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 16.94, 20.49, 23.54 [14:13:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.22, 22.48, 23.49 [14:15:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.40, 23.22, 23.67 [14:19:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.16, 22.42, 23.05 [14:35:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.62, 21.44, 23.00 [14:45:36] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [14:49:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 15.44, 17.98, 20.07 [14:55:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 21.78, 20.81, 20.64 [15:01:48] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.69, 19.47, 15.11 [15:03:44] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 19.96, 19.56, 15.65 [15:03:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 18.28, 19.90, 20.30 [15:11:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 22.81, 19.44, 16.82 [15:11:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.32, 21.28, 20.49 [15:13:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.66, 21.90, 18.07 [15:14:07] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 24 seconds ago with 0 failures [15:15:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.70, 21.61, 18.43 [15:15:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.87, 22.51, 21.07 [15:17:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 26.18, 23.19, 19.39 [15:18:17] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 22.61, 21.16, 18.58 [15:19:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 23.31, 23.23, 19.88 [15:19:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.83, 23.98, 22.04 [15:21:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.98, 24.20, 22.35 [15:25:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.31, 24.30, 21.20 [15:26:01] RECOVERY - mw152 Current Load on mw152 is OK: LOAD OK - total load average: 16.48, 19.14, 18.92 [15:31:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 19.49, 23.55, 22.15 [15:31:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.55, 22.77, 23.01 [15:33:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 28.83, 25.75, 23.13 [15:33:48] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.19, 23.56, 23.26 [15:35:48] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 19.54, 21.90, 22.68 [15:41:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 18.71, 22.92, 23.18 [15:43:48] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 14.99, 17.10, 20.07 [15:46:12] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [15:46:38] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [15:48:07] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.001167416573 secs [15:52:25] [Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [15:55:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.48, 21.99, 21.85 [15:56:46] PROBLEM - fashiondreamer.wiki - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'fashiondreamer.wiki' expires in 15 day(s) (Sat 03 Aug 2024 03:36:43 PM GMT +0000). [15:57:02] PROBLEM - mw151 Current Load on mw151 is WARNING: LOAD WARNING - total load average: 21.92, 21.69, 21.74 [15:57:03] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/387182613581...560a5def2700 [15:57:06] [02ssl] 07WikiTideSSLBot 03560a5de - Bot: Update SSL cert for fashiondreamer.wiki [15:57:58] PROBLEM - wiki.pixandbit.gay - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'wiki.pixandbit.gay' expires in 15 day(s) (Sat 03 Aug 2024 03:36:11 PM GMT +0000). [15:58:09] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/560a5def2700...2e7121efece4 [15:58:11] [02ssl] 07WikiTideSSLBot 032e7121e - Bot: Update SSL cert for wiki.pixandbit.gay [15:59:02] PROBLEM - mw151 Current Load on mw151 is CRITICAL: LOAD CRITICAL - total load average: 25.28, 22.84, 22.15 [15:59:58] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.77, 22.97, 19.95 [16:01:54] PROBLEM - mw152 Current Load on mw152 is WARNING: LOAD WARNING - total load average: 23.48, 23.09, 20.36 [16:03:50] PROBLEM - mw152 Current Load on mw152 is CRITICAL: LOAD CRITICAL - total load average: 26.80, 24.49, 21.22 [16:05:55] PROBLEM - archives.nsgov.org - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'archives.nsgov.org' expires in 15 day(s) (Sat 03 Aug 2024 03:41:55 PM GMT +0000). [16:06:06] [02ssl] 07WikiTideSSLBot pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/ssl/compare/2e7121efece4...d0b2532d2a05 [16:06:09] [02ssl] 07WikiTideSSLBot 03d0b2532 - Bot: Update SSL cert for archives.nsgov.org [16:15:18] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [16:15:42] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 23.79, 22.08, 20.27 [16:21:40] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 25.70, 22.38, 20.87 [16:22:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [16:26:02] RECOVERY - fashiondreamer.wiki - LetsEncrypt on sslhost is OK: OK - Certificate 'fashiondreamer.wiki' will expire on Wed 16 Oct 2024 02:56:55 PM GMT +0000. [16:26:56] RECOVERY - wiki.pixandbit.gay - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.pixandbit.gay' will expire on Wed 16 Oct 2024 02:58:04 PM GMT +0000. [16:35:18] RECOVERY - archives.nsgov.org - LetsEncrypt on sslhost is OK: OK - Certificate 'archives.nsgov.org' will expire on Wed 16 Oct 2024 03:06:00 PM GMT +0000. [16:49:40] PROBLEM - ns2 NTP time on ns2 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [16:51:35] RECOVERY - ns2 NTP time on ns2 is OK: NTP OK: Offset -0.0009868144989 secs [17:01:30] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 15.83, 21.47, 23.55 [17:03:30] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 27.74, 24.00, 24.22 [17:46:27] PROBLEM - ns2 Puppet on ns2 is CRITICAL: CRITICAL: Failed to apply catalog, zero resources tracked by Puppet. It might be a dependency cycle. [17:49:18] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 18.79, 22.08, 23.90 [17:55:17] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.97, 22.29, 23.30 [17:57:16] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 17.82, 20.26, 22.41 [18:01:15] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 11.08, 15.89, 20.20 [18:07:25] [Grafana] RESOLVED: PHP-FPM Worker Usage High https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [18:14:37] RECOVERY - ns2 Puppet on ns2 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [18:27:08] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.98, 20.33, 17.87 [18:29:07] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 17.91, 20.28, 18.22 [18:35:05] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 22.71, 22.22, 19.64 [18:37:25] [Grafana] FIRING: Some MediaWiki Appservers are running out of PHP-FPM workers. https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1[Grafana] !tech FIRING: The mediawiki job queue has more than 500 unclaimed jobs https://grafana.wikitide.net/d/GtxbP1Xnk?orgId=1 [18:39:04] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 16.59, 19.65, 19.19 [18:49:01] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 24.66, 21.70, 20.28 [18:52:59] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 24.33, 20.02, 16.19 [18:54:59] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 20.80, 20.33, 16.78 [18:56:59] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 27.47, 22.98, 18.18 [19:00:59] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 22.89, 23.97, 19.72 [19:02:59] PROBLEM - mw182 Current Load on mw182 is CRITICAL: LOAD CRITICAL - total load average: 25.43, 25.02, 20.65 [19:06:27] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 2 backends are down. mw152 mw162 [19:06:41] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 1 backends are down. mw162 [19:07:10] PROBLEM - cp41 Varnish Backends on cp41 is CRITICAL: 1 backends are down. mw151 [19:08:40] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [19:09:06] RECOVERY - cp41 Varnish Backends on cp41 is OK: All 19 backends are healthy [19:09:35] PROBLEM - cp51 Varnish Backends on cp51 is CRITICAL: 2 backends are down. mw162 mw181 [19:12:32] PROBLEM - cp37 Varnish Backends on cp37 is CRITICAL: 2 backends are down. mw162 mw181 [19:12:35] PROBLEM - cp36 Varnish Backends on cp36 is CRITICAL: 2 backends are down. mw162 mw181 [19:12:55] PROBLEM - mw181 Current Load on mw181 is WARNING: LOAD WARNING - total load average: 8.38, 18.05, 22.90 [19:12:59] PROBLEM - mw182 Current Load on mw182 is WARNING: LOAD WARNING - total load average: 21.73, 23.60, 22.42 [19:13:23] PROBLEM - cp27 Varnish Backends on cp27 is CRITICAL: 1 backends are down. mw151 [19:14:24] RECOVERY - cp26 Varnish Backends on cp26 is OK: All 19 backends are healthy [19:14:27] RECOVERY - cp37 Varnish Backends on cp37 is OK: All 19 backends are healthy [19:14:33] RECOVERY - cp36 Varnish Backends on cp36 is OK: All 19 backends are healthy [19:16:54] RECOVERY - mw181 Current Load on mw181 is OK: LOAD OK - total load average: 7.20, 12.13, 19.39 [19:17:24] RECOVERY - cp51 Varnish Backends on cp51 is OK: All 19 backends are healthy [19:19:16] RECOVERY - cp27 Varnish Backends on cp27 is OK: All 19 backends are healthy [19:20:59] RECOVERY - mw182 Current Load on mw182 is OK: LOAD OK - total load average: 14.48, 17.27, 19.87 [19:22:52] PROBLEM - mw181 Current Load on mw181 is CRITICAL: LOAD CRITICAL - total load average: 26.61, 18.72, 19.52 [19:28:20] PROBLEM - cp26 Varnish Backends on cp26 is CRITICAL: 1 backends are down. mw152 [19:28:38] PROBLEM - ns2 NTP time on ns2 is UNKNOWN: check_ntp_time: Invalid hostname/address - time.cloudflare.comUsage: check_ntp_time -H [-4|-6] [-w ] [-c ] [-v verbose] [-o