[00:01:02] RECOVERY - mw131 Disk Space on mw131 is OK: DISK OK - free space: / 3189 MB (13% inode=82%); [00:03:46] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 4.00, 3.95, 3.29 [00:05:44] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.40, 4.99, 3.76 [00:07:08] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.90, 3.77, 4.00 [00:08:15] RECOVERY - wiki.mastodon.kr - LetsEncrypt on sslhost is OK: OK - Certificate 'wiki.mastodon.kr' will expire on Mon 30 Jan 2023 22:42:37 GMT +0000. [00:09:08] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 8.01, 4.70, 4.27 [00:10:30] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 8.78, 5.42, 3.11 [00:11:44] PROBLEM - cp22 Current Load on cp22 is CRITICAL: CRITICAL - load average: 4.31, 4.21, 3.18 [00:15:07] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 2.00, 3.49, 3.96 [00:15:30] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.13, 3.32, 3.70 [00:16:22] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03add-job [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/091aca04625b...a41d6459480d [00:16:23] [02miraheze/CreateWiki] 07Universal-Omega 03a41d645 - Update WikiManager.php [00:16:24] [02CreateWiki] 07Universal-Omega synchronize pull request 03#351: Add new SetContainersAccessJob - 13https://github.com/miraheze/CreateWiki/pull/351 [00:17:07] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 4.76, 4.44, 4.27 [00:17:27] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.43, 2.63, 3.40 [00:19:32] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 0.61, 3.54, 3.81 [00:20:07] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 2.30, 3.29, 3.21 [00:20:26] miraheze/CreateWiki - Universal-Omega the build has errored. [00:23:26] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 0.48, 2.07, 3.15 [00:23:57] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03add-job [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/a41d6459480d...ebb4cdaddec1 [00:23:58] [02miraheze/CreateWiki] 07Universal-Omega 03ebb4cda - Update WikiManagerTest.php [00:24:01] [02CreateWiki] 07Universal-Omega synchronize pull request 03#351: Add new SetContainersAccessJob - 13https://github.com/miraheze/CreateWiki/pull/351 [00:24:20] [02miraheze/mw-config] 07paladox pushed 031 commit to 03paladox-patch-1 [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/3e7261c21c8a [00:24:21] [02miraheze/mw-config] 07paladox 033e7261c - Set wgImageMagickTempDir to /tmp/magick-tmp [00:24:23] [02mw-config] 07paladox created branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [00:24:26] [02mw-config] 07paladox opened pull request 03#4973: Set wgImageMagickTempDir to /tmp/magick-tmp - 13https://github.com/miraheze/mw-config/pull/4973 [00:25:20] miraheze/mw-config - paladox the build passed. [00:27:23] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/e255318d15fb [00:27:25] [02miraheze/puppet] 07paladox 03e255318 - mediawiki: create /tmp/magick-tmp and wipe it regularly [00:27:28] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [00:27:31] [02puppet] 07paladox opened pull request 03#2977: mediawiki: create /tmp/magick-tmp and wipe it regularly - 13https://github.com/miraheze/puppet/pull/2977 [00:28:02] miraheze/CreateWiki - Universal-Omega the build has errored. [00:28:31] [02puppet] 07paladox closed pull request 03#2977: mediawiki: create /tmp/magick-tmp and wipe it regularly - 13https://github.com/miraheze/puppet/pull/2977 [00:28:33] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/dd2e3b1f4693...aad60e83d9b3 [00:28:35] [02miraheze/puppet] 07paladox 03aad60e8 - mediawiki: create /tmp/magick-tmp and wipe it regularly (#2977) [00:28:36] [02miraheze/puppet] 07paladox deleted branch 03paladox-patch-5 [00:28:39] [02puppet] 07paladox deleted branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [00:30:34] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/aad60e83d9b3...3fc02473c763 [00:30:37] [02miraheze/puppet] 07paladox 033fc0247 - mediawiki: purge .jpg regularly in /tmp/magick-tmp [00:32:17] [02CreateWiki] 07Universal-Omega synchronize pull request 03#351: Add new SetContainersAccessJob - 13https://github.com/miraheze/CreateWiki/pull/351 [00:32:18] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03add-job [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/ebb4cdaddec1...7e28d1c7cd29 [00:32:20] [02miraheze/CreateWiki] 07Universal-Omega 037e28d1c - Update WikiManagerTest.php [00:33:27] [02mw-config] 07paladox closed pull request 03#4973: Set wgImageMagickTempDir to /tmp/magick-tmp - 13https://github.com/miraheze/mw-config/pull/4973 [00:33:30] [02miraheze/mw-config] 07paladox deleted branch 03paladox-patch-1 [00:33:32] [02miraheze/mw-config] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/6c49191ffbe1...68588697d9a3 [00:33:35] [02miraheze/mw-config] 07paladox 036858869 - Set wgImageMagickTempDir to /tmp/magick-tmp (#4973) [00:33:38] [02mw-config] 07paladox deleted branch 03paladox-patch-1 - 13https://github.com/miraheze/mw-config [00:33:42] !log [paladox@mwtask141] starting deploy of {'pull': 'config', 'config': True} to all [00:33:51] !log [paladox@mwtask141] finished deploy of {'pull': 'config', 'config': True} to all - SUCCESS in 8s [00:33:52] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:34:02] !log [paladox@mwtask141] starting deploy of {'config': True} to all [00:34:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:34:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:34:20] !log [paladox@mwtask141] finished deploy of {'config': True} to all - SUCCESS in 18s [00:34:25] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:34:34] PROBLEM - puppet141 Current Load on puppet141 is CRITICAL: CRITICAL - load average: 4.42, 2.90, 1.94 [00:34:39] miraheze/mw-config - paladox the build passed. [00:35:05] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 2.35, 3.18, 3.90 [00:36:17] miraheze/CreateWiki - Universal-Omega the build has errored. [00:36:20] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03add-job [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/7e28d1c7cd29...9ddacb8057cf [00:36:23] [02miraheze/CreateWiki] 07Universal-Omega 039ddacb8 - Update WikiManagerTest.php [00:36:25] [02CreateWiki] 07Universal-Omega synchronize pull request 03#351: Add new SetContainersAccessJob - 13https://github.com/miraheze/CreateWiki/pull/351 [00:36:27] CosmicAlpha: fixed [00:36:34] RECOVERY - puppet141 Current Load on puppet141 is OK: OK - load average: 3.00, 3.02, 2.11 [00:36:38] !log [@test131] starting deploy of {'config': True} to all [00:36:39] !log [@test131] DEPLOY ABORTED: Canary check failed for beta.betaheze.org@localhost [00:36:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:37:03] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:37:04] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.29, 4.22, 3.65 [00:37:05] why is beta read only CosmicAlpha ? [00:37:18] oh it's private [00:37:44] @paladox it is already wiped regularly, yet still happened. All image files in tmp are wiped every 30 minutes (or if not accessed in 30 minutes) [00:37:59] !log [paladox@test131] starting deploy of {'pull': 'config', 'config': True, 'force': True} to all [00:38:01] !log [paladox@test131] finished deploy of {'pull': 'config', 'config': True, 'force': True} to all - SUCCESS in 2s [00:38:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:38:07] !log [paladox@test131] starting deploy of {'config': True, 'force': True} to all [00:38:08] !log [paladox@test131] finished deploy of {'config': True, 'force': True} to all - SUCCESS in 0s [00:38:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:38:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:38:31] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [00:38:35] [02miraheze/CreateWiki] 07github-actions[bot] pushed 031 commit to 03add-job [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/9ddacb8057cf...541238c36319 [00:38:36] [02miraheze/CreateWiki] 07github-actions 03541238c - CI: lint code to MediaWiki standards [00:38:39] [02CreateWiki] 07github-actions[bot] synchronize pull request 03#351: Add new SetContainersAccessJob - 13https://github.com/miraheze/CreateWiki/pull/351 [00:39:04] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 4.03, 3.80, 4.01 [00:39:07] it's not wiped regularly though CosmicAlpha. Because the change i just did actually purged the tmp of those files (i compared the before and after) [00:39:46] one file: [00:39:48] > -rw-r--r-- 1 www-data www-data 1.2M Oct 31 17:53 localcopy_e5cedb602583.gif [00:40:05] Oh localcopy is new [00:40:29] reduced /tmp from 9g to 405M [00:40:54] PROBLEM - test131 Puppet on test131 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 3 minutes ago with 1 failures. Failed resources (up to 3 shown): Exec[MediaWiki Config Sync] [00:40:59] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.95, 3.93, 3.69 [00:41:03] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 3.36, 3.52, 3.88 [00:41:53] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-1 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/7e86d53011bb [00:41:56] [02miraheze/puppet] 07Universal-Omega 037e86d53 - Add localcopy to clean-tmp-files [00:41:57] [02puppet] 07Universal-Omega created branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/puppet [00:42:49] CosmicAlpha: we can remove that cron. [00:43:00] my change purges additional things. [00:43:01] [02puppet] 07Universal-Omega opened pull request 03#2978: Add localcopy to clean-tmp-files - 13https://github.com/miraheze/puppet/pull/2978 [00:43:03] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 5.65, 5.14, 4.49 [00:44:53] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.48, 2.59, 3.20 [00:45:00] @paladox but your change does ctime, not atime, it should be atime, as sometimes files are not changed but needed, so go off the access time, not the change time I think? [00:45:21] wikimedia use ctime. [00:45:41] before they removed it as they changed their infra [00:46:20] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/3fc02473c763...83b29636655b [00:46:22] [02miraheze/puppet] 07paladox 0383b2963 - mediawiki: purge .gif regularly in /tmp/magick-tmp [00:47:00] miraheze/CreateWiki - Universal-Omega the build passed. [00:47:07] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 5.31, 3.41, 2.49 [00:47:31] https://github.com/wikimedia/puppet/commit/75b815dc055aaa0db8bec0d40445bbfa4c813d90 [00:47:32] [url] Remove obsolete fontconfig/imagemagick code from mediawiki::multimedia · wikimedia/puppet@75b815d · GitHub | github.com [00:48:00] PROBLEM - mw122 Current Load on mw122 is WARNING: WARNING - load average: 11.85, 10.47, 8.49 [00:49:05] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.49, 3.03, 2.47 [00:49:34] paladox: I like atime better to make sure we don't clear stuff still needed, if access time, also in that case (using atime instead of ctime) we can purge every 30 minutes instead of 2 hours (as FYI I purged it 45 minutes earlier also before it ran out of space again 45 minutes later, as I purged once in it first alerted to low space also) [00:49:53] It has a 2 hour time [00:49:54] RECOVERY - mw122 Current Load on mw122 is OK: OK - load average: 7.07, 9.19, 8.25 [00:50:05] paladox: I know, but that is to much. [00:50:19] I purged it 45 minutes before the disk space ran out. [00:50:20] But also yours purges less whereas mine is cleaner + purges more. [00:51:07] you can change ctime -> atime [00:51:21] I cleared tmp of everything yours purges, 45 minutes before it ran out completly. So it recreated everything in 45 minutes. 2 hours won't prevent the issue. [00:52:14] I cleared /tmp/*.* 45 minutes before we ran out. [00:52:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03paladox-patch-5 [+0/-0/±1] 13https://github.com/miraheze/puppet/commit/43f51315f277 [00:52:23] [02miraheze/puppet] 07paladox 0343f5131 - mediawiki: change type for tidy to atime [00:52:26] [02puppet] 07paladox created branch 03paladox-patch-5 - 13https://github.com/miraheze/puppet [00:52:28] [02puppet] 07paladox opened pull request 03#2979: mediawiki: change type for tidy to atime - 13https://github.com/miraheze/puppet/pull/2979 [00:52:43] CosmicAlpha: the files i saw were created well before 45 minutes before [00:52:49] they were created days ago [00:52:56] Well which server? [00:53:10] mw131 [00:53:21] I cleared on mw121, and then 45 minutes later it ran out. [00:53:49] > -rw-r--r-- 1 www-data www-data 20K Oct 27 20:21 trollpastawiki_2112_l.jpg [00:54:22] https://www.irccloud.com/pastebin/FOXSLYsZ/ [00:54:24] https://www.irccloud.com/pastebin/20gSCnt5/ [00:54:36] seems my change has stabilised things then. [00:57:38] (i didn't see we had a cron for /tmp at the time i did my change) [00:58:20] [02puppet] 07paladox closed pull request 03#2978: Add localcopy to clean-tmp-files - 13https://github.com/miraheze/puppet/pull/2978 [00:58:22] [02miraheze/puppet] 07paladox pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/83b29636655b...8fd91edbaf8e [00:58:24] [02miraheze/puppet] 07Universal-Omega 038fd91ed - Add localcopy to clean-tmp-files (#2978) [00:58:25] [02miraheze/puppet] 07paladox deleted branch 03Universal-Omega-patch-1 [00:58:28] [02puppet] 07paladox deleted branch 03Universal-Omega-patch-1 - 13https://github.com/miraheze/puppet [00:58:44] [02miraheze/puppet] 07paladox pushed 032 commits to 03paladox-patch-5 [+0/-0/±2] 13https://github.com/miraheze/puppet/compare/43f51315f277...1fef510ec581 [00:58:46] [02miraheze/puppet] 07paladox 031fef510 - Merge branch 'master' into paladox-patch-5 [00:58:47] [02puppet] 07paladox synchronize pull request 03#2979: mediawiki: change type for tidy to atime - 13https://github.com/miraheze/puppet/pull/2979 [00:58:53] RECOVERY - test131 Puppet on test131 is OK: OK: Puppet is currently enabled, last run 31 seconds ago with 0 failures [00:59:18] @paladox I guess we will see. I hope so, but we will see if it was just a rare instance to regenerate within 45 minutes or not. I guess keep at ctime also if we are doing 2 hours, or atime will keep things a little longer also. And yeah, I added the other cron a while ago because of the same type of issue, but not quite a severe as this time, since this time 3 mw servers ran out of storage at around the same time. [01:00:34] ok [01:01:54] I also did https://github.com/miraheze/mw-config/commit/68588697d9a364008c874b9690800ce1a59eadf9 CosmicAlpha [01:01:55] [url] Set wgImageMagickTempDir to /tmp/magick-tmp (#4973) · miraheze/mw-config@6858869 · GitHub | github.com [01:03:58] thanks paladox [01:04:43] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.15, 3.60, 2.89 [01:05:21] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03add-job [01:05:22] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+1/-0/±4] 13https://github.com/miraheze/CreateWiki/compare/4ab1c09e05b1...d1537ad9bab2 [01:05:23] [02miraheze/CreateWiki] 07Universal-Omega 03d1537ad - Add new SetContainersAccessJob (#351) [01:05:24] [02CreateWiki] 07Universal-Omega closed pull request 03#351: Add new SetContainersAccessJob - 13https://github.com/miraheze/CreateWiki/pull/351 [01:05:27] [02CreateWiki] 07Universal-Omega deleted branch 03add-job - 13https://github.com/miraheze/CreateWiki [01:06:26] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 5.50, 3.60, 3.11 [01:06:40] [02miraheze/puppet] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/puppet/compare/8fd91edbaf8e...7143474cef91 [01:06:40] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 3.25, 3.26, 2.84 [01:06:42] [02miraheze/puppet] 07Universal-Omega 037143474 - jobQueueCollector: add SetContainersAccessJob job [01:08:28] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/16a1afc66eb7...49d8ac8c24db [01:08:30] [02miraheze/mediawiki] 07Universal-Omega 0349d8ac8 - Update CreateWiki [01:08:45] !log [universalomega@mwtask141] starting deploy of {'world': True} to all [01:08:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:08:58] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 2.75, 3.49, 3.91 [01:10:15] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/dfd8a70e43da...b3cb4109746a [01:10:18] [02miraheze/mediawiki] 07Universal-Omega 03b3cb410 - Update CreateWiki [01:12:58] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 4.04, 3.53, 3.79 [01:13:46] PROBLEM - mw131 MediaWiki Rendering on mw131 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:14:10] miraheze/CreateWiki - Universal-Omega the build passed. [01:15:12] !log [universalomega@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 386s [01:15:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:15:38] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.74, 3.00, 2.34 [01:15:45] RECOVERY - mw131 MediaWiki Rendering on mw131 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.691 second response time [01:15:56] PROBLEM - gluster121 Current Load on gluster121 is CRITICAL: CRITICAL - load average: 4.46, 3.04, 2.23 [01:15:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.42, 3.79, 2.73 [01:17:36] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.10, 2.81, 2.36 [01:17:56] RECOVERY - gluster121 Current Load on gluster121 is OK: OK - load average: 2.09, 2.91, 2.30 [01:17:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.75, 3.02, 2.59 [01:18:09] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.10, 3.54, 3.86 [01:18:57] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 2.35, 3.56, 3.77 [01:22:09] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.65, 2.60, 3.40 [01:22:56] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 1.50, 2.49, 3.28 [01:26:15] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://github.com/miraheze/CreateWiki/compare/d1537ad9bab2...176f4d3e6aed [01:26:16] [02miraheze/CreateWiki] 07Universal-Omega 03176f4d3 - Use setContainersAccess script not job for wiki creation [01:27:08] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_38 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/49d8ac8c24db...26d9112905ab [01:27:09] [02miraheze/mediawiki] 07Universal-Omega 0326d9112 - Update CreateWiki [01:27:23] !log [universalomega@mwtask141] starting deploy of {'world': True} to all [01:27:37] CosmicAlpha: [01:27:39] https://www.irccloud.com/pastebin/4dWvxjHC/ [01:27:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:27:47] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_39 [+0/-0/±1] 13https://github.com/miraheze/mediawiki/compare/b3cb4109746a...7cbd121c0ec9 [01:27:49] [02miraheze/mediawiki] 07Universal-Omega 037cbd121 - Update CreateWiki [01:28:38] paladox: yes, femiwiki related, will be fixed once puppet runs, it happens after deploys, until next puppet run it seems. [01:28:47] oh [01:29:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.79, 3.21, 2.84 [01:31:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.75, 2.91, 2.76 [01:32:09] !log [universalomega@mwtask141] finished deploy of {'world': True} to all - SUCCESS in 286s [01:33:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.48, 3.44, 2.74 [01:34:24] PROBLEM - mw142 MediaWiki Rendering on mw142 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:34:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [01:34:54] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 3.81, 3.70, 3.46 [01:35:25] PROBLEM - mw132 MediaWiki Rendering on mw132 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:35:29] PROBLEM - swiftobject111 Current Load on swiftobject111 is WARNING: WARNING - load average: 3.79, 2.87, 2.31 [01:35:57] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.78, 2.67, 2.53 [01:36:29] miraheze/CreateWiki - Universal-Omega the build passed. [01:36:53] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 2.82, 3.36, 3.36 [01:37:17] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [01:37:22] RECOVERY - mw132 MediaWiki Rendering on mw132 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.689 second response time [01:37:23] RECOVERY - swiftobject111 Current Load on swiftobject111 is OK: OK - load average: 2.75, 2.79, 2.35 [01:38:26] RECOVERY - mw142 MediaWiki Rendering on mw142 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.769 second response time [01:39:09] PROBLEM - cp23 NTP time on cp23 is WARNING: NTP WARNING: Offset 0.1407226622 secs [01:39:14] PROBLEM - graylog121 Current Load on graylog121 is CRITICAL: CRITICAL - load average: 4.20, 3.25, 2.45 [01:40:52] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 3.14, 3.57, 3.47 [01:41:15] RECOVERY - graylog121 Current Load on graylog121 is OK: OK - load average: 2.14, 2.94, 2.45 [01:41:28] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 3.074 second response time [01:46:46] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.51, 3.46, 2.97 [01:48:41] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.93, 2.97, 2.85 [01:52:50] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 2.02, 3.01, 3.32 [01:58:40] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.02, 3.15, 2.86 [02:00:37] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.90, 3.10, 2.89 [02:05:09] RECOVERY - cp23 NTP time on cp23 is OK: NTP OK: Offset 0.06761232018 secs [02:06:14] PROBLEM - db112 Current Load on db112 is WARNING: WARNING - load average: 0.98, 2.41, 7.54 [02:06:31] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 6.54, 4.69, 3.57 [02:06:46] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 3.86, 3.16, 3.07 [02:06:48] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.16, 3.20, 2.62 [02:08:14] RECOVERY - db112 Current Load on db112 is OK: OK - load average: 0.37, 1.73, 6.67 [02:08:46] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 4.57, 3.79, 3.32 [02:08:46] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.66, 2.83, 2.54 [02:12:22] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.39, 3.47, 3.39 [02:12:45] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 3.04, 3.97, 3.55 [02:13:45] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03secure-containers [+0/-0/±1] 13https://github.com/miraheze/mw-config/commit/31996f1cbe9b [02:13:46] [02miraheze/mw-config] 07Universal-Omega 0331996f1 - Enable secure containers [02:13:48] [02mw-config] 07Universal-Omega created branch 03secure-containers - 13https://github.com/miraheze/mw-config [02:13:51] [02mw-config] 07Universal-Omega opened pull request 03#4974: Enable secure containers - 13https://github.com/miraheze/mw-config/pull/4974 [02:15:01] miraheze/mw-config - Universal-Omega the build passed. [02:15:19] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03secure-containers [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/31996f1cbe9b...237f18325a73 [02:15:22] [02miraheze/mw-config] 07Universal-Omega 03237f183 - Update GlobalSettings.php [02:15:25] [02mw-config] 07Universal-Omega synchronize pull request 03#4974: Enable secure containers - 13https://github.com/miraheze/mw-config/pull/4974 [02:16:25] miraheze/mw-config - Universal-Omega the build passed. [02:16:43] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03secure-containers [+0/-0/±1] 13https://github.com/miraheze/mw-config/compare/237f18325a73...ca2d045ff193 [02:16:46] [02miraheze/mw-config] 07Universal-Omega 03ca2d045 - Update FileBackend.php [02:16:49] [02mw-config] 07Universal-Omega synchronize pull request 03#4974: Enable secure containers - 13https://github.com/miraheze/mw-config/pull/4974 [02:17:56] miraheze/mw-config - Universal-Omega the build passed. [02:18:13] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.63, 3.16, 3.32 [02:20:43] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 4.70, 3.95, 3.62 [02:22:43] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 3.29, 3.68, 3.55 [02:24:42] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 5.01, 4.19, 3.75 [02:26:26] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.43, 3.80, 3.30 [02:28:09] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.02, 5.15, 4.20 [02:34:09] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 1.86, 3.42, 3.76 [02:36:40] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 2.30, 3.47, 3.75 [02:38:09] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 8.47, 5.31, 4.36 [02:38:40] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 5.06, 4.32, 4.04 [02:39:57] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.88, 3.61, 3.68 [02:44:09] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.05, 3.50, 3.77 [02:44:39] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 1.76, 3.38, 3.80 [02:45:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.72, 4.21, 3.85 [02:47:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.77, 3.68, 3.70 [02:48:09] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.11, 2.77, 3.40 [02:53:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.47, 2.65, 3.27 [02:54:37] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 2.35, 2.88, 3.40 [03:03:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 8.77, 6.25, 3.91 [03:03:58] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.63, 3.42, 3.25 [03:07:52] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.38, 3.55, 3.43 [03:08:33] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 4.07, 3.63, 3.32 [03:09:49] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.09, 2.97, 3.23 [03:09:57] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.30, 3.77, 3.52 [03:10:33] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 3.53, 3.61, 3.36 [03:11:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.02, 3.24, 2.88 [03:11:56] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.89, 3.91, 3.58 [03:13:47] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.39, 3.91, 3.56 [03:13:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.38, 2.57, 2.69 [03:13:57] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.63, 3.43, 3.45 [03:14:32] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 4.80, 3.84, 3.48 [03:15:44] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.23, 3.27, 3.36 [03:15:57] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.16, 2.59, 3.13 [03:16:32] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 3.86, 3.83, 3.52 [03:18:31] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 4.53, 4.21, 3.71 [03:23:56] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.23, 3.69, 3.06 [03:25:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.95, 3.07, 2.91 [03:29:55] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.41, 3.24, 2.00 [03:31:55] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.68, 3.19, 2.13 [03:33:55] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.78, 2.29, 1.93 [03:36:28] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 3.12, 3.44, 3.91 [03:44:26] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 2.64, 2.72, 3.33 [03:46:28] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.80, 3.42, 2.99 [03:48:28] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.48, 3.07, 2.90 [03:56:04] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 6.97, 4.36, 3.28 [03:58:00] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.48, 3.98, 3.27 [03:58:24] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 4.15, 3.54, 3.30 [03:59:57] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 5.38, 4.48, 3.53 [04:00:24] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 2.51, 3.24, 3.23 [04:01:56] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.40, 3.84, 3.42 [04:02:28] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.59, 4.22, 3.44 [04:03:57] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.05, 3.26, 3.26 [04:03:58] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 4.74, 3.70, 3.19 [04:04:22] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 3.45, 3.61, 3.40 [04:04:28] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.43, 3.91, 3.42 [04:05:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.15, 3.26, 3.10 [04:05:56] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:06:28] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.75, 3.16, 3.20 [04:08:22] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 3.02, 3.27, 3.31 [04:08:35] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [04:14:20] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 4.39, 3.72, 3.47 [04:16:20] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 3.33, 3.58, 3.45 [04:20:19] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 4.25, 3.72, 3.51 [04:22:18] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 3.22, 3.52, 3.46 [04:24:18] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 2.31, 3.08, 3.30 [04:46:04] PROBLEM - cloud11 SMART on cloud11 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 60 seconds. [04:48:47] PROBLEM - cloud11 SMART on cloud11 is WARNING: WARNING: [cciss,6] - No health status line found, Self-test log contains errors --- [cciss,0] - Device is clean --- [cciss,1] - Device is clean --- [cciss,2] - Device is clean --- [cciss,3] - Device is clean --- [cciss,4] - Device is clean --- [cciss,5] - Device is clean| [04:50:13] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.19, 2.97, 2.37 [04:56:03] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 2.95, 3.31, 2.77 [05:07:04] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.94, 3.28, 2.54 [05:09:01] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.38, 2.55, 2.36 [05:18:17] [02miraheze/RemovePII] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-codesniffer-40.0.1 [+0/-0/±1] 13https://github.com/miraheze/RemovePII/commit/20d8fdcfd87d [05:18:19] [02miraheze/RemovePII] 07dependabot[bot] 0320d8fdc - Update mediawiki/mediawiki-codesniffer requirement from 39.0.0 to 40.0.1 [05:18:20] [02RemovePII] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-40.0.1 - 13https://github.com/miraheze/RemovePII [05:18:22] [02RemovePII] 07dependabot[bot] opened pull request 03#64: Update mediawiki/mediawiki-codesniffer requirement from 39.0.0 to 40.0.1 - 13https://github.com/miraheze/RemovePII/pull/64 [05:18:25] [02RemovePII] 07dependabot[bot] labeled pull request 03#64: Update mediawiki/mediawiki-codesniffer requirement from 39.0.0 to 40.0.1 - 13https://github.com/miraheze/RemovePII/pull/64 [05:18:26] [02RemovePII] 07dependabot[bot] labeled pull request 03#64: Update mediawiki/mediawiki-codesniffer requirement from 39.0.0 to 40.0.1 - 13https://github.com/miraheze/RemovePII/pull/64 [05:19:47] PROBLEM - ping6 on ns1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 122.14 ms [05:19:54] PROBLEM - ping6 on cp32 is WARNING: PING WARNING - Packet loss = 0%, RTA = 122.21 ms [05:20:06] PROBLEM - ping6 on cp33 is WARNING: PING WARNING - Packet loss = 0%, RTA = 122.15 ms [05:21:52] miraheze/RemovePII - dependabot[bot] the build passed. [05:23:24] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 6.47, 4.28, 3.28 [05:24:18] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.87, 3.10, 2.77 [05:25:23] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.20, 3.92, 3.27 [05:26:14] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.23, 2.78, 2.69 [05:27:21] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 2.44, 3.40, 3.16 [05:29:54] RECOVERY - ping6 on cp32 is OK: PING OK - Packet loss = 0%, RTA = 105.26 ms [05:29:57] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 1.65, 3.47, 2.99 [05:30:01] RECOVERY - ping6 on ns1 is OK: PING OK - Packet loss = 0%, RTA = 107.04 ms [05:30:04] RECOVERY - ping6 on cp33 is OK: PING OK - Packet loss = 0%, RTA = 106.28 ms [05:31:16] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.91, 4.41, 3.61 [05:31:57] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.07, 2.69, 2.76 [05:32:03] PROBLEM - gluster101 Current Load on gluster101 is CRITICAL: CRITICAL - load average: 5.37, 3.99, 3.19 [05:37:11] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.32, 3.53, 3.58 [05:37:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 2.10, 3.42, 3.26 [05:39:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 2.72, 3.01, 3.12 [05:41:07] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.16, 2.53, 3.17 [05:41:27] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.31, 3.45, 2.97 [05:43:24] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.46, 2.70, 2.75 [05:53:13] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 4.03, 2.99, 2.72 [05:53:55] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 5.48, 3.30, 2.16 [05:55:55] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 3.69, 3.31, 2.30 [05:57:10] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.57, 3.31, 2.92 [05:57:55] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 4.14, 3.65, 2.56 [05:59:55] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 1.64, 2.96, 2.44 [06:00:12] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 3.50, 3.79, 3.26 [06:06:01] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.85, 3.56, 3.23 [06:06:12] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 3.28, 3.38, 3.23 [06:09:55] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 2.92, 3.36, 3.24 [06:10:12] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 4.78, 3.97, 3.51 [06:14:12] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 2.25, 3.45, 3.44 [06:16:12] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 2.11, 2.99, 3.27 [06:20:12] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 5.31, 4.03, 3.63 [06:22:12] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 2.79, 3.53, 3.49 [06:24:12] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 2.33, 3.09, 3.33 [06:24:28] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.48, 4.83, 3.57 [06:24:34] PROBLEM - mw121 MediaWiki Rendering on mw121 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [06:26:33] RECOVERY - mw121 MediaWiki Rendering on mw121 is OK: HTTP OK: HTTP/1.1 200 OK - 8191 bytes in 0.292 second response time [06:28:28] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.78, 3.26, 3.23 [06:28:47] PROBLEM - cp23 Current Load on cp23 is CRITICAL: CRITICAL - load average: 7.26, 5.21, 3.04 [06:28:52] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 3.22, 4.29, 3.40 [06:29:03] PROBLEM - cp22 Current Load on cp22 is WARNING: WARNING - load average: 2.92, 3.70, 2.23 [06:29:29] PROBLEM - cp32 Current Load on cp32 is CRITICAL: CRITICAL - load average: 2.47, 4.32, 3.36 [06:30:48] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 2.19, 3.57, 3.24 [06:31:03] RECOVERY - cp22 Current Load on cp22 is OK: OK - load average: 2.43, 3.20, 2.21 [06:31:04] [02miraheze/phabricator-extensions] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-codesniffer-40.0.1 [+0/-0/±1] 13https://github.com/miraheze/phabricator-extensions/commit/2a3413768061 [06:31:06] [02miraheze/phabricator-extensions] 07dependabot[bot] 032a34137 - Update mediawiki/mediawiki-codesniffer requirement from 39.0.0 to 40.0.1 [06:31:08] [02phabricator-extensions] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-40.0.1 - 13https://github.com/miraheze/phabricator-extensions [06:31:11] [02phabricator-extensions] 07dependabot[bot] opened pull request 03#32: Update mediawiki/mediawiki-codesniffer requirement from 39.0.0 to 40.0.1 - 13https://github.com/miraheze/phabricator-extensions/pull/32 [06:31:12] [02phabricator-extensions] 07dependabot[bot] labeled pull request 03#32: Update mediawiki/mediawiki-codesniffer requirement from 39.0.0 to 40.0.1 - 13https://github.com/miraheze/phabricator-extensions/pull/32 [06:31:15] [02phabricator-extensions] 07dependabot[bot] labeled pull request 03#32: Update mediawiki/mediawiki-codesniffer requirement from 39.0.0 to 40.0.1 - 13https://github.com/miraheze/phabricator-extensions/pull/32 [06:31:26] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 2.19, 3.60, 3.21 [06:32:03] miraheze/phabricator-extensions - dependabot[bot] the build passed. [06:32:38] PROBLEM - cp23 Current Load on cp23 is WARNING: WARNING - load average: 2.17, 3.75, 2.93 [06:32:45] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.92, 3.18, 3.14 [06:33:23] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 1.65, 2.83, 2.96 [06:34:33] RECOVERY - cp23 Current Load on cp23 is OK: OK - load average: 0.58, 2.64, 2.62 [06:41:10] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 5.13, 3.87, 3.40 [06:41:42] PROBLEM - mw121 Current Load on mw121 is WARNING: WARNING - load average: 11.50, 10.07, 8.26 [06:43:42] RECOVERY - mw121 Current Load on mw121 is OK: OK - load average: 5.78, 8.50, 7.91 [06:45:56] PROBLEM - gluster101 Current Load on gluster101 is WARNING: WARNING - load average: 3.61, 3.72, 2.89 [06:46:28] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 2.98, 3.79, 3.25 [06:47:56] RECOVERY - gluster101 Current Load on gluster101 is OK: OK - load average: 1.41, 2.82, 2.66 [06:48:28] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.80, 3.06, 3.05 [06:51:08] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 3.07, 3.96, 3.77 [06:54:28] PROBLEM - gluster102 Current Load on gluster102 is WARNING: WARNING - load average: 3.69, 3.51, 3.22 [06:56:04] PROBLEM - cp33 Current Load on cp33 is WARNING: WARNING - load average: 3.67, 3.42, 2.75 [06:56:28] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.90, 2.92, 3.03 [06:57:07] PROBLEM - test131 Current Load on test131 is CRITICAL: CRITICAL - load average: 5.40, 4.43, 3.98 [06:58:00] RECOVERY - cp33 Current Load on cp33 is OK: OK - load average: 1.55, 2.81, 2.61 [06:59:07] PROBLEM - test131 Current Load on test131 is WARNING: WARNING - load average: 3.13, 3.97, 3.87 [07:00:28] PROBLEM - gluster102 Current Load on gluster102 is CRITICAL: CRITICAL - load average: 4.04, 4.33, 3.67 [07:02:28] RECOVERY - gluster102 Current Load on gluster102 is OK: OK - load average: 1.82, 3.35, 3.38 [07:05:06] RECOVERY - test131 Current Load on test131 is OK: OK - load average: 2.43, 2.89, 3.38 [07:07:35] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.59, 3.15, 2.67 [07:09:32] RECOVERY - cp32 Current Load on cp32 is OK: OK - load average: 0.95, 2.33, 2.43 [07:25:57] PROBLEM - cp33 Current Load on cp33 is CRITICAL: CRITICAL - load average: 4.07, 2.90, 2.27 [07:29:09] PROBLEM - cp32 Current Load on cp32 is WARNING: WARNING - load average: 3.16, 3.71, 3.06 [07:29:56]