[00:02:32] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.54, 4.98, 3.54 [00:02:43] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 27.78, 21.24, 17.52 [00:03:22] PROBLEM - wikiben.tk - ZeroSSL on sslhost is WARNING: WARNING - Certificate 'wikiben.tk' expires in 30 day(s) (Tue 23 Nov 2021 23:59:59 GMT +0000). [00:06:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.47, 5.39, 4.10 [00:07:28] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.61, 5.99, 4.85 [00:08:32] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 7.00, 6.00, 4.47 [00:09:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 4.65, 5.48, 4.81 [00:10:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.78, 5.94, 4.66 [00:12:32] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.96, 7.06, 5.22 [00:35:39] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.23, 5.98, 4.82 [00:37:39] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 5.36, 5.66, 4.84 [00:51:24] PROBLEM - ping6 on jobchron1 is WARNING: PING WARNING - Packet loss = 0%, RTA = 163.48 ms [00:53:27] RECOVERY - ping6 on jobchron1 is OK: PING OK - Packet loss = 0%, RTA = 0.71 ms [00:58:23] RECOVERY - bacula2 Bacula Private Git on bacula2 is OK: OK: Full, 6457 files, 49.67MB, 2021-10-24 00:58:00 (23.0 seconds ago) [01:08:55] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.40, 4.06, 3.50 [01:10:51] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.88, 3.57, 3.39 [01:16:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.36, 5.06, 5.88 [01:16:37] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.65, 3.25, 3.33 [01:18:43] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 23.41, 19.68, 23.93 [01:20:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.65, 3.33, 3.33 [01:24:32] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 1.82, 3.75, 5.02 [01:24:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.81, 3.34, 3.36 [01:28:43] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.07, 14.50, 19.34 [02:07:06] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.62, 3.75, 3.27 [02:09:02] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.20, 3.11, 3.09 [03:10:13] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JisqW [03:10:15] [02miraheze/mediawiki] 07Universal-Omega 03b14148e - Update Cosmos [03:12:05] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JisqX [03:12:06] [02miraheze/mediawiki] 07Universal-Omega 0328946b2 - Update Cosmos [03:13:55] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/Jisqb [03:13:57] [02miraheze/mediawiki] 07Universal-Omega 031035802 - Update MirahezeMagic [03:15:33] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JismL [03:15:34] [02miraheze/mediawiki] 07Universal-Omega 03c98a16c - Update MirahezeMagic [03:17:08] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/Jism0 [03:17:09] [02miraheze/mediawiki] 07Universal-Omega 03b88c23e - Update WikiDiscover [03:17:16] !log [universalomega@mw11] starting deploy of {'world': True, 'l10n': True, 'gitinfo': True} to all [03:17:19] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:18:20] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/Jismr [03:18:21] [02miraheze/mediawiki] 07Universal-Omega 0360aae50 - Update WikiDiscover [03:18:39] !log [universalomega@test3] starting deploy of {'world': True, 'l10n': True, 'gitinfo': True} to skip [03:18:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:28:54] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.02, 6.68, 5.10 [03:29:38] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.04, 3.72, 3.26 [03:30:54] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.96, 6.07, 5.06 [03:31:07] !log [universalomega@test3] finished deploy of {'world': True, 'l10n': True, 'gitinfo': True} to skip - SUCCESS in 747s [03:31:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:31:33] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.99, 3.15, 3.10 [03:33:55] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 9.42, 7.00, 5.11 [03:35:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 20.03, 10.62, 6.34 [03:36:02] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 17.41, 10.26, 6.20 [03:36:30] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:36:43] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 32.77, 29.43, 20.63 [03:40:32] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.47, 5.28, 3.64 [03:41:30] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [03:42:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.22, 5.49, 3.94 [03:43:07] !log [universalomega@mw11] finished deploy of {'world': True, 'l10n': True, 'gitinfo': True} to all - SUCCESS in 1550s [03:43:18] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [03:43:25] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 5.54, 7.99, 6.83 [03:43:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.07, 7.24, 6.54 [03:43:57] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 4.71, 7.41, 6.76 [03:44:09] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 21.31, 20.89, 17.92 [03:44:32] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.02, 4.77, 3.85 [03:45:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.94, 6.09, 6.20 [03:45:56] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.25, 6.47, 6.50 [03:46:09] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 13.56, 18.29, 17.34 [03:46:43] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 14.16, 21.81, 22.11 [03:47:13] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 2.58, 5.44, 6.07 [03:54:43] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 13.01, 17.51, 20.02 [04:05:19] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.98, 3.28, 3.03 [04:07:15] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.68, 3.21 [04:07:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.80, 6.82, 5.07 [04:09:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.64, 3.35, 3.15 [04:09:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.92, 5.69, 4.87 [04:18:50] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.94, 3.70, 3.40 [04:26:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.82, 3.14, 3.26 [05:11:56] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 44738 MB (10% inode=98%); [05:15:33] PROBLEM - wiki-asterix.cf - LetsEncrypt on sslhost is CRITICAL: CRITICAL - Certificate 'wiki-asterix.cf' expires in 7 day(s) (Mon 01 Nov 2021 05:09:56 GMT +0000). [05:20:57] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/Jis0P [05:20:59] [02miraheze/mw-config] 07Universal-Omega 0364188ec - Blacklist the `viewincidents` right in ManageWiki [05:22:05] miraheze/mw-config - Universal-Omega the build passed. [05:25:56] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 52533 MB (11% inode=98%); [05:29:56] PROBLEM - db12 Disk Space on db12 is WARNING: DISK WARNING - free space: / 45193 MB (10% inode=98%); [05:31:56] RECOVERY - db12 Disk Space on db12 is OK: DISK OK - free space: / 60063 MB (13% inode=98%); [05:33:11] !log [@test3] starting deploy of {'config': True} to skip [05:33:12] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [05:33:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:33:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:34:45] !log [@mw11] starting deploy of {'config': True} to all [05:34:51] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:34:59] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 14s [05:35:02] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [05:38:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.38, 4.54, 3.58 [05:40:32] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.06, 4.30, 3.61 [06:05:05] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.66, 5.61, 3.54 [06:06:15] [02miraheze/MirahezeMagic] 07Universal-Omega pushed 031 commit to 03Universal-Omega-patch-5 [+0/-0/±1] 13https://git.io/JisVS [06:06:16] [02miraheze/MirahezeMagic] 07Universal-Omega 0396b6bf2 - Add user to captcha debug logs [06:06:18] [02MirahezeMagic] 07Universal-Omega created branch 03Universal-Omega-patch-5 - 13https://git.io/fQRGX [06:06:19] [02MirahezeMagic] 07Universal-Omega opened pull request 03#301: Add user to captcha debug logs - 13https://git.io/JisV9 [06:07:05] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 8.26, 6.49, 4.12 [06:07:23] miraheze/MirahezeMagic - Universal-Omega the build passed. [06:09:05] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.41, 6.86, 4.55 [06:11:05] PROBLEM - db11 Current Load on db11 is CRITICAL: CRITICAL - load average: 9.29, 7.62, 5.10 [06:17:05] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 5.63, 7.18, 5.83 [06:21:05] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.97, 6.48, 5.87 [06:57:05] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.37, 6.78, 6.14 [06:59:05] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 4.97, 6.11, 5.98 [07:19:54] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.72, 6.74, 6.12 [07:21:52] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.29, 6.02, 5.92 [07:31:51] [02mediawiki] 07Universal-Omega closed pull request 03#3804: Revert "Add temporary HistoryBlobStub hack" - 13https://git.io/Ji3R4 [07:31:53] [02miraheze/mediawiki] 07Universal-Omega deleted branch 03revert-3802-Universal-Omega-patch-1 [07:31:54] [02mediawiki] 07Universal-Omega deleted branch 03revert-3802-Universal-Omega-patch-1 - 13https://git.io/vbL5b [07:31:56] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/Jisjq [07:31:57] [02miraheze/mediawiki] 07Universal-Omega 03a5f93e0 - Revert "Add temporary HistoryBlobStub hack (#3802)" (#3804) [08:35:05] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.19, 7.00, 6.48 [08:37:32] CosmicAlpha: no sync? [08:51:05] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 4.84, 6.35, 6.59 [09:20:00] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 6.64, 6.84, 6.66 [09:21:55] !log [universalomega@mw11] starting deploy of {'world': True, 'gitinfo': True} to all [09:21:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:21:59] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 5.49, 6.37, 6.51 [09:22:11] !log [universalomega@test3] starting deploy of {'world': True, 'gitinfo': True} to skip [09:22:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:23:07] Spookreeeno: forgot ^ done now. [09:25:51] !log [universalomega@test3] finished deploy of {'world': True, 'gitinfo': True} to skip - SUCCESS in 219s [09:25:54] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:27:12] !log [universalomega@mw11] finished deploy of {'world': True, 'gitinfo': True} to all - SUCCESS in 317s [09:27:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [09:27:53] PROBLEM - db11 Current Load on db11 is WARNING: WARNING - load average: 7.88, 6.88, 6.59 [09:29:51] RECOVERY - db11 Current Load on db11 is OK: OK - load average: 6.19, 6.64, 6.54 [10:16:47] PROBLEM - duepedia.uk.to - LetsEncrypt on sslhost is WARNING: WARNING - Certificate 'duepedia.uk.to' expires in 15 day(s) (Tue 09 Nov 2021 10:14:23 GMT +0000). [10:24:39] [02mw-config] 07JohnFLewis commented on commit 0364188ecf453e1801904f6948e6591badd92bc50b - 13https://git.io/JiG88 [10:27:03] [02mw-config] 07JohnFLewis commented on commit 0364188ecf453e1801904f6948e6591badd92bc50b - 13https://git.io/JiG8P [12:51:26] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.92, 3.00, 2.71 [12:53:21] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.41, 2.70, 2.63 [13:02:43] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.94, 19.16, 16.46 [13:02:47] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.34, 4.58, 3.60 [13:04:43] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.10, 4.73, 3.80 [13:04:57] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.96, 3.37, 2.95 [13:06:52] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.10, 2.90, 2.83 [13:10:43] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.15, 18.18, 17.45 [13:22:02] !log [reception@mwtask1] sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=christianrootsofnazismwiki /home/reception/religiosityofnazism_pages_full.xml --username-prefix wikia:religiosity-of-nazism --report 1 (END - exit=0) [13:22:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [13:58:37] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 5.92, 4.35, 3.40 [14:00:33] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 3.38, 3.77, 3.30 [14:34:15] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 38.86, 25.54, 19.17 [14:34:32] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 13.45, 7.80, 4.76 [14:36:53] PROBLEM - ping4 on jobchron1 is CRITICAL: PING CRITICAL - Packet loss = 0%, RTA = 526.41 ms [14:38:04] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 18.30, 23.19, 19.80 [14:38:51] RECOVERY - ping4 on jobchron1 is OK: PING OK - Packet loss = 0%, RTA = 3.32 ms [14:41:54] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 25.00, 22.93, 20.34 [14:42:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.26, 3.72, 3.21 [14:44:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.61, 3.56, 3.21 [14:45:43] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.64, 21.17, 20.27 [14:46:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.09, 3.28, 3.14 [14:46:32] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.62, 5.00, 5.03 [14:47:38] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.11, 18.60, 19.44 [14:50:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.10, 3.74, 3.35 [14:52:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.35, 3.15, 3.18 [15:23:02] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.75, 3.06, 2.83 [15:24:58] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.81, 3.17, 2.91 [15:28:50] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.55, 3.44, 3.09 [15:30:45] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.13, 2.99, 2.97 [15:50:59] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.31, 3.92, 3.34 [15:52:55] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.56, 3.63, 3.29 [15:54:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.63, 3.82, 3.39 [15:56:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.59, 3.72, 3.40 [16:02:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.35, 3.77, 3.50 [16:10:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.76, 3.94, 3.76 [16:12:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.88, 4.24, 3.89 [16:14:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.84, 3.78 [16:18:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.05, 3.72, 3.72 [16:22:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.79, 3.56, 3.65 [16:30:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.89, 2.99, 3.35 [16:50:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.27, 3.43, 3.15 [16:52:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.30, 3.57, 3.25 [16:53:13] [02mw-config] 07Universal-Omega commented on commit 0364188ecf453e1801904f6948e6591badd92bc50b - 13https://git.io/JiZsR [16:54:38] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiZsw [16:54:39] [02miraheze/mw-config] 07Universal-Omega 03858db42 - Remove the `viewincidents` right from ManageWiki blacklist [16:55:46] miraheze/mw-config - Universal-Omega the build passed. [17:00:22] JohnLewis: ^ apologies for that misunderstanding. Reverted. [17:02:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.20, 2.96, 3.17 [17:03:02] !log [@test3] starting deploy of {'config': True} to skip [17:03:03] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [17:03:07] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:03:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:04:36] !log [@mw11] starting deploy of {'config': True} to all [17:04:42] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:05:08] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 32s [17:05:15] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [17:09:30] [02miraheze/mediawiki] 07Universal-Omega pushed 032 commits to 03REL1_37 [+0/-0/±2] 13https://git.io/JiZZ9 [17:09:32] [02miraheze/mediawiki] 07Universal-Omega 03c948ea4 - checkStorage: pass no parameters to WikiRevision::getContent() [17:09:33] [02miraheze/mediawiki] 07Universal-Omega 03f0ac622 - Merge branch 'wikimedia:REL1_37' into REL1_37 [17:10:16] CosmicAlpha: no problem [17:10:16] [tell] JohnLewis: 2021-10-21 - 08:11:21UTC tell JohnLewis any ideas on https://phabricator.miraheze.org/T8184#165279? I'm out of ideas with this script. I don't what to do now. That happened on 3/3 wikis I tried it on. [17:10:58] Disregard that. Figured it out. [17:12:43] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 28.75, 23.81, 19.03 [17:14:52] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 9.35, 6.39, 4.24 [17:21:32] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.31, 3.77, 2.98 [17:23:32] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 1.61, 3.04, 2.82 [17:35:32] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.97, 3.42, 2.97 [17:37:32] PROBLEM - graylog2 Current Load on graylog2 is CRITICAL: CRITICAL - load average: 4.16, 3.56, 3.07 [17:39:33] PROBLEM - graylog2 Current Load on graylog2 is WARNING: WARNING - load average: 3.55, 3.60, 3.15 [17:41:47] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiZWF [17:41:48] [02miraheze/mw-config] 07Universal-Omega 032f4a73a - Disable SQLBagOStuff logging [17:42:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.39, 3.27, 2.79 [17:42:59] miraheze/mw-config - Universal-Omega the build passed. [17:43:32] RECOVERY - graylog2 Current Load on graylog2 is OK: OK - load average: 2.80, 3.33, 3.16 [17:44:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.23, 3.39, 2.91 [17:56:48] [02miraheze/mw-config] 07dependabot[bot] pushed 031 commit to 03dependabot/composer/mediawiki/mediawiki-codesniffer-38.0.0 [+0/-0/±1] 13https://git.io/JiZ8D [17:56:49] [02miraheze/mw-config] 07dependabot[bot] 0330ca339 - Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 [17:56:51] [02mw-config] 07dependabot[bot] opened pull request 03#4175: Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 - 13https://git.io/JiZ8y [17:56:52] [02mw-config] 07dependabot[bot] created branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-38.0.0 - 13https://git.io/vbvb3 [17:56:54] [02mw-config] 07dependabot[bot] labeled pull request 03#4175: Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 - 13https://git.io/JiZ8y [17:56:55] [02mw-config] 07dependabot[bot] labeled pull request 03#4175: Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 - 13https://git.io/JiZ8y [17:57:48] miraheze/mw-config - dependabot[bot] the build passed. [17:58:08] [02mw-config] 07Universal-Omega closed pull request 03#4175: Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 - 13https://git.io/JiZ8y [17:58:10] [02miraheze/mw-config] 07Universal-Omega deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-38.0.0 [17:58:11] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiZ8x [17:58:13] [02miraheze/mw-config] 07dependabot[bot] 03e6f3b1d - Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 (#4175) [17:58:14] [02mw-config] 07Universal-Omega deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-38.0.0 - 13https://git.io/vbvb3 [17:58:53] [02CreateWiki] 07Universal-Omega closed pull request 03#252: Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 - 13https://git.io/Jim1x [17:58:55] [02miraheze/CreateWiki] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiZ4J [17:58:57] [02miraheze/CreateWiki] 07dependabot[bot] 036377b19 - Update mediawiki/mediawiki-codesniffer requirement from 37.0.0 to 38.0.0 (#252) [17:58:58] [02miraheze/CreateWiki] 07Universal-Omega deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-38.0.0 [17:58:59] [02CreateWiki] 07Universal-Omega deleted branch 03dependabot/composer/mediawiki/mediawiki-codesniffer-38.0.0 - 13https://git.io/vpJTL [17:59:12] miraheze/mw-config - Universal-Omega the build passed. [18:02:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.17, 4.45, 6.00 [18:03:56] !log [@test3] starting deploy of {'config': True} to skip [18:03:57] !log [@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [18:04:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:04:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:04:22] !log [@mw11] starting deploy of {'config': True} to all [18:04:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:04:34] !log [@mw11] finished deploy of {'config': True} to all - SUCCESS in 12s [18:04:38] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [18:07:08] miraheze/CreateWiki - Universal-Omega the build passed. [18:08:32] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 6.96, 5.13, 5.68 [18:10:13] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.37, 3.86, 3.35 [18:10:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.13, 4.59, 5.43 [18:12:08] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.54, 3.72, 3.36 [18:14:32] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.82, 3.97, 5.02 [18:14:43] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.65, 19.86, 23.93 [18:19:50] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.09, 3.76, 3.51 [18:21:46] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.43, 3.49, 3.45 [18:23:41] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.56, 3.07, 3.30 [18:28:43] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 15.31, 16.94, 19.95 [18:34:32] PROBLEM - gluster3 Current Load on gluster3 is CRITICAL: CRITICAL - load average: 8.53, 6.92, 5.36 [18:34:43] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 42.31, 28.63, 23.53 [18:42:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 3.65, 5.41, 5.35 [18:42:43] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 16.30, 21.78, 22.62 [18:46:32] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 4.27, 4.66, 5.05 [18:48:43] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 29.43, 22.96, 22.62 [18:50:32] PROBLEM - gluster3 Current Load on gluster3 is WARNING: WARNING - load average: 4.71, 5.31, 5.25 [18:52:32] RECOVERY - gluster3 Current Load on gluster3 is OK: OK - load average: 2.18, 4.22, 4.86 [18:52:43] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 17.92, 21.05, 22.04 [18:58:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.73, 3.67, 3.38 [19:02:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.41, 3.04, 3.18 [19:06:43] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 26.06, 21.44, 21.15 [19:08:43] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 19.52, 20.59, 20.87 [19:10:43] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.16, 22.11, 21.39 [19:12:43] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 15.03, 19.26, 20.42 [19:14:43] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 12.30, 16.72, 19.34 [19:20:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.61, 3.56, 3.20 [19:22:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.76, 3.16, 3.09 [19:26:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.71, 3.78, 3.35 [19:28:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.95, 3.54, 3.32 [19:30:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.01, 3.31, 3.26 [19:34:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.24, 3.57, 3.39 [19:42:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 2.73, 3.15, 3.31 [19:53:15] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.74, 3.50, 3.40 [19:54:13] JohnLewis: I've discovered that somehow on the 22nd we had 140k views and 23rd it dropped to 38k so I feel like something went wrong with Matomo [19:55:10] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 3.23, 3.38, 3.37 [19:55:31] JohnLewis: oh definitely something wrong, today is showing 0 views [19:55:36] and "It looks like reports for this period have not been processed yet. Do you want to see what's happening now? Check out the Visits log or choose a different date period until the reports are generated." [19:55:50] cc paladox [19:56:15] ok, guess the cron failed? [19:57:29] !log run /usr/local/bin/runMatomoArchive.sh - mon2 [19:57:34] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [19:57:54] it's weird because real time visitors is working [19:58:04] CosmicAlpha.... Any news about my PR adjustment [19:58:37] but this is the graph https://usercontent.irccloud-cdn.com/file/jf3c3Vku/image.png [19:59:02] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.88, 4.04, 3.68 [20:02:41] first ones fixed [20:02:44] the 2nd will soon [20:04:02] thanks! [20:04:43] PROBLEM - cloud4 Current Load on cloud4 is CRITICAL: CRITICAL - load average: 24.10, 19.39, 17.23 [20:06:43] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 14.86, 17.38, 16.75 [20:08:05] Reception123: data [20:08:14] For today won't work while midnight [20:35:07] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 19.96, 10.41, 6.69 [20:35:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 8.31, 6.71, 5.20 [20:35:29] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 7.96, 6.91, 5.26 [20:37:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.77, 5.66, 5.01 [20:37:29] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 4.59, 5.89, 5.09 [20:41:06] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.69, 7.47, 6.74 [20:45:06] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 5.28, 6.33, 6.44 [20:46:49] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.25, 6.77, 6.08 [20:52:49] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 8.13, 7.51, 6.63 [20:54:09] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 20.69, 19.37, 17.75 [20:58:09] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 14.19, 17.81, 17.58 [20:58:49] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 4.25, 6.52, 6.60 [21:11:06] PROBLEM - mw12 Current Load on mw12 is CRITICAL: CRITICAL - load average: 8.16, 6.31, 5.61 [21:12:43] PROBLEM - cloud4 Current Load on cloud4 is WARNING: WARNING - load average: 21.57, 19.08, 17.80 [21:13:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 7.72, 6.28, 5.42 [21:13:30] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 8.36, 7.05, 5.59 [21:15:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 6.36, 6.26, 5.52 [21:16:43] RECOVERY - cloud4 Current Load on cloud4 is OK: OK - load average: 19.80, 19.61, 18.30 [21:17:40] PROBLEM - cp12 Stunnel Http for mw12 on cp12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.246 second response time [21:17:55] PROBLEM - cp13 Stunnel Http for mw12 on cp13 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.011 second response time [21:18:01] PROBLEM - cp14 Stunnel Http for mw12 on cp14 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.011 second response time [21:18:16] PROBLEM - mw12 MediaWiki Rendering on mw12 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.008 second response time [21:18:21] PROBLEM - cp15 Varnish Backends on cp15 is CRITICAL: 1 backends are down. mw12 [21:18:50] PROBLEM - cp12 Varnish Backends on cp12 is CRITICAL: 1 backends are down. mw12 [21:19:06] PROBLEM - mw12 Current Load on mw12 is WARNING: WARNING - load average: 4.42, 7.60, 6.95 [21:19:08] PROBLEM - cp15 Stunnel Http for mw12 on cp15 is CRITICAL: HTTP CRITICAL: HTTP/1.1 502 Bad Gateway - 344 bytes in 0.271 second response time [21:19:27] PROBLEM - mw8 Current Load on mw8 is CRITICAL: CRITICAL - load average: 14.80, 10.06, 7.14 [21:19:41] PROBLEM - cp14 Varnish Backends on cp14 is CRITICAL: 1 backends are down. mw12 [21:20:00] alerting : [FIRING:1] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:20:04] PROBLEM - cp13 Varnish Backends on cp13 is CRITICAL: 1 backends are down. mw12 [21:21:06] RECOVERY - mw12 Current Load on mw12 is OK: OK - load average: 3.00, 6.02, 6.44 [21:24:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.54, 3.46, 3.94 [21:25:00] ok : [RESOLVED] (PHP-FPM Worker Usage High mediawiki) https://grafana.miraheze.org/d/dsHv5-4nz/mediawiki [21:29:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 4.00, 7.83, 7.80 [21:32:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 3.86, 3.99, 4.02 [21:32:54] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.99, 6.44, 5.20 [21:33:21] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.82, 8.07, 6.24 [21:33:34] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 12.25, 8.58, 6.03 [21:33:45] PROBLEM - wiki.tigcity.tk - reverse DNS on sslhost is WARNING: rDNS WARNING - reverse DNS entry for wiki.tigcity.tk could not be found [21:34:54] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 9.19, 7.65, 5.82 [21:35:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 2.73, 4.95, 6.58 [21:36:54] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.57, 7.13, 5.85 [21:37:12] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.76, 7.44, 6.45 [21:37:29] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 3.80, 5.47, 7.61 [21:37:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.78, 7.66, 6.31 [21:38:54] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.42, 6.17, 5.66 [21:39:07] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.31, 6.66, 6.27 [21:39:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.22, 6.45, 6.03 [21:41:29] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.27, 4.63, 6.78 [21:47:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.02, 6.73, 6.27 [21:49:34] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 6.70, 6.64, 6.29 [21:50:27] [02miraheze/ManageWiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JiZML [21:50:29] [02miraheze/ManageWiki] 07RhinosF1 03f1bdb11 - Update ManageWikiFormFactoryBuilder.php [21:50:30] [02ManageWiki] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/vpSns [21:50:40] [02ManageWiki] 07RhinosF1 opened pull request 03#307: Add an experimental option - 13https://git.io/JiZMt [21:51:37] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.84, 6.76, 6.30 [21:53:32] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.49, 6.71, 6.34 [21:54:11] miraheze/ManageWiki - RhinosF1 the build has errored. [21:55:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.93, 7.51, 6.70 [21:57:23] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.78, 7.43, 6.74 [21:58:41] [02miraheze/CreateWiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JiZMa [21:58:42] [02miraheze/CreateWiki] 07RhinosF1 03ef4fc6e - Update RemoteWiki.php [21:58:44] [02CreateWiki] 07RhinosF1 created branch 03RhinosF1-patch-1 - 13https://git.io/vpJTL [21:58:54] [02CreateWiki] 07RhinosF1 opened pull request 03#253: Add experimental flag - 13https://git.io/JiZMw [21:59:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.79, 6.78, 6.61 [22:00:30] [02miraheze/CreateWiki] 07github-actions[bot] pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JiZMi [22:00:31] [02miraheze/CreateWiki] 07github-actions 03a343115 - CI: lint code to MediaWiki standards [22:00:33] [02CreateWiki] 07github-actions[bot] synchronize pull request 03#253: Add experimental flag - 13https://git.io/JiZMw [22:01:16] [02miraheze/CreateWiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JiZMD [22:01:18] [02miraheze/CreateWiki] 07RhinosF1 03503e573 - Update CreateWikiJson.php [22:01:19] [02CreateWiki] 07RhinosF1 synchronize pull request 03#253: Add experimental flag - 13https://git.io/JiZMw [22:02:25] [02miraheze/CreateWiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JiZMH [22:02:27] [02miraheze/CreateWiki] 07RhinosF1 03beac90e - Update WikiInitialise.php [22:02:28] [02CreateWiki] 07RhinosF1 synchronize pull request 03#253: Add experimental flag - 13https://git.io/JiZMw [22:03:08] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 10.93, 7.94, 7.07 [22:03:21] [02miraheze/CreateWiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JiZMF [22:03:22] [02miraheze/CreateWiki] 07RhinosF1 03cb143f8 - Update cw_wikis.sql [22:03:24] [02CreateWiki] 07RhinosF1 synchronize pull request 03#253: Add experimental flag - 13https://git.io/JiZMw [22:03:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 9.31, 7.94, 7.10 [22:04:35] [02miraheze/CreateWiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+1/-0/±0] 13https://git.io/JiZMN [22:04:37] [02miraheze/CreateWiki] 07RhinosF1 036d88018 - Create patch-cw_wikis-add-cw_experimental.sql [22:04:38] [02CreateWiki] 07RhinosF1 synchronize pull request 03#253: Add experimental flag - 13https://git.io/JiZMw [22:05:37] [02miraheze/CreateWiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JiZMj [22:05:39] [02miraheze/CreateWiki] 07RhinosF1 03af20a3f - Update extension.json [22:05:40] [02CreateWiki] 07RhinosF1 synchronize pull request 03#253: Add experimental flag - 13https://git.io/JiZMw [22:06:15] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.70, 7.24, 6.52 [22:06:20] [02miraheze/CreateWiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JiZDv [22:06:22] [02miraheze/CreateWiki] 07RhinosF1 039d5acd9 - Update patch-cw_wikis-add-cw_experimental.sql [22:06:23] [02CreateWiki] 07RhinosF1 synchronize pull request 03#253: Add experimental flag - 13https://git.io/JiZMw [22:07:04] CosmicAlpha: ^ [22:07:24] miraheze/CreateWiki - RhinosF1 the build has errored. [22:08:51] miraheze/CreateWiki - RhinosF1 the build has errored. [22:08:54] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.23, 7.90, 7.40 [22:09:10] [02miraheze/CreateWiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JiZD3 [22:09:11] [02miraheze/CreateWiki] 07RhinosF1 0335fa485 - Update cw_wikis.sql [22:09:13] [02CreateWiki] 07RhinosF1 synchronize pull request 03#253: Add experimental flag - 13https://git.io/JiZMw [22:09:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.40, 7.94, 7.51 [22:10:16] miraheze/CreateWiki - RhinosF1 the build has errored. [22:10:22] miraheze/CreateWiki - RhinosF1 the build has errored. [22:12:04] miraheze/CreateWiki - RhinosF1 the build has errored. [22:14:32] [02miraheze/CreateWiki] 07RhinosF1 pushed 031 commit to 03RhinosF1-patch-1 [+0/-0/±1] 13https://git.io/JiZD2 [22:14:33] [02miraheze/CreateWiki] 07RhinosF1 038db4d74 - Update globals.php [22:14:35] [02CreateWiki] 07RhinosF1 synchronize pull request 03#253: Add experimental flag - 13https://git.io/JiZMw [22:15:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.32, 7.71, 7.45 [22:15:43] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.60, 6.52, 6.59 [22:17:06] miraheze/CreateWiki - RhinosF1 the build has errored. [22:17:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 7.06, 7.62, 7.46 [22:18:02] [02ManageWiki] 07RhinosF1 edited pull request 03#307: Add an experimental option - 13https://git.io/JiZMt [22:18:49] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.61, 6.27, 6.74 [22:21:33] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 8.12, 7.53, 7.43 [22:21:40] JohnLewis or paladox: is it safe to run the sql from https://github.com/miraheze/CreateWiki/pull/253/files to see if I can do some work on it from test3? [22:21:41] [url] Add experimental flag by RhinosF1 · Pull Request #253 · miraheze/CreateWiki · GitHub | github.com [22:22:05] I assume prod will just ignore the column it doesn't yet know about [22:22:21] yes [22:23:17] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.04, 7.08, 6.77 [22:23:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.69, 7.39, 7.41 [22:24:49] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 5.62, 6.62, 6.81 [22:25:11] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.56, 6.50, 6.59 [22:26:40] !log > ALTER TABLE cw_wikis ADD COLUMN cw_experimental SMALLINT NOT NULL DEFAULT '0'; on mhglobal [22:26:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:26:49] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 6.73, 6.67, 6.80 [22:27:07] !log [rhinos@test3] starting deploy of {'folders': 'w/extensions/ManageWiki'} to skip [22:27:08] !log [rhinos@test3] finished deploy of {'folders': 'w/extensions/ManageWiki'} to skip - SUCCESS in 0s [22:27:10] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:27:14] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:27:38] !log [rhinos@test3] starting deploy of {'world': True} to skip [22:27:43] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:31:13] !log [rhinos@test3] finished deploy of {'world': True} to skip - SUCCESS in 215s [22:31:17] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [22:32:35] Spookreeeno: I think we can do this without another column, with a bit of SiteConfiguration stuff. Or we could do it like I did the description, not have much of anything in CreateWiki changes, but rather a config that ManageWikiCore sets through ManageWikiSettings, and can be accessed that way. [22:35:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 5.93, 6.15, 6.79 [22:35:44] CosmicAlpha: it makes sense for it to be treated like any other cw variable in my head [22:36:54] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 6.48, 7.02, 6.78 [22:38:54] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 5.98, 6.54, 6.62 [22:40:45] Spookreeeno: Yeah the more I think about it yeah, but how are you planning to use $cwExpermental? The way CW is now, using that is no different then a manual list variable in LS (as you'd have to add manual if statements, checking `if ($cwExperemental){$wgConfigExample = 'example'}`, unless we add it as a possible tag of SiteConfiguration. I guess how it is now is a bit better then manually maintained list, but still adding to [22:40:45] SiteConfiguration would be best I think. [22:41:13] CosmicAlpha: same way we have the closed & private config [22:43:52] CosmicAlpha: patch doesn't work yet though [22:43:55] https://www.irccloud.com/pastebin/otG0K4L5/ [22:46:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.80, 3.53, 3.87 [22:51:27] * Spookreeeno is off for night [22:52:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.82, 4.32, 4.12 [23:03:32] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_36 [+0/-0/±1] 13https://git.io/JiZHB [23:03:33] [02miraheze/mediawiki] 07Universal-Omega 031547453 - Update Cosmos [23:04:08] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 10.34, 7.90, 6.49 [23:05:27] PROBLEM - mw8 Current Load on mw8 is WARNING: WARNING - load average: 6.89, 6.23, 4.62 [23:05:29] PROBLEM - mw9 Current Load on mw9 is CRITICAL: CRITICAL - load average: 16.22, 9.40, 5.60 [23:05:39] [02miraheze/mediawiki] 07Universal-Omega pushed 031 commit to 03REL1_37 [+0/-0/±1] 13https://git.io/JiZHr [23:05:41] [02miraheze/mediawiki] 07Universal-Omega 0322fc3da - Update Cosmos [23:06:08] PROBLEM - mw9 APT on mw9 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:06:49] PROBLEM - mw13 Current Load on mw13 is CRITICAL: CRITICAL - load average: 11.70, 7.43, 6.08 [23:06:54] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 14.69, 9.58, 6.97 [23:07:04] PROBLEM - mw10 APT on mw10 is CRITICAL: CHECK_NRPE STATE CRITICAL: Socket timeout after 10 seconds. [23:07:27] RECOVERY - mw8 Current Load on mw8 is OK: OK - load average: 3.44, 5.38, 4.52 [23:07:29] PROBLEM - mw9 Current Load on mw9 is WARNING: WARNING - load average: 5.38, 7.69, 5.45 [23:08:05] RECOVERY - mw9 APT on mw9 is OK: APT OK: 0 packages available for upgrade (0 critical updates). [23:08:09] PROBLEM - cloud5 Current Load on cloud5 is CRITICAL: CRITICAL - load average: 25.05, 21.33, 17.55 [23:08:50] PROBLEM - mw13 Current Load on mw13 is WARNING: WARNING - load average: 7.89, 7.56, 6.30 [23:09:00] RECOVERY - mw10 APT on mw10 is OK: APT OK: 16 packages available for upgrade (0 critical updates). [23:09:29] RECOVERY - mw9 Current Load on mw9 is OK: OK - load average: 3.60, 6.28, 5.20 [23:10:09] PROBLEM - cloud5 Current Load on cloud5 is WARNING: WARNING - load average: 19.40, 20.52, 17.73 [23:10:49] RECOVERY - mw13 Current Load on mw13 is OK: OK - load average: 5.23, 6.54, 6.07 [23:12:09] RECOVERY - cloud5 Current Load on cloud5 is OK: OK - load average: 16.95, 19.43, 17.69 [23:12:54] PROBLEM - mw11 Current Load on mw11 is WARNING: WARNING - load average: 7.42, 7.99, 7.15 [23:14:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 2.45, 3.32, 3.85 [23:14:49] !log [universalomega@test3] starting deploy of {'folders': 'w/skins/Cosmos'} to skip [23:14:50] !log [universalomega@test3] finished deploy of {'folders': 'w/skins/Cosmos'} to skip - SUCCESS in 0s [23:14:58] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:15:04] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:15:52] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 6.84, 7.75, 7.88 [23:16:21] !log [universalomega@mw11] starting deploy of {'folders': 'w/skins/Cosmos'} to all [23:16:24] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:16:29] !log [universalomega@mw11] DEPLOY ABORTED: Canary check failed for mw12 [23:16:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.94, 4.10, 4.08 [23:16:32] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:18:20] paladox: mw12 is down? Looks like it has been for 2 hours. [23:18:29] Oh [23:18:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.33, 3.83, 3.99 [23:18:54] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 3.88, 5.68, 6.43 [23:19:10] !log reboot mw12 [23:19:13] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:20:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 4.64, 4.16, 4.09 [23:21:03] PROBLEM - mw12 Check Gluster Clients on mw12 is CRITICAL: PROCS CRITICAL: 0 processes with args '/usr/sbin/glusterfs' [23:21:08] RECOVERY - cp15 Stunnel Http for mw12 on cp15 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.371 second response time [23:21:32] RECOVERY - cp12 Stunnel Http for mw12 on cp12 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.311 second response time [23:21:41] RECOVERY - cp14 Varnish Backends on cp14 is OK: All 9 backends are healthy [23:21:41] thanks paladox! [23:21:49] RECOVERY - cp13 Stunnel Http for mw12 on cp13 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.034 second response time [23:22:01] !log [universalomega@test3] starting deploy of {'folders': 'w/skins/Cosmos'} to skip [23:22:01] RECOVERY - cp14 Stunnel Http for mw12 on cp14 is OK: HTTP OK: HTTP/1.1 200 OK - 15861 bytes in 0.005 second response time [23:22:02] !log [universalomega@test3] finished deploy of {'folders': 'w/skins/Cosmos'} to skip - SUCCESS in 0s [23:22:04] RECOVERY - cp13 Varnish Backends on cp13 is OK: All 9 backends are healthy [23:22:05] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:08] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:16] !log [universalomega@mw11] starting deploy of {'folders': 'w/skins/Cosmos'} to all [23:22:16] RECOVERY - mw12 MediaWiki Rendering on mw12 is OK: HTTP OK: HTTP/1.1 200 OK - 20011 bytes in 0.168 second response time [23:22:20] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:22] RECOVERY - cp15 Varnish Backends on cp15 is OK: All 9 backends are healthy [23:22:23] !log [universalomega@mw11] finished deploy of {'folders': 'w/skins/Cosmos'} to all - SUCCESS in 7s [23:22:26] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:22:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.24, 3.83, 3.98 [23:22:50] RECOVERY - cp12 Varnish Backends on cp12 is OK: All 9 backends are healthy [23:23:03] RECOVERY - mw12 Check Gluster Clients on mw12 is OK: PROCS OK: 1 process with args '/usr/sbin/glusterfs' [23:23:40] PROBLEM - mw10 Current Load on mw10 is CRITICAL: CRITICAL - load average: 11.10, 6.89, 7.11 [23:24:32] PROBLEM - mon2 Current Load on mon2 is CRITICAL: CRITICAL - load average: 5.05, 4.12, 4.06 [23:25:37] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 4.50, 5.87, 6.71 [23:25:53] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiZQj [23:25:55] [02miraheze/mw-config] 07Universal-Omega 03795a01b - add `$wgCosmosContentWidth` [23:26:32] PROBLEM - mon2 Current Load on mon2 is WARNING: WARNING - load average: 3.24, 3.70, 3.90 [23:27:02] miraheze/mw-config - Universal-Omega the build passed. [23:29:00] [02miraheze/mw-config] 07Universal-Omega pushed 031 commit to 03master [+0/-0/±1] 13https://git.io/JiZ7Z [23:29:01] [02miraheze/mw-config] 07Universal-Omega 037d049e1 - Add `$wgCosmosContentWidth` [23:30:06] !log [universalomega@test3] starting deploy of {'config': True} to skip [23:30:06] miraheze/mw-config - Universal-Omega the build passed. [23:30:07] !log [universalomega@test3] finished deploy of {'config': True} to skip - SUCCESS in 0s [23:30:09] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:12] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:41] !log [universalomega@mw11] starting deploy of {'config': True} to all [23:30:44] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:30:47] !log [universalomega@mw11] finished deploy of {'config': True} to all - SUCCESS in 6s [23:30:50] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:36:54] PROBLEM - mw11 Current Load on mw11 is CRITICAL: CRITICAL - load average: 10.86, 8.03, 6.24 [23:37:33] PROBLEM - mw10 Current Load on mw10 is WARNING: WARNING - load average: 5.82, 7.08, 6.52 [23:38:54] RECOVERY - mw11 Current Load on mw11 is OK: OK - load average: 4.57, 6.64, 5.95 [23:39:33] RECOVERY - mw10 Current Load on mw10 is OK: OK - load average: 3.68, 5.97, 6.18 [23:44:24] !log [universalomega@test3] DEPLOY ABORTED: Canary check failed for localhost [23:44:27] Logged the message at https://meta.miraheze.org/wiki/Tech:Server_admin_log [23:44:32] RECOVERY - mon2 Current Load on mon2 is OK: OK - load average: 1.97, 2.84, 3.27 [23:55:17] [02mw-config] 07Universal-Omega closed pull request 03#4142: Set PHP time limit from PhpAutoPrepend - 13https://git.io/JoxrS [23:55:18] [02mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1 - 13https://git.io/vbvb3 [23:55:20] [02miraheze/mw-config] 07Universal-Omega deleted branch 03Universal-Omega-patch-1