[05:40:42] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_38) - 10https://gerrit.wikimedia.org/r/815486 (owner: 10L10n-bot) [06:36:40] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/815641 (owner: 10L10n-bot) [07:19:52] (03CR) 10Raimond Spekking: [C: 03+2] "false positive" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/815641 (owner: 10L10n-bot) [12:00:03] (03CR) 10Esanders: [C: 03+1] Banner editor: use mw.notify [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/808391 (https://phabricator.wikimedia.org/T262722) (owner: 10Tacsipacsi) [13:47:16] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14656 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [13:52:16] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14956 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [13:57:14] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15256 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [14:02:14] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15556 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [14:07:18] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 15856 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [14:12:14] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 16156 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [14:17:14] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14763 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [14:22:14] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 13624 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [14:27:14] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 12464 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [14:28:39] (03CR) 10Tacsipacsi: [C: 04-1] "I just realized that I forgot to clean up the now-unused messages. I hope to do it today or tomorrow, but the patch should not be merged b" [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/808391 (https://phabricator.wikimedia.org/T262722) (owner: 10Tacsipacsi) [14:32:14] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11463 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [14:37:14] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10438 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [14:42:14] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9000 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [14:47:14] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7021 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [14:52:14] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5038 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [14:57:14] PROBLEM - check_mysql on frdb1002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3002 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [15:07:14] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 157258 Threads: 11 Questions: 48294618 Slow queries: 85 Opens: 77101746 Flush tables: 1 Open tables: 200 Queries per second avg: 307.104 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [17:51:42] whew, replag [19:32:17] PROBLEM - check_apache2 on payments1005 is CRITICAL: PROCS CRITICAL: 257 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_apache2 [19:32:23] PROBLEM - check_load on payments1005 is CRITICAL: CRITICAL - load average: 94.62, 60.28, 31.29 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_load [19:33:19] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 52683 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 65 days 5 hours - memory use is 45.03M (peak 50.22M, 0.54% of max, fragmentation 1.01%), connected_slaves is 3, donations is 13, jobs is 0, jobs-adyen is 4, jobs-paypal is 11, payments-init is 7, pending is 0, recurring is 4, refund is 0, unsubscribe is 13 https://icinga.wik [19:33:19] g/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [19:35:13] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 16.01, 42.54, 28.31 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [19:37:27] PROBLEM - check_apache2 on payments1005 is CRITICAL: PROCS CRITICAL: 257 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_apache2 [19:37:27] PROBLEM - check_load on payments1005 is CRITICAL: CRITICAL - load average: 104.47, 88.19, 51.47 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_load [19:38:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 65818 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 65 days 5 hours - memory use is 38.28M (peak 52.19M, 0.59% of max, fragmentation 1.24%), connected_slaves is 3, donations is 10, jobs is 0, jobs-adyen is 1, jobs-paypal is 10, payments-init is 79, pending is 0, recurring is 10, refund is 0, unsubscribe is 14 https://icinga.w [19:38:17] org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [19:40:23] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 86.32, 49.02, 33.06 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [19:42:19] PROBLEM - check_apache2 on payments1005 is CRITICAL: PROCS CRITICAL: 257 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_apache2 [19:42:23] PROBLEM - check_load on payments1005 is CRITICAL: CRITICAL - load average: 95.67, 89.12, 61.98 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_load [19:43:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 105339 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 65 days 5 hours - memory use is 67.28M (peak 78.88M, 0.85% of max, fragmentation 1.01%), connected_slaves is 3, donations is 44, jobs is 0, jobs-adyen is 12, jobs-paypal is 3, payments-init is 7, pending is 0, recurring is 3, refund is 0, unsubscribe is 16 https://icinga.wi [19:43:17] rg/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [19:45:13] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 22.22, 48.34, 39.17 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [19:47:13] PROBLEM - check_apache2 on payments1005 is CRITICAL: PROCS CRITICAL: 252 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_apache2 [19:47:13] PROBLEM - check_load on payments1005 is CRITICAL: CRITICAL - load average: 91.76, 97.82, 73.55 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_load [19:48:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 131074 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 65 days 6 hours - memory use is 86.03M (peak 95.79M, 1.08% of max, fragmentation 1.03%), connected_slaves is 3, donations is 13, jobs is 0, jobs-adyen is 5, jobs-paypal is 8, payments-init is 80, pending is 0, recurring is 4, refund is 0, unsubscribe is 17 https://icinga.wi [19:48:17] rg/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [19:50:23] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 97.37, 63.51, 46.20 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [19:52:17] PROBLEM - check_apache2 on payments1005 is CRITICAL: PROCS CRITICAL: 257 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_apache2 [19:52:21] PROBLEM - check_load on payments1005 is CRITICAL: CRITICAL - load average: 97.63, 90.32, 76.66 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_load [19:53:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 169215 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 65 days 6 hours - memory use is 113.27M (peak 117.56M, 1.32% of max, fragmentation 0.94%), connected_slaves is 3, donations is 13, jobs is 0, jobs-adyen is 1, jobs-paypal is 5, payments-init is 7, pending is 0, recurring is 2, refund is 0, unsubscribe is 20 https://icinga.w [19:53:17] org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [19:55:19] PROBLEM - check_apache2 on payments1007 is CRITICAL: PROCS CRITICAL: 257 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_apache2 [19:55:23] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 105.09, 78.07, 56.35 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [19:57:27] PROBLEM - check_apache2 on payments1005 is CRITICAL: PROCS CRITICAL: 257 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_apache2 [19:57:29] PROBLEM - check_load on payments1005 is CRITICAL: CRITICAL - load average: 158.40, 99.27, 82.55 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_load [19:58:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 162567 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 65 days 6 hours - memory use is 87.39M (peak 121.65M, 1.16% of max, fragmentation 1.08%), connected_slaves is 3, donations is 24, jobs is 0, jobs-adyen is 2, jobs-paypal is 0, payments-init is 66, pending is 0, recurring is 4, refund is 0, unsubscribe is 23 https://icinga.wi [19:58:17] rg/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [20:00:09] RECOVERY - check_apache2 on payments1007 is OK: PROCS OK: 13 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_apache2 [20:00:13] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 6.98, 45.21, 49.89 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [20:02:29] PROBLEM - check_apache2 on payments1005 is CRITICAL: PROCS CRITICAL: 257 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_apache2 [20:02:29] PROBLEM - check_load on payments1005 is CRITICAL: CRITICAL - load average: 242.43, 190.37, 126.83 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_load [20:03:19] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 120111 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 3 keys, up 65 days 6 hours - memory use is 65.08M (peak 121.65M, 0.89% of max, fragmentation 1.12%), connected_slaves is 3, donations is 0, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 6, pending is 0, recurring is 0, refund is 0, unsubscribe is 0 https://icinga.wikim [20:03:19] cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [20:05:13] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 3.66, 18.65, 37.01 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [20:07:23] PROBLEM - check_apache2 on payments1005 is CRITICAL: PROCS CRITICAL: 257 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_apache2 [20:07:29] PROBLEM - check_load on payments1005 is CRITICAL: CRITICAL - load average: 244.75, 224.14, 159.06 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_load [20:08:08] ACKNOWLEDGEMENT - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 120111 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 3 keys, up 65 days 6 hours - memory use is 65.08M (peak 121.65M, 0.89% of max, fragmentation 1.12%), connected_slaves is 3, donations is 0, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 6, pending is 0, recurring is 0, refund is 0, unsubscribe is 0 Elliott Eggl [20:08:08] re of the issue and dealing with it https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [20:10:13] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 2.05, 8.29, 27.44 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [20:12:28] PROBLEM - check_apache2 on payments1005 is CRITICAL: PROCS CRITICAL: 257 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_apache2 [20:12:29] PROBLEM - check_load on payments1005 is CRITICAL: CRITICAL - load average: 240.39, 235.32, 181.98 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_load [20:14:35] 10Fundraising-Backlog: Adyen getAccountConfig() calls failures - https://phabricator.wikimedia.org/T313441 (10jgleeson) [20:14:51] 10Fundraising-Backlog: Adyen getAccountConfig() call failures - https://phabricator.wikimedia.org/T313441 (10jgleeson) [20:15:14] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 1.67, 4.38, 20.47 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [20:17:18] PROBLEM - check_apache2 on payments1005 is CRITICAL: PROCS CRITICAL: 257 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_apache2 [20:17:24] PROBLEM - check_load on payments1005 is CRITICAL: CRITICAL - load average: 241.57, 239.77, 198.71 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_load [20:18:18] RECOVERY - check_redis on frqueue1003 is OK: OK: REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 65 days 6 hours - memory use is 1.99M (peak 121.65M, 0.11% of max, fragmentation 4.63%), connected_slaves is 3, donations is 6, jobs is 0, jobs-adyen is 0, jobs-paypal is 1, payments-antifraud is 6, payments-init is 69, pending is 0, recurring is 3, refund is 20, unsubscribe is 2 https://icinga.wikimedia.org/cgi-bin/icing [20:18:18] .cgi?type=2&host=frqueue1003&service=check_redis [20:18:56] (03PS2) 10Tacsipacsi: Banner editor: use mw.notify [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/808391 (https://phabricator.wikimedia.org/T262722) [20:20:28] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 216.99, 96.05, 50.62 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [20:20:56] (03CR) 10Tacsipacsi: Banner editor: use mw.notify (031 comment) [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/808391 (https://phabricator.wikimedia.org/T262722) (owner: 10Tacsipacsi) [20:22:09] RECOVERY - check_apache2 on payments1005 is OK: PROCS OK: 95 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_apache2 [20:22:14] PROBLEM - check_load on payments1005 is CRITICAL: CRITICAL - load average: 43.02, 155.90, 177.51 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_load [20:25:24] PROBLEM - check_apache2 on payments1007 is CRITICAL: PROCS CRITICAL: 257 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_apache2 [20:25:26] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 240.16, 188.12, 103.35 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [20:27:14] PROBLEM - check_load on payments1005 is CRITICAL: CRITICAL - load average: 5.51, 60.66, 130.07 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_load [20:30:23] PROBLEM - check_apache2 on payments1007 is CRITICAL: PROCS CRITICAL: 257 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_apache2 [20:30:29] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 216.54, 212.59, 138.50 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [20:32:13] PROBLEM - check_load on payments1005 is CRITICAL: CRITICAL - load average: 4.11, 25.12, 95.43 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_load [20:35:13] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 173.74, 215.59, 161.31 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [20:37:13] PROBLEM - check_load on payments1005 is CRITICAL: CRITICAL - load average: 1.84, 10.65, 69.74 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_load [20:40:09] RECOVERY - check_apache2 on payments1007 is OK: PROCS OK: 11 processes with command name apache2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_apache2 [20:40:13] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 4.33, 81.03, 117.67 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [20:42:13] PROBLEM - check_load on payments1005 is CRITICAL: CRITICAL - load average: 0.69, 4.75, 50.88 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_load [20:45:13] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 2.31, 31.42, 85.97 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [20:47:13] PROBLEM - check_load on payments1005 is CRITICAL: CRITICAL - load average: 0.58, 2.08, 36.97 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_load [20:50:13] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 1.02, 12.56, 62.71 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [20:52:13] PROBLEM - check_load on payments1005 is CRITICAL: CRITICAL - load average: 0.53, 1.04, 26.86 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_load [20:55:13] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 0.45, 4.96, 45.55 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [21:00:13] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 0.97, 2.18, 33.11 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [21:05:13] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 0.73, 1.18, 24.12 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [21:22:13] RECOVERY - check_load on payments1005 is OK: OK - load average: 0.85, 0.74, 4.44 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_load [21:35:13] RECOVERY - check_load on payments1007 is OK: OK - load average: 0.50, 0.56, 3.95 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load