[06:37:12] PROBLEM - check_puppetrun on frdev1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[aide-common] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_puppetrun [06:42:18] PROBLEM - check_puppetrun on frdev1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 10 minutes ago with 1 failures. Failed resources (up to 3 shown): Package[aide-common] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_puppetrun [06:47:18] RECOVERY - check_puppetrun on frdev1002 is OK: OK: Puppet is currently enabled, last run 45 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1002&service=check_puppetrun [11:12:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 9946 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 14 keys, up 57 days 18 hours - memory use is 7.63M (peak 28.06M, 0.18% of max, fragmentation 1.94%), connected_slaves is 3, donations is 65, jobs is 0, jobs-adyen is 15, jobs-dlocal is 0, jobs-paypal is 7, payments-antifraud is 1, payments-init is 23, pending is 0, refund is 0, unsubscribe is 2 ht [11:12:18] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [11:17:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 10806 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 57 days 18 hours - memory use is 8.17M (peak 28.06M, 0.18% of max, fragmentation 1.87%), connected_slaves is 3, donations is 58, jobs is 0, jobs-adyen is 0, jobs-dlocal is 0, jobs-paypal is 3, payments-antifraud is 1, payments-init is 1, pending is 0, refund is 0, unsubscribe is 2 htt [11:17:18] ga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [11:22:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 11392 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 16 keys, up 57 days 18 hours - memory use is 8.46M (peak 28.06M, 0.19% of max, fragmentation 1.84%), connected_slaves is 3, donations is 83, jobs is 0, jobs-adyen is 11, jobs-dlocal is 0, jobs-paypal is 7, payments-antifraud is 1, payments-init is 24, pending is 0, refund is 0, unsubscribe is 2 h [11:22:18] inga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [11:27:16] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 12207 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 15 keys, up 57 days 18 hours - memory use is 8.85M (peak 28.06M, 0.20% of max, fragmentation 1.81%), connected_slaves is 3, donations is 65, jobs is 0, jobs-adyen is 17, jobs-dlocal is 0, jobs-paypal is 5, payments-antifraud is 1, payments-init is 0, pending is 1, refund is 0, unsubscribe is 4 ht [11:27:16] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [11:32:12] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 12804 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 15 keys, up 57 days 18 hours - memory use is 9.36M (peak 28.06M, 0.20% of max, fragmentation 1.73%), connected_slaves is 3, donations is 60, jobs is 0, jobs-adyen is 0, jobs-dlocal is 0, jobs-paypal is 6, payments-antifraud is 3, payments-init is 24, pending is 1, refund is 0, unsubscribe is 4 ht [11:32:12] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [11:37:12] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 13625 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 14 keys, up 57 days 18 hours - memory use is 9.72M (peak 28.06M, 0.20% of max, fragmentation 1.74%), connected_slaves is 3, donations is 65, jobs is 0, jobs-adyen is 23, jobs-dlocal is 0, jobs-paypal is 3, payments-antifraud is 10, payments-init is 1, pending is 0, refund is 0, unsubscribe is 6 h [11:37:12] inga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [11:42:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 14055 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 14 keys, up 57 days 19 hours - memory use is 9.94M (peak 28.06M, 0.21% of max, fragmentation 1.70%), connected_slaves is 3, donations is 70, jobs is 0, jobs-adyen is 12, jobs-dlocal is 0, jobs-paypal is 1, payments-antifraud is 1, payments-init is 25, pending is 0, refund is 0, unsubscribe is 6 h [11:42:18] inga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [11:47:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 14064 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 57 days 19 hours - memory use is 9.95M (peak 28.06M, 0.20% of max, fragmentation 1.62%), connected_slaves is 3, donations is 68, jobs is 0, jobs-adyen is 0, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 4, payments-init is 0, pending is 0, refund is 0, unsubscribe is 6 htt [11:47:18] ga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [11:52:12] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 13833 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 57 days 19 hours - memory use is 9.90M (peak 28.06M, 0.20% of max, fragmentation 1.63%), connected_slaves is 3, donations is 84, jobs is 0, jobs-adyen is 39, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 4, payments-init is 19, pending is 0, refund is 0, unsubscribe is 6 h [11:52:12] inga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [11:57:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 13833 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 57 days 19 hours - memory use is 9.77M (peak 28.06M, 0.20% of max, fragmentation 1.65%), connected_slaves is 3, donations is 52, jobs is 0, jobs-adyen is 12, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 4, payments-init is 2, pending is 0, refund is 0, unsubscribe is 8 ht [11:57:18] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [12:02:16] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 13571 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 57 days 19 hours - memory use is 9.68M (peak 28.06M, 0.19% of max, fragmentation 1.65%), connected_slaves is 3, donations is 98, jobs is 0, jobs-adyen is 0, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 3, payments-init is 14, pending is 0, refund is 0, unsubscribe is 1 ht [12:02:16] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [12:07:16] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 13580 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 57 days 19 hours - memory use is 9.63M (peak 28.06M, 0.19% of max, fragmentation 1.67%), connected_slaves is 3, donations is 66, jobs is 0, jobs-adyen is 29, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 2, payments-init is 0, pending is 0, refund is 0, unsubscribe is 2 ht [12:07:16] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [12:12:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 13304 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 16 keys, up 57 days 19 hours - memory use is 9.53M (peak 28.06M, 0.19% of max, fragmentation 1.66%), connected_slaves is 3, donations is 64, jobs is 0, jobs-adyen is 11, jobs-dlocal is 1, jobs-paypal is 1, payments-antifraud is 4, payments-init is 18, pending is 0, refund is 0, unsubscribe is 2 h [12:12:18] inga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [12:17:12] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 13304 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 57 days 19 hours - memory use is 9.61M (peak 28.06M, 0.19% of max, fragmentation 1.65%), connected_slaves is 3, donations is 72, jobs is 0, jobs-adyen is 0, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 3, payments-init is 1, pending is 0, refund is 0, unsubscribe is 2 htt [12:17:12] ga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [12:22:12] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 13080 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 14 keys, up 57 days 19 hours - memory use is 9.45M (peak 28.06M, 0.19% of max, fragmentation 1.66%), connected_slaves is 3, donations is 87, jobs is 0, jobs-adyen is 29, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 2, payments-init is 25, pending is 0, refund is 0, unsubscribe is 2 h [12:22:12] inga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [12:27:12] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 13084 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 57 days 19 hours - memory use is 9.36M (peak 28.06M, 0.19% of max, fragmentation 1.69%), connected_slaves is 3, donations is 74, jobs is 0, jobs-adyen is 25, jobs-dlocal is 0, jobs-paypal is 3, payments-antifraud is 2, payments-init is 0, pending is 0, refund is 0, unsubscribe is 2 ht [12:27:12] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [12:32:12] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 12812 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 15 keys, up 57 days 19 hours - memory use is 9.38M (peak 28.06M, 0.19% of max, fragmentation 1.66%), connected_slaves is 3, donations is 82, jobs is 0, jobs-adyen is 10, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 3, payments-init is 17, pending is 0, refund is 0, unsubscribe is 2 h [12:32:12] inga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [12:37:12] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 12818 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 57 days 19 hours - memory use is 9.25M (peak 28.06M, 0.19% of max, fragmentation 1.70%), connected_slaves is 3, donations is 106, jobs is 0, jobs-adyen is 20, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 5, payments-init is 2, pending is 0, refund is 0, unsubscribe is 2 h [12:37:12] inga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [12:42:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 12549 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 57 days 20 hours - memory use is 9.09M (peak 28.06M, 0.19% of max, fragmentation 1.69%), connected_slaves is 3, donations is 68, jobs is 0, jobs-adyen is 22, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 0, payments-init is 25, pending is 0, refund is 0, unsubscribe is 4 h [12:42:18] inga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [12:47:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 12550 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 14 keys, up 57 days 20 hours - memory use is 9.27M (peak 28.06M, 0.19% of max, fragmentation 1.69%), connected_slaves is 3, donations is 66, jobs is 0, jobs-adyen is 8, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 7, payments-init is 1, pending is 0, refund is 0, unsubscribe is 5 htt [12:47:18] ga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [12:52:12] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 12314 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 15 keys, up 57 days 20 hours - memory use is 8.97M (peak 28.06M, 0.19% of max, fragmentation 1.70%), connected_slaves is 3, donations is 79, jobs is 0, jobs-adyen is 29, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 5, payments-init is 12, pending is 0, refund is 0, unsubscribe is 5 h [12:52:12] inga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [12:57:12] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 12324 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 14 keys, up 57 days 20 hours - memory use is 8.94M (peak 28.06M, 0.19% of max, fragmentation 1.71%), connected_slaves is 3, donations is 62, jobs is 0, jobs-adyen is 31, jobs-dlocal is 0, jobs-paypal is 1, payments-antifraud is 2, payments-init is 3, pending is 0, refund is 0, unsubscribe is 5 ht [12:57:12] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [13:02:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 12104 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 14 keys, up 57 days 20 hours - memory use is 8.90M (peak 28.06M, 0.18% of max, fragmentation 1.70%), connected_slaves is 3, donations is 98, jobs is 0, jobs-adyen is 0, jobs-dlocal is 1, jobs-paypal is 0, payments-antifraud is 3, payments-init is 20, pending is 0, refund is 0, unsubscribe is 0 ht [13:02:18] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [13:07:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 12106 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 57 days 20 hours - memory use is 8.87M (peak 28.06M, 0.19% of max, fragmentation 1.73%), connected_slaves is 3, donations is 97, jobs is 0, jobs-adyen is 31, jobs-dlocal is 0, jobs-paypal is 2, payments-antifraud is 7, payments-init is 2, pending is 0, refund is 0, unsubscribe is 0 ht [13:07:18] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [13:12:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 11868 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 57 days 20 hours - memory use is 8.70M (peak 28.06M, 0.18% of max, fragmentation 1.72%), connected_slaves is 3, donations is 61, jobs is 0, jobs-adyen is 5, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 0, payments-init is 16, pending is 0, refund is 0, unsubscribe is 1 ht [13:12:18] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [13:17:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 11870 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 14 keys, up 57 days 20 hours - memory use is 8.82M (peak 28.06M, 0.18% of max, fragmentation 1.71%), connected_slaves is 3, donations is 58, jobs is 0, jobs-adyen is 0, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 5, payments-init is 2, pending is 0, refund is 0, unsubscribe is 1 htt [13:17:18] ga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [13:22:12] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 11638 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 15 keys, up 57 days 20 hours - memory use is 8.68M (peak 28.06M, 0.18% of max, fragmentation 1.72%), connected_slaves is 3, donations is 79, jobs is 0, jobs-adyen is 40, jobs-dlocal is 0, jobs-paypal is 1, payments-antifraud is 6, payments-init is 26, pending is 0, refund is 0, unsubscribe is 3 h [13:22:12] inga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [13:27:12] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 11641 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 15 keys, up 57 days 20 hours - memory use is 8.70M (peak 28.06M, 0.18% of max, fragmentation 1.71%), connected_slaves is 3, donations is 65, jobs is 0, jobs-adyen is 22, jobs-dlocal is 0, jobs-paypal is 1, payments-antifraud is 2, payments-init is 1, pending is 0, refund is 0, unsubscribe is 3 ht [13:27:12] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [13:32:12] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 11365 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 57 days 20 hours - memory use is 8.53M (peak 28.06M, 0.18% of max, fragmentation 1.72%), connected_slaves is 3, donations is 85, jobs is 0, jobs-adyen is 0, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 0, payments-init is 24, pending is 0, refund is 0, unsubscribe is 4 ht [13:32:12] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [13:37:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 11366 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 14 keys, up 57 days 20 hours - memory use is 8.43M (peak 28.06M, 0.18% of max, fragmentation 1.75%), connected_slaves is 3, donations is 69, jobs is 0, jobs-adyen is 33, jobs-dlocal is 0, jobs-paypal is 1, payments-antifraud is 1, payments-init is 1, pending is 0, refund is 0, unsubscribe is 5 ht [13:37:18] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [13:42:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 11107 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 57 days 21 hours - memory use is 8.40M (peak 28.06M, 0.18% of max, fragmentation 1.75%), connected_slaves is 3, donations is 69, jobs is 0, jobs-adyen is 21, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 4, payments-init is 24, pending is 0, refund is 0, unsubscribe is 6 h [13:42:18] inga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [13:47:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 11113 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 15 keys, up 57 days 21 hours - memory use is 8.48M (peak 28.06M, 0.18% of max, fragmentation 1.73%), connected_slaves is 3, donations is 73, jobs is 0, jobs-adyen is 8, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 9, payments-init is 1, pending is 0, refund is 0, unsubscribe is 6 htt [13:47:18] ga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [13:52:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 10849 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 57 days 21 hours - memory use is 8.17M (peak 28.06M, 0.18% of max, fragmentation 1.78%), connected_slaves is 3, donations is 92, jobs is 0, jobs-adyen is 27, jobs-dlocal is 0, jobs-paypal is 1, payments-antifraud is 0, payments-init is 25, pending is 0, refund is 0, unsubscribe is 7 h [13:52:18] inga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [13:57:12] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 10856 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 14 keys, up 57 days 21 hours - memory use is 8.17M (peak 28.06M, 0.18% of max, fragmentation 1.79%), connected_slaves is 3, donations is 82, jobs is 0, jobs-adyen is 17, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 5, payments-init is 2, pending is 0, refund is 0, unsubscribe is 9 ht [13:57:12] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [14:02:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 10606 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 16 keys, up 57 days 21 hours - memory use is 8.16M (peak 28.06M, 0.17% of max, fragmentation 1.77%), connected_slaves is 3, donations is 133, jobs is 0, jobs-adyen is 0, jobs-dlocal is 0, jobs-paypal is 1, payments-antifraud is 2, payments-init is 31, pending is 1, refund is 0, unsubscribe is 1 h [14:02:18] inga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [14:07:12] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 10606 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 57 days 21 hours - memory use is 8.09M (peak 28.06M, 0.17% of max, fragmentation 1.80%), connected_slaves is 3, donations is 86, jobs is 0, jobs-adyen is 50, jobs-dlocal is 0, jobs-paypal is 1, payments-antifraud is 0, payments-init is 2, pending is 0, refund is 0, unsubscribe is 3 ht [14:07:12] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [14:12:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 10365 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 16 keys, up 57 days 21 hours - memory use is 7.94M (peak 28.06M, 0.17% of max, fragmentation 1.80%), connected_slaves is 3, donations is 72, jobs is 0, jobs-adyen is 21, jobs-dlocal is 0, jobs-paypal is 2, payments-antifraud is 4, payments-init is 27, pending is 1, refund is 0, unsubscribe is 3 h [14:12:18] inga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [14:17:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 10368 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 14 keys, up 57 days 21 hours - memory use is 7.93M (peak 28.06M, 0.17% of max, fragmentation 1.79%), connected_slaves is 3, donations is 74, jobs is 0, jobs-adyen is 6, jobs-dlocal is 0, jobs-paypal is 2, payments-antifraud is 2, payments-init is 0, pending is 0, refund is 0, unsubscribe is 5 htt [14:17:18] ga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [14:22:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 10120 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 14 keys, up 57 days 21 hours - memory use is 7.84M (peak 28.06M, 0.17% of max, fragmentation 1.80%), connected_slaves is 3, donations is 78, jobs is 0, jobs-adyen is 47, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 2, payments-init is 14, pending is 0, refund is 0, unsubscribe is 5 h [14:22:18] inga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [14:27:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 10130 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 57 days 21 hours - memory use is 7.74M (peak 28.06M, 0.17% of max, fragmentation 1.82%), connected_slaves is 3, donations is 73, jobs is 0, jobs-adyen is 13, jobs-dlocal is 0, jobs-paypal is 2, payments-antifraud is 3, payments-init is 0, pending is 0, refund is 0, unsubscribe is 5 ht [14:27:18] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [14:32:12] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 9844 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 14 keys, up 57 days 21 hours - memory use is 7.76M (peak 28.06M, 0.17% of max, fragmentation 1.80%), connected_slaves is 3, donations is 91, jobs is 0, jobs-adyen is 0, jobs-dlocal is 0, jobs-paypal is 1, payments-antifraud is 3, payments-init is 26, pending is 0, refund is 0, unsubscribe is 7 htt [14:32:12] ga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [14:37:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 9845 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 16 keys, up 57 days 21 hours - memory use is 7.62M (peak 28.06M, 0.17% of max, fragmentation 1.84%), connected_slaves is 3, donations is 87, jobs is 0, jobs-adyen is 34, jobs-dlocal is 0, jobs-paypal is 1, payments-antifraud is 7, payments-init is 1, pending is 1, refund is 0, unsubscribe is 10 ht [14:37:18] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [14:42:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 9569 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 14 keys, up 57 days 22 hours - memory use is 7.50M (peak 28.06M, 0.17% of max, fragmentation 1.83%), connected_slaves is 3, donations is 80, jobs is 0, jobs-adyen is 20, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 2, payments-init is 29, pending is 0, refund is 0, unsubscribe is 10 h [14:42:18] inga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [14:47:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: recurring is 9578 9500 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 14 keys, up 57 days 22 hours - memory use is 7.50M (peak 28.06M, 0.17% of max, fragmentation 1.80%), connected_slaves is 3, donations is 75, jobs is 0, jobs-adyen is 0, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 5, payments-init is 2, pending is 0, refund is 0, unsubscribe is 12 htt [14:47:18] ga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [15:52:18] RECOVERY - check_redis on frqueue1004 is OK: OK: REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 15 keys, up 57 days 23 hours - memory use is 6.57M (peak 28.06M, 0.16% of max, fragmentation 1.95%), connected_slaves is 3, donations is 82, jobs is 0, jobs-adyen is 33, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 1, payments-init is 30, pending is 0, recurring is 7855, refund is 0, unsubscribe is 11 https://icinga.wik [15:52:18] g/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [18:25:23] 10Fundraising-Backlog: Original Campaign Coding [EPIC] - https://phabricator.wikimedia.org/T327299 (10AKanji-WMF) [18:40:59] 10Fundraising Sprint Spacebar Oddity 👾, 10Fundraising-Backlog: Fundraising Sprint T - Priorities - https://phabricator.wikimedia.org/T348408 (10AKanji-WMF) [18:41:52] 10Fundraising Sprint Spacebar Oddity 👾, 10Fundraising-Backlog: Fundraising Sprint T - Priorities - https://phabricator.wikimedia.org/T348408 (10AKanji-WMF)