[01:23:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 34559 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 15 days 23 hours - memory use is 27.08M (peak 27.88M, 0.38% of max, fragmentation 1.15%), connected_slaves is 3, donations is 1, jobs is 0, jobs-adyen is 0, jobs-paypal is 1, payments-init is 7, pending is 0, recurring is 0, refund is 1, unsubscribe is 0 https://icinga.wikime [01:23:17] gi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [01:28:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 51397 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 16 days 2 minutes - memory use is 39.03M (peak 39.41M, 0.53% of max, fragmentation 1.12%), connected_slaves is 3, donations is 34, jobs is 0, jobs-adyen is 1, jobs-paypal is 0, payments-init is 99, pending is 0, recurring is 3, refund is 0, unsubscribe is 0 https://icinga.wik [01:28:17] g/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [01:33:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 68342 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 16 days 7 minutes - memory use is 50.65M (peak 51.26M, 0.68% of max, fragmentation 1.09%), connected_slaves is 3, donations is 22, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 6, pending is 0, recurring is 1, refund is 0, unsubscribe is 0 https://icinga.wiki [01:33:17] /cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [01:38:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 85096 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 16 days 12 minutes - memory use is 61.61M (peak 62.04M, 0.81% of max, fragmentation 1.08%), connected_slaves is 3, donations is 66, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 101, pending is 0, recurring is 2, refund is 0, unsubscribe is 0 https://icinga.w [01:38:17] org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [01:39:51] (03CR) 10AndyRussG: [C: 03+2] fix encoding from utf8 to utf8mb4 in 001_create.sql [wikimedia/fundraising/tools/DjangoBannerStats] - 10https://gerrit.wikimedia.org/r/888273 (owner: 10Jgreen) [01:40:16] (03Merged) 10jenkins-bot: fix encoding from utf8 to utf8mb4 in 001_create.sql [wikimedia/fundraising/tools/DjangoBannerStats] - 10https://gerrit.wikimedia.org/r/888273 (owner: 10Jgreen) [01:43:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 101494 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 16 days 17 minutes - memory use is 72.76M (peak 72.80M, 0.95% of max, fragmentation 1.08%), connected_slaves is 3, donations is 38, jobs is 0, jobs-adyen is 2, jobs-paypal is 0, payments-init is 6, pending is 0, recurring is 2, refund is 0, unsubscribe is 0 https://icinga.wi [01:43:17] rg/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [01:47:25] 10Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for andyrussg - https://phabricator.wikimedia.org/T328377 (10Dwisehaupt) SSL certificate renewed and files in user's home dir on frpm1002 [01:48:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 102662 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 16 days 22 minutes - memory use is 59.39M (peak 79.41M, 0.80% of max, fragmentation 1.10%), connected_slaves is 3, donations is 0, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 100, pending is 0, recurring is 2, refund is 0, unsubscribe is 0 https://icinga.w [01:48:17] org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [01:53:23] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 83874 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 7 keys, up 16 days 27 minutes - memory use is 49.65M (peak 79.41M, 0.69% of max, fragmentation 1.13%), connected_slaves is 3, donations is 2, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 9, pending is 0, recurring is 0, refund is 0, unsubscribe is 0 https://icinga.wiki [01:53:23] /cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [01:58:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 64915 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 16 days 32 minutes - memory use is 39.86M (peak 79.41M, 0.56% of max, fragmentation 1.16%), connected_slaves is 3, donations is 50, jobs is 0, jobs-adyen is 31, jobs-paypal is 1, payments-init is 97, pending is 0, recurring is 0, refund is 0, unsubscribe is 0 https://icinga.w [01:58:17] org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [02:03:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 44423 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 9 keys, up 16 days 37 minutes - memory use is 28.99M (peak 79.41M, 0.43% of max, fragmentation 1.21%), connected_slaves is 3, donations is 21, jobs is 0, jobs-adyen is 20, jobs-paypal is 0, payments-init is 10, pending is 0, recurring is 0, refund is 0, unsubscribe is 0 https://icinga.w [02:03:17] org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [02:08:17] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: payments-antifraud is 22507 10000 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 7 keys, up 16 days 42 minutes - memory use is 16.53M (peak 79.41M, 0.28% of max, fragmentation 1.44%), connected_slaves is 3, donations is 28, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-init is 0, pending is 0, recurring is 0, refund is 0, unsubscribe is 1 https://icinga.wik [02:08:17] g/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [02:13:17] RECOVERY - check_redis on frqueue1003 is OK: OK: REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 8 keys, up 16 days 47 minutes - memory use is 6.20M (peak 79.41M, 0.16% of max, fragmentation 2.02%), connected_slaves is 3, donations is 72, jobs is 0, jobs-adyen is 22, jobs-paypal is 0, payments-antifraud is 3934, payments-init is 0, pending is 0, recurring is 0, refund is 0, unsubscribe is 1 https://icinga.wikimedia.org/cgi-bin/ [02:13:17] tinfo.cgi?type=2&host=frqueue1003&service=check_redis [02:34:53] (03PS1) 10Cstone: WIP: Separate astropay (old dlocal config) from dlocal (new dlocal config) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/888280 (https://phabricator.wikimedia.org/T328701)