[00:07:07] (03PS4) 10Wfan: Braintree Audit parse donation, refund and chargeback [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/812100 (https://phabricator.wikimedia.org/T310757) [00:07:34] (03CR) 10CI reject: [V: 04-1] Braintree Audit parse donation, refund and chargeback [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/812100 (https://phabricator.wikimedia.org/T310757) (owner: 10Wfan) [00:11:16] (03PS5) 10Wfan: Braintree Audit parse donation, refund and chargeback [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/812100 (https://phabricator.wikimedia.org/T310757) [00:11:26] (03PS6) 10Wfan: Braintree Audit parse donation, refund and chargeback [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/812100 (https://phabricator.wikimedia.org/T310757) [12:33:18] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: recurring is 9839 9500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 4 keys, up 52 days 22 hours - memory use is 7.53M (peak 7.96M, 0.15% of max, fragmentation 1.65%), connected_slaves is 3, donations is 0, jobs is 0, jobs-adyen is 0, jobs-paypal is 73, payments-antifraud is 0, payments-init is 0, pending is 0, refund is 0, unsubscribe is 0 https://icinga.wikimedia [12:33:18] bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [12:38:18] PROBLEM - check_redis on frqueue1003 is CRITICAL: CRITICAL: recurring is 10014 9500 - REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 4 keys, up 52 days 22 hours - memory use is 7.52M (peak 7.96M, 0.15% of max, fragmentation 1.61%), connected_slaves is 3, donations is 0, jobs is 0, jobs-adyen is 0, jobs-paypal is 4, payments-antifraud is 0, payments-init is 0, pending is 0, refund is 0, unsubscribe is 0 https://icinga.wikimedia [12:38:18] bin/icinga/extinfo.cgi?type=2&host=frqueue1003&service=check_redis [13:23:18] RECOVERY - check_redis on frqueue1003 is OK: OK: REDIS 5.0.14 on 127.0.0.1:6379 has 1 databases (db0) with 2 keys, up 52 days 23 hours - memory use is 6.30M (peak 7.96M, 0.13% of max, fragmentation 1.72%), connected_slaves is 3, donations is 0, jobs is 0, jobs-adyen is 0, jobs-paypal is 0, payments-antifraud is 0, payments-init is 0, pending is 0, recurring is 7859, refund is 0, unsubscribe is 0 https://icinga.wikimedia.org/cgi-bin/icing [13:23:18] .cgi?type=2&host=frqueue1003&service=check_redis [13:55:18] PROBLEM - check_puppetrun on civi1002 is CRITICAL: CRITICAL: Puppet has 23 failures. Last run 9 minutes ago with 23 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_puppetrun [14:00:10] RECOVERY - check_puppetrun on civi1002 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_puppetrun [15:09:27] PROBLEM - Host frdb1002 is DOWN: PING CRITICAL - Packet loss = 100% [15:15:11] RECOVERY - Host frdb1002 is UP: PING OK - Packet loss = 0%, RTA = 0.40 ms [15:53:12] 10Fundraising-Backlog: Check need to account for opt-in param loss from 2022-06-15 through 2022-06-30 - https://phabricator.wikimedia.org/T311974 (10AndyRussG) >>! In T311974#8062689, @HNordeenWMF wrote: > Opt-in not required as default but can still be seen OK hmmm I guess that does complicate it a bit for tho... [16:12:56] (03PS8) 10Wfan: Add test for civi run audit [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809644 (https://phabricator.wikimedia.org/T310758) [16:26:48] (03CR) 10CI reject: [V: 04-1] Add test for civi run audit [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809644 (https://phabricator.wikimedia.org/T310758) (owner: 10Wfan) [16:48:53] 10Fundraising-Backlog: Check need to account for opt-in param loss from 2022-06-15 through 2022-06-30 - https://phabricator.wikimedia.org/T311974 (10HNordeenWMF) I'll do my best to clarify but @Pcoombe might need to jump in and help explain. (We're talking about how opt-in requirements work!) //Opt-in not requi... [17:21:03] 10Fundraising-Backlog: Ingenico/Worldline production key expiring 14 July - https://phabricator.wikimedia.org/T312662 (10EMartin) [17:40:52] 10fundraising-tech-ops: Install and configure new host frdb1006 - https://phabricator.wikimedia.org/T312584 (10Jgreen) [17:41:37] 10fundraising-tech-ops: Install and configure new host frdb1006 - https://phabricator.wikimedia.org/T312584 (10Jgreen) [17:42:26] 10fundraising-tech-ops: Install and configure new host frdb1005 - https://phabricator.wikimedia.org/T312583 (10Jgreen) [17:43:21] 10fundraising-tech-ops: Install and configure new host frlog1002 - https://phabricator.wikimedia.org/T312581 (10Jgreen) [18:43:19] 10fundraising-tech-ops: Install and configure new host frdb1006 - https://phabricator.wikimedia.org/T312584 (10Jgreen) p:05Triage→03Medium a:03Jgreen [18:43:38] 10fundraising-tech-ops: Install and configure new host frdb1006 - https://phabricator.wikimedia.org/T312584 (10Jgreen) [18:52:38] 10fundraising-tech-ops: FRACK environment support for H750 RAID controller - https://phabricator.wikimedia.org/T312669 (10Jgreen) [18:53:04] 10fundraising-tech-ops: Install and configure new host frdb1006 - https://phabricator.wikimedia.org/T312584 (10Jgreen) [18:53:30] 10fundraising-tech-ops: Install and configure new host frdb1005 - https://phabricator.wikimedia.org/T312583 (10Jgreen) [18:53:32] 10fundraising-tech-ops: FRACK environment support for H750 RAID controller - https://phabricator.wikimedia.org/T312669 (10Jgreen) [18:54:00] 10fundraising-tech-ops: Install and configure new host frdb1005 - https://phabricator.wikimedia.org/T312583 (10Jgreen) [20:11:40] 10fundraising-tech-ops: FRACK environment support for H750 RAID controller - https://phabricator.wikimedia.org/T312669 (10Jgreen) [20:19:35] 10fundraising-tech-ops: FRACK environment support for H750 RAID controller - https://phabricator.wikimedia.org/T312669 (10Jgreen) [20:22:50] 10fundraising-tech-ops: FRACK environment support for H750 RAID controller - https://phabricator.wikimedia.org/T312669 (10Jgreen)