[00:02:14] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-antifraud is 20769 10000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 9 days 3 hours - memory use is 208.60M (peak 421.51M, 2.77% of max, fragmentation 1.09%), connected_slaves is 3, donations is 5, jobs is 0, jobs-adyen is 0, jobs-dlocal is 1, jobs-paypal is 0, payments-init is 2, pending is 0, recurring is 2, refund is 0, unsubscribe is 0 ht [00:02:14] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [00:04:52] keeping an eye on that queue. [00:05:39] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/942037 [00:06:01] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/942037 (owner: 10Eileen) [00:06:55] (03Merged) 10jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/942037 (owner: 10Eileen) [00:07:16] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-antifraud is 36499 10000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 9 days 3 hours - memory use is 231.65M (peak 421.51M, 3.04% of max, fragmentation 1.07%), connected_slaves is 3, donations is 6, jobs is 0, jobs-adyen is 5, jobs-dlocal is 0, jobs-paypal is 0, payments-init is 0, pending is 0, recurring is 3, refund is 0, unsubscribe is 0 ht [00:07:16] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [00:12:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-antifraud is 55874 10000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 9 days 3 hours - memory use is 258.34M (peak 421.51M, 3.34% of max, fragmentation 1.06%), connected_slaves is 3, donations is 0, jobs is 0, jobs-adyen is 0, jobs-dlocal is 0, jobs-paypal is 1, payments-init is 1, pending is 1, recurring is 3, refund is 0, unsubscribe is 0 ht [00:12:18] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [00:17:14] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-antifraud is 77210 10000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 9 days 3 hours - memory use is 289.51M (peak 421.51M, 3.69% of max, fragmentation 1.04%), connected_slaves is 3, donations is 1, jobs is 0, jobs-adyen is 0, jobs-dlocal is 0, jobs-paypal is 1, payments-init is 1, pending is 0, recurring is 5, refund is 0, unsubscribe is 0 ht [00:17:14] nga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [00:17:16] !log civicrm upgraded from 853c14f3 to 68f29b70 [00:17:17] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:20:01] !log civicrm rolled back from 68f29b70 to 853c14f3 (locked) [00:20:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:20:14] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 28.86, 25.96, 21.80 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [00:20:14] !log rollback because I got an error when I tried to view - so let's see [00:20:16] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:21:14] PROBLEM - check_load on payments1006 is CRITICAL: CRITICAL - load average: 24.43, 22.87, 20.30 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_load [00:22:14] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-antifraud is 100318 10000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 9 days 3 hours - memory use is 316.73M (peak 421.51M, 4.04% of max, fragmentation 1.04%), connected_slaves is 3, donations is 40, jobs is 0, jobs-adyen is 2, jobs-dlocal is 0, jobs-paypal is 0, payments-init is 0, pending is 0, recurring is 7, refund is 0, unsubscribe is 0 [00:22:14] cinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [00:25:14] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 35.12, 28.67, 23.89 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [00:26:08] PROBLEM - check_load on payments1006 is CRITICAL: CRITICAL - load average: 26.65, 24.66, 21.75 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_load [00:26:11] (03PS1) 10Eileen: Fix lybunt value typo, the key & value should match [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942039 [00:27:14] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-antifraud is 125563 10000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 9 days 4 hours - memory use is 346.50M (peak 421.51M, 4.41% of max, fragmentation 1.04%), connected_slaves is 3, donations is 201, jobs is 0, jobs-adyen is 0, jobs-dlocal is 0, jobs-paypal is 0, payments-init is 0, pending is 0, recurring is 11, refund is 0, unsubscribe is [00:27:14] /icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [00:28:29] cstone: are you still about - I just had to rollback that deploy because of a super small mistake -https://gerrit.wikimedia.org/r/942039 [00:30:24] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 58.30, 35.20, 27.26 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [00:31:16] PROBLEM - check_load on payments1006 is CRITICAL: CRITICAL - load average: 73.67, 41.76, 28.77 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_load [00:32:14] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-antifraud is 152394 10000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 9 days 4 hours - memory use is 394.95M (peak 421.51M, 4.90% of max, fragmentation 1.01%), connected_slaves is 3, donations is 360, jobs is 0, jobs-adyen is 0, jobs-dlocal is 1, jobs-paypal is 1, payments-init is 4, pending is 0, recurring is 6, refund is 0, unsubscribe is 0 [00:32:14] icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [00:35:14] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 25.78, 44.19, 34.93 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [00:35:22] (03CR) 10XenoRyet: [C: 03+2] Fix lybunt value typo, the key & value should match [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942039 (owner: 10Eileen) [00:36:08] PROBLEM - check_load on payments1006 is CRITICAL: CRITICAL - load average: 24.69, 39.45, 32.40 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_load [00:36:13] cstone: ignore above, done [00:37:14] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-antifraud is 162449 10000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 9 days 4 hours - memory use is 396.96M (peak 421.51M, 5.05% of max, fragmentation 1.04%), connected_slaves is 3, donations is 536, jobs is 0, jobs-adyen is 4, jobs-dlocal is 0, jobs-paypal is 0, payments-init is 0, pending is 0, recurring is 10, refund is 0, unsubscribe is [00:37:14] /icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [00:40:14] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 13.05, 25.44, 29.46 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [00:41:14] PROBLEM - check_load on payments1006 is CRITICAL: CRITICAL - load average: 12.29, 22.96, 27.16 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_load [00:42:18] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-antifraud is 148459 10000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 9 days 4 hours - memory use is 390.68M (peak 421.51M, 4.98% of max, fragmentation 1.04%), connected_slaves is 3, donations is 602, jobs is 0, jobs-adyen is 3, jobs-dlocal is 1, jobs-paypal is 0, payments-init is 6, pending is 0, recurring is 9, refund is 0, unsubscribe is 0 [00:42:18] icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [00:43:16] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/942040 [00:44:12] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/942040 (owner: 10Eileen) [00:45:14] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 13.39, 17.64, 24.89 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [00:46:14] PROBLEM - check_load on payments1006 is CRITICAL: CRITICAL - load average: 13.99, 16.96, 23.34 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_load [00:47:19] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-antifraud is 134243 10000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 9 days 4 hours - memory use is 385.72M (peak 421.51M, 4.91% of max, fragmentation 1.04%), connected_slaves is 3, donations is 606, jobs is 0, jobs-adyen is 1, jobs-dlocal is 0, jobs-paypal is 0, payments-init is 0, pending is 0, recurring is 14, refund is 0, unsubscribe is [00:47:19] /icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [00:50:14] PROBLEM - check_load on payments1007 is CRITICAL: CRITICAL - load average: 13.72, 14.68, 21.58 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [00:51:08] PROBLEM - check_load on payments1006 is CRITICAL: CRITICAL - load average: 14.43, 14.39, 20.39 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_load [00:51:18] !log civicrm upgraded from 853c14f3 to 68f29b70 [00:51:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:51:36] (03Merged) 10jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/942040 (owner: 10Eileen) [00:52:14] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-antifraud is 127420 10000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 9 days 4 hours - memory use is 386.07M (peak 421.51M, 4.92% of max, fragmentation 1.04%), connected_slaves is 3, donations is 650, jobs is 0, jobs-adyen is 6, jobs-dlocal is 0, jobs-paypal is 0, payments-init is 0, pending is 0, recurring is 12, refund is 0, unsubscribe is [00:52:14] /icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [00:54:53] !log civicrm upgraded from 68f29b70 to 5a84b138 [00:54:55] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:57:16] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-antifraud is 119819 10000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 12 keys, up 9 days 4 hours - memory use is 387.35M (peak 421.51M, 4.92% of max, fragmentation 1.04%), connected_slaves is 3, donations is 829, jobs is 0, jobs-adyen is 0, jobs-dlocal is 2, jobs-paypal is 0, payments-init is 0, pending is 0, recurring is 12, refund is 0, unsubscribe is [00:57:16] /icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [01:02:14] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-antifraud is 115528 10000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 9 days 4 hours - memory use is 390.82M (peak 421.51M, 4.96% of max, fragmentation 1.04%), connected_slaves is 3, donations is 997, jobs is 0, jobs-adyen is 0, jobs-dlocal is 1, jobs-paypal is 0, payments-init is 1, pending is 0, recurring is 10, refund is 0, unsubscribe is [01:02:14] /icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [01:07:16] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-antifraud is 116136 10000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 10 keys, up 9 days 4 hours - memory use is 396.91M (peak 421.51M, 5.05% of max, fragmentation 1.04%), connected_slaves is 3, donations is 1096, jobs is 0, jobs-adyen is 1, jobs-dlocal is 0, jobs-paypal is 0, payments-init is 0, pending is 0, recurring is 14, refund is 0, unsubscribe is [01:07:16] //icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [01:12:14] PROBLEM - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-antifraud is 127846 10000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 9 days 4 hours - memory use is 412.69M (peak 421.51M, 5.24% of max, fragmentation 1.04%), connected_slaves is 3, donations is 32, jobs is 0, jobs-adyen is 2, jobs-dlocal is 0, jobs-paypal is 0, payments-init is 1, pending is 0, recurring is 3, refund is 0, unsubscribe is 0 [01:12:14] cinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [01:12:23] ACKNOWLEDGEMENT - check_redis on frqueue1004 is CRITICAL: CRITICAL: payments-antifraud is 127846 10000 - REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 11 keys, up 9 days 4 hours - memory use is 412.69M (peak 421.51M, 5.24% of max, fragmentation 1.04%), connected_slaves is 3, donations is 32, jobs is 0, jobs-adyen is 2, jobs-dlocal is 0, jobs-paypal is 0, payments-init is 1, pending is 0, recurring is 3, refund is 0, unsubscri [01:12:23] wisehaupt known https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [01:21:02] (03PS1) 10Wfan: make the composer fix consistant as payments and smashpig for easy to remember [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942043 [01:23:20] (03CR) 10CI reject: [V: 04-1] make the composer fix consistant as payments and smashpig for easy to remember [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942043 (owner: 10Wfan) [01:47:39] (03PS1) 10Eileen: Update triggers to get new segments [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942044 (https://phabricator.wikimedia.org/T342165) [01:50:06] (03CR) 10CI reject: [V: 04-1] Update triggers to get new segments [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942044 (https://phabricator.wikimedia.org/T342165) (owner: 10Eileen) [01:52:51] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942044 (https://phabricator.wikimedia.org/T342165) (owner: 10Eileen) [01:58:13] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942044 (https://phabricator.wikimedia.org/T342165) (owner: 10Eileen) [02:10:43] (03PS1) 10Eileen: Test - see how this impacts build [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942045 [02:14:52] (03CR) 10CI reject: [V: 04-1] Test - see how this impacts build [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942045 (owner: 10Eileen) [02:15:12] (03PS2) 10Wfan: rename the composer civilintcbf to fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942043 [02:15:20] (03CR) 10CI reject: [V: 04-1] rename the composer civilintcbf to fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942043 (owner: 10Wfan) [02:16:35] (03PS3) 10Wfan: rename the composer civilintcbf to fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942043 [02:20:40] (03CR) 10CI reject: [V: 04-1] rename the composer civilintcbf to fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942043 (owner: 10Wfan) [02:26:21] (03PS1) 10Eileen: Port buildkit fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942046 [02:30:35] (03CR) 10CI reject: [V: 04-1] Port buildkit fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942046 (owner: 10Eileen) [02:31:15] (03PS4) 10Wfan: rename the composer civilintcbf to fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942043 [02:35:31] (03CR) 10CI reject: [V: 04-1] rename the composer civilintcbf to fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942043 (owner: 10Wfan) [02:37:49] (03CR) 10Eileen: "test this please" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942046 (owner: 10Eileen) [02:41:08] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942044 (https://phabricator.wikimedia.org/T342165) (owner: 10Eileen) [02:42:34] (03PS5) 10Wfan: rename the composer civilintcbf to fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942043 [02:46:37] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942046 (owner: 10Eileen) [02:54:39] (03PS2) 10Eileen: Port buildkit fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942046 [02:54:47] (03Abandoned) 10Eileen: Test - see how this impacts build [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942045 (owner: 10Eileen) [02:56:36] (03PS1) 10Cstone: Update logging [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/942048 [02:58:05] (03CR) 10CI reject: [V: 04-1] Update logging [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/942048 (owner: 10Cstone) [02:58:39] (03PS2) 10Cstone: Update logging [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/942048 [03:01:38] (03CR) 10Dwisehaupt: [C: 03+2] "Quite a diff." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942044 (https://phabricator.wikimedia.org/T342165) (owner: 10Eileen) [03:02:11] (03CR) 10Eileen: [V: 03+2] Update triggers to get new segments [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942044 (https://phabricator.wikimedia.org/T342165) (owner: 10Eileen) [03:02:47] (03CR) 10Dwisehaupt: [C: 03+2] "Looks great." [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/942048 (owner: 10Cstone) [03:03:09] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/942049 [03:03:37] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/942049 (owner: 10Eileen) [03:04:17] (03Merged) 10jenkins-bot: Update logging [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/942048 (owner: 10Cstone) [03:04:42] !log civicrm upgraded from 5a84b138 to 16c2e58a [03:04:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:06:53] wfan: I just deployed your composer change & revert to it - just checking that the reason for the revert wasn't due to CI failing - cos that was something to do with buildkit not your patch [03:07:48] Ha thanks Eileen :) [03:10:14] RECOVERY - check_load on payments1007 is OK: OK - load average: 0.06, 0.75, 4.91 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_load [03:11:15] (03CR) 10Cstone: [C: 03+2] Port buildkit fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942046 (owner: 10Eileen) [03:13:18] PROBLEM - check_mysql on frdb1004 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [03:15:22] (03PS1) 10Cstone: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/942050 [03:16:08] RECOVERY - check_load on payments1006 is OK: OK - load average: 0.18, 0.36, 3.67 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_load [03:17:14] (03CR) 10Cstone: [C: 03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/942050 (owner: 10Cstone) [03:18:01] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/942050 (owner: 10Cstone) [03:18:16] RECOVERY - check_mysql on frdb1004 is OK: Uptime: 802117 Threads: 4 Questions: 116915285 Slow queries: 402 Opens: 2414 Open tables: 1018 Queries per second avg: 145.758 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [03:19:45] !log payments-wiki upgraded from 2a68dfe2 to 1a6ca7ab [03:19:47] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [03:20:07] (03PS1) 10Eileen: Slightly clearer wording [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942051 [03:20:54] Just FYI I have made further changes to upstream buildkit - this feeds through to our CI immediately so if ^^ doesn't pass then they went wrong [03:32:28] (03Merged) 10jenkins-bot: Port buildkit fix [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942046 (owner: 10Eileen) [04:57:11] 10Fundraising Sprint Never gonna git you update, 10Fundraising-Backlog: Figure out how to schedule preliminary updating of segment data - https://phabricator.wikimedia.org/T339186 (10Eileenmcnaughton) I manually did the updates for a bunch of these on the command line to see how it goes - 100k contacts at once... [05:02:23] 10Fundraising Sprint Never gonna git you update, 10Fundraising-Backlog: Figure out how to schedule preliminary updating of segment data - https://phabricator.wikimedia.org/T339186 (10Eileenmcnaughton) Note - query select donor_segment_id, count(*), min(entity_id), max(entity_id) FROM wmf_donor WHERE donor_seg... [05:07:14] RECOVERY - check_redis on frqueue1004 is OK: OK: REDIS 6.0.16 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 9 days 8 hours - memory use is 343.54M (peak 1.17G, 4.60% of max, fragmentation 1.10%), connected_slaves is 3, donations is 29, jobs is 0, jobs-adyen is 45, jobs-dlocal is 0, jobs-paypal is 0, payments-antifraud is 27, payments-init is 0, pending is 6, recurring is 1, refund is 0, unsubscribe is 1 https://icinga.wikimedi [05:07:14] -bin/icinga/extinfo.cgi?type=2&host=frqueue1004&service=check_redis [06:36:59] 10Fundraising-Backlog: We need to list of canceled recurrings in India to send to dLocal for them to stop sending pre-notification texts to donors - https://phabricator.wikimedia.org/T340242 (10RKumar_WMF) I took the list of recurring created between April-June 2023 which is around 2500 subscriptions. I checked... [07:19:46] 10Fundraising-Backlog: Need a list of active monthly recurring subscription for India - https://phabricator.wikimedia.org/T342328 (10RKumar_WMF) This is great, thank you. So the subscriptions that are not billed after April, are already rejected three times, for example: CID: 45016267, should not this be in "Can... [07:56:00] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_40) - 10https://gerrit.wikimedia.org/r/942132 (owner: 10L10n-bot) [11:30:35] (03PS4) 10Damilare Adedoyin: Throw error in SmashPigRecurringProcessor when invoice_id is missing [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/941937 (https://phabricator.wikimedia.org/T341886) [12:32:13] (03PS5) 10Damilare Adedoyin: Throw error in SmashPigRecurringProcessor when invoice_id is missing [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/941937 (https://phabricator.wikimedia.org/T341886) [12:34:43] (03CR) 10Damilare Adedoyin: "Thanks jgleeson for the review, I've updated the patch to resolve your comments." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/941937 (https://phabricator.wikimedia.org/T341886) (owner: 10Damilare Adedoyin) [12:46:29] (03CR) 10CI reject: [V: 04-1] Throw error in SmashPigRecurringProcessor when invoice_id is missing [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/941937 (https://phabricator.wikimedia.org/T341886) (owner: 10Damilare Adedoyin) [12:48:52] (03CR) 10Damilare Adedoyin: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/941937 (https://phabricator.wikimedia.org/T341886) (owner: 10Damilare Adedoyin) [13:03:12] (03CR) 10CI reject: [V: 04-1] Throw error in SmashPigRecurringProcessor when invoice_id is missing [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/941937 (https://phabricator.wikimedia.org/T341886) (owner: 10Damilare Adedoyin) [13:31:41] (03PS6) 10Damilare Adedoyin: Throw error in SmashPigRecurringProcessor when invoice_id is missing [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/941937 (https://phabricator.wikimedia.org/T341886) [13:45:13] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Email: Civi to Acoustic Export: Job Status Failed - https://phabricator.wikimedia.org/T342879 (10LMedley-WMF) [13:45:44] (03CR) 10CI reject: [V: 04-1] Throw error in SmashPigRecurringProcessor when invoice_id is missing [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/941937 (https://phabricator.wikimedia.org/T341886) (owner: 10Damilare Adedoyin) [13:51:12] (03PS7) 10Damilare Adedoyin: Throw error in SmashPigRecurringProcessor when invoice_id is missing [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/941937 (https://phabricator.wikimedia.org/T341886) [13:52:03] (03CR) 10CI reject: [V: 04-1] Throw error in SmashPigRecurringProcessor when invoice_id is missing [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/941937 (https://phabricator.wikimedia.org/T341886) (owner: 10Damilare Adedoyin) [13:57:08] (03CR) 10Damilare Adedoyin: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/941937 (https://phabricator.wikimedia.org/T341886) (owner: 10Damilare Adedoyin) [14:32:51] (03CR) 10Abijeet Patro: [V: 03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_40) - 10https://gerrit.wikimedia.org/r/942132 (owner: 10L10n-bot) [14:36:18] 10fundraising-tech-ops: encrypt fundraising database client->server communication - https://phabricator.wikimedia.org/T170321 (10Jgreen) [14:36:34] 10fundraising-tech-ops: encrypt fundraising database client->server communication - https://phabricator.wikimedia.org/T170321 (10Jgreen) [14:36:37] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising-tech-ops: Staging in a mess (but don't look until after your public holiday!) - https://phabricator.wikimedia.org/T341029 (10Jgreen) [14:36:39] 10Fundraising Sprint Interstitial ads halfway down the coaster hill, 10Fundraising Sprint Ketchup Flume Ride, 10Fundraising Sprint Never gonna git you update, 10Fundraising-Backlog, and 3 others: Enable SSL for CiviCRM DB connections - https://phabricator.wikimedia.org/T280080 (10Jgreen) [14:50:03] 10Fundraising Sprint Interstitial ads halfway down the coaster hill, 10Fundraising Sprint Ketchup Flume Ride, 10Fundraising Sprint Never gonna git you update, 10Fundraising-Backlog, and 3 others: Enable SSL for CiviCRM DB connections - https://phabricator.wikimedia.org/T280080 (10Jgreen) @Eileenmcnaughton... [16:11:43] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Email: Civi to Acoustic Export: Job Status Failed - https://phabricator.wikimedia.org/T342879 (10KHaggard) Thanks for logging this ticket @LMedley-WMF I logged into Acoustic and tried out running the jobs manually and the civi export files were fo... [18:49:22] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10FR-Email: Civi to Acoustic Export: Job Status Failed - https://phabricator.wikimedia.org/T342879 (10Cstone) Looks like it normally takes ~45 minutes for the export to build and upload, starting at 4am UTC and finishing around 4:45. [21:31:20] (03PS1) 10Damilare Adedoyin: Update redis [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942507 (https://phabricator.wikimedia.org/T342186) [21:32:49] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Patch-For-Review: Deprecated function: Function Redis::delete() is deprecated in Redis_Lock_PhpRedis->lockRelease() - https://phabricator.wikimedia.org/T342186 (10Damilare) a:03Damilare [22:06:18] (03CR) 10Eileen: [C: 03+2] Update redis [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942507 (https://phabricator.wikimedia.org/T342186) (owner: 10Damilare Adedoyin) [22:07:14] (03CR) 10Damilare Adedoyin: [C: 03+2] "Thanks for this. LGTM" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942051 (owner: 10Eileen) [22:09:21] (03PS1) 10Wfan: Add refund script for braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/942513 [22:09:49] (03PS2) 10Wfan: Add refund script for braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/942513 (https://phabricator.wikimedia.org/T342788) [22:10:17] 10Fundraising-Backlog: Send browser data to Adyen on CC transactions so we can optionally toggle on/off 3DS on demand - https://phabricator.wikimedia.org/T341299 (10EMartin) @AKanji-WMF Hi Anil, we are seeing high declines in Italy because we are not sending this browser data and the next campaign is scheduled... [22:21:34] (03Merged) 10jenkins-bot: Slightly clearer wording [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942051 (owner: 10Eileen)