[00:07:03] (03PS1) 10Eileen: Improve query criteria on disputes [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229213 [00:07:39] (03CR) 10CI reject: [V:04-1] Improve query criteria on disputes [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229213 (owner: 10Eileen) [00:24:07] 06Fundraising-Backlog: Handle Third-Party Donation Emails as Secondary with Source Tracking - https://phabricator.wikimedia.org/T414538#11539909 (10AnnWF) [00:35:48] (03PS1) 10Eileen: Add calculated payout rows for Braintree files [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229222 [00:36:21] (03CR) 10CI reject: [V:04-1] Add calculated payout rows for Braintree files [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229222 (owner: 10Eileen) [00:36:35] (03PS1) 10Eileen: Add calculated payout rows for Braintree files [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229223 [00:37:32] (03CR) 10CI reject: [V:04-1] Add calculated payout rows for Braintree files [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229223 (owner: 10Eileen) [00:39:15] (03PS2) 10Eileen: Improve query criteria on disputes [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229213 [00:39:15] (03PS2) 10Eileen: Add calculated payout rows for Braintree files [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229223 [00:40:09] (03CR) 10CI reject: [V:04-1] Improve query criteria on disputes [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229213 (owner: 10Eileen) [00:40:33] (03CR) 10CI reject: [V:04-1] Add calculated payout rows for Braintree files [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229223 (owner: 10Eileen) [00:47:52] (03PS3) 10Eileen: Improve query criteria on disputes [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229213 [00:47:52] (03PS3) 10Eileen: Add calculated payout rows for Braintree files [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229223 [00:48:46] (03CR) 10CI reject: [V:04-1] Add calculated payout rows for Braintree files [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229223 (owner: 10Eileen) [00:49:38] (03Abandoned) 10Eileen: Add calculated payout rows for Braintree files [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229222 (owner: 10Eileen) [00:51:04] (03PS4) 10Eileen: Add calculated payout rows for Braintree files [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229223 [01:02:45] 14Fundraising Sprint Michelin Star, 03Fundraising Sprint: ARMchair, 14Fundraising Sprint: Vixen, 06Fundraising-Backlog, and 2 others: 2nd Recurring Failure Email in Civi - https://phabricator.wikimedia.org/T365488#11540031 (10Cstone) [02:06:33] (03PS1) 10Eileen: Upgrade phpsec and add compatibility layer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229246 [02:15:08] (03PS2) 10Eileen: Upgrade phpsec and add compatibility layer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229246 [02:23:08] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing, 13Patch-For-Review: NCOA Update import not working - https://phabricator.wikimedia.org/T410574#11540097 (10Lars) @MDemosWMF I just did the whole file, with these settings: {F71579997} Work... [02:24:53] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Provide checksum_value token (which doesn't include cs=) - https://phabricator.wikimedia.org/T415092#11540114 (10Lars) [02:25:23] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Provide checksum_value token (which doesn't include cs=) - https://phabricator.wikimedia.org/T415092#11540116 (10Lars) [[ https://github.com/civicrm/civicrm-core/pull/34531 | Upstream PR for option 2 ]] [02:25:46] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Provide checksum_value token (which doesn't include cs=) - https://phabricator.wikimedia.org/T415092#11540117 (10Lars) [02:51:07] (03PS1) 10Eileen: maintenance: add SFTPDownload (unifies dlocal + PayPal audit downloaders) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229257 [02:51:52] (03CR) 10CI reject: [V:04-1] maintenance: add SFTPDownload (unifies dlocal + PayPal audit downloaders) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229257 (owner: 10Eileen) [03:02:24] (03CR) 10Eileen: [C:03+2] Mark unsub event rows processed when no email match [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229194 (https://phabricator.wikimedia.org/T409499) (owner: 10Ejegg) [03:21:11] (03PS2) 10Eileen: maintenance: add SFTPDownload (unifies dlocal + PayPal audit downloaders) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229257 [03:25:16] (03CR) 10Eileen: "I haven't read the ticket super carefully but this seems to be an expected donation amount and maybe the field should be called that - the" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229165 (https://phabricator.wikimedia.org/T386031) (owner: 10Lars SG) [03:26:25] (03CR) 10Eileen: "Also - extending type donation - will that mean the table size is huge? Or is it only rows if there is data - probably the latter but I di" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229165 (https://phabricator.wikimedia.org/T386031) (owner: 10Lars SG) [03:47:51] 03Fundraising Sprint: ARMchair, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing, and 2 others: Chariot import mapping template - 2 soft credits - https://phabricator.wikimedia.org/T411555#11540174 (10Lars) @MDemosWMF Great, if you can get a screenshot and share the import b... [04:01:22] 03Fundraising Sprint: ARMchair, 06Fundraising-Backlog, 10fundraising-tech-ops: Create more generic sftp download for reconciliation reports - https://phabricator.wikimedia.org/T415161 (10Eileenmcnaughton) 03NEW [04:13:12] PROBLEM - check_load on frban1002 is CRITICAL: LOAD CRITICAL - total load average: 119.82, 94.28, 50.63 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [04:15:18] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 84.86, 97.00, 58.53 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [04:19:25] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Allow creation of addt soft credits and other entities on import - https://phabricator.wikimedia.org/T415163 (10Lars) 03NEW [04:19:41] (03PS5) 10Cstone: Add second recurring failure email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1199940 (https://phabricator.wikimedia.org/T365488) [04:20:18] PROBLEM - check_load on frban2002 is CRITICAL: LOAD CRITICAL - total load average: 37.02, 62.80, 54.80 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [04:24:51] (03PS3) 10Eileen: maintenance: add SFTPDownload (unifies dlocal + PayPal audit downloaders) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229257 (https://phabricator.wikimedia.org/T415161) [04:25:34] (03CR) 10CI reject: [V:04-1] maintenance: add SFTPDownload (unifies dlocal + PayPal audit downloaders) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229257 (https://phabricator.wikimedia.org/T415161) (owner: 10Eileen) [04:33:10] RECOVERY - check_load on frban1002 is OK: LOAD OK - total load average: 3.19, 8.37, 22.95 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1002&service=check_load [04:35:16] RECOVERY - check_load on frban2002 is OK: LOAD OK - total load average: 2.17, 5.57, 22.86 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_load [05:17:05] (03CR) 10Cstone: [C:03+2] Braintree chargeback - lost rather than won = WE lost [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229208 (owner: 10Eileen) [05:17:34] (03CR) 10Cstone: [C:03+2] Do not swallow graphql errors [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229209 (owner: 10Eileen) [05:17:36] (03Merged) 10jenkins-bot: Braintree chargeback - lost rather than won = WE lost [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229208 (owner: 10Eileen) [05:18:04] (03Merged) 10jenkins-bot: Do not swallow graphql errors [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229209 (owner: 10Eileen) [05:20:48] (03CR) 10Cstone: [C:03+2] Improve query criteria on disputes [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229213 (owner: 10Eileen) [05:21:20] (03Merged) 10jenkins-bot: Improve query criteria on disputes [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229213 (owner: 10Eileen) [05:22:45] (03CR) 10Cstone: [C:03+2] Add calculated payout rows for Braintree files [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229223 (owner: 10Eileen) [05:23:16] (03Merged) 10jenkins-bot: Add calculated payout rows for Braintree files [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229223 (owner: 10Eileen) [05:42:58] (03PS1) 10Cstone: Update smash-pig to v1.1.1 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229437 [05:43:31] (03CR) 10Cstone: [C:03+2] Update smash-pig to v1.1.1 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229437 (owner: 10Cstone) [05:45:50] !log config revision changed from 92dff26b to d75e49b5 [05:45:52] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:53:54] !log config revision changed from d75e49b5 to db79212a [05:53:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:57:36] (03PS4) 10Eileen: maintenance: add SFTPDownload (unifies dlocal + PayPal audit downloaders) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229257 (https://phabricator.wikimedia.org/T415161) [05:58:05] (03CR) 10CI reject: [V:04-1] maintenance: add SFTPDownload (unifies dlocal + PayPal audit downloaders) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229257 (https://phabricator.wikimedia.org/T415161) (owner: 10Eileen) [05:58:50] (03PS5) 10Eileen: maintenance: add SFTPDownload (unifies dlocal + PayPal audit downloaders) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229257 (https://phabricator.wikimedia.org/T415161) [05:59:16] (03CR) 10CI reject: [V:04-1] maintenance: add SFTPDownload (unifies dlocal + PayPal audit downloaders) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229257 (https://phabricator.wikimedia.org/T415161) (owner: 10Eileen) [06:02:03] (03Merged) 10jenkins-bot: Update smash-pig to v1.1.1 [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229437 (owner: 10Cstone) [06:02:15] (03PS6) 10Eileen: maintenance: add SFTPDownload (unifies dlocal + PayPal audit downloaders) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229257 (https://phabricator.wikimedia.org/T415161) [06:02:48] (03CR) 10CI reject: [V:04-1] maintenance: add SFTPDownload (unifies dlocal + PayPal audit downloaders) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229257 (https://phabricator.wikimedia.org/T415161) (owner: 10Eileen) [06:03:52] (03PS7) 10Eileen: maintenance: add SFTPDownload (unifies dlocal + PayPal audit downloaders) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229257 (https://phabricator.wikimedia.org/T415161) [06:10:35] (03PS1) 10Eileen: Upgrading wikimedia/smash-pig (v1.1.0 => v1.1.1): [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1229443 [06:10:53] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1229444 [06:11:03] (03CR) 10Eileen: [C:03+2] Upgrading wikimedia/smash-pig (v1.1.0 => v1.1.1): [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1229443 (owner: 10Eileen) [06:11:18] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1229444 (owner: 10Eileen) [06:14:21] !log civicrm upgraded from 9cdb1915 to d137ca87 [06:14:24] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [06:16:34] (03CR) 10Eileen: "Currently working for paypal and there is a job in process control it works with." [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229257 (https://phabricator.wikimedia.org/T415161) (owner: 10Eileen) [06:32:15] (03Merged) 10jenkins-bot: Upgrading wikimedia/smash-pig (v1.1.0 => v1.1.1): [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1229443 (owner: 10Eileen) [12:57:55] /nick damilare|brb [14:59:14] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing: Creation of new anonymous records on donation import - https://phabricator.wikimedia.org/T394731#11542079 (10Lars) About 2000 anon contacts have been merged. [15:06:51] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Add text to payments form on badly formatted links - https://phabricator.wikimedia.org/T414497#11542126 (10Damilare) [15:52:17] hah, I just added a long explanation of the acoustic failmails at https://wikitech.wikimedia.org/wiki/Fundraising/Data_and_flow/Failmail_zoo#Fail_Mail_(civi1002)_run-job:_Silverpop_emails_-_Check_main_database_file_upload_completed_(DatabaseUpdate)_failed_with_code_1 [15:52:53] and after saving realized I'd added a lot of those same details at https://wikitech.wikimedia.org/wiki/Fundraising/Data_and_flow/Failmail_zoo#Fail_Mail_(civi1001)_run-job:_Check_forget_me_processing_failed_with_code_1 [17:22:11] FIRING: [2x] PfwCoreBGPDown: Fundraising Firewall core BGP session down between pfw1-codfw and (null) (10.195.0.248) - group VPN - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DPfwCoreBGPDown [17:25:13] PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [17:25:19] PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [17:26:09] PROBLEM - check_mysql on frdb2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [17:26:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:27:09] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [17:27:11] PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql [17:27:15] PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [17:27:15] PROBLEM - check_redis_donor_prefs on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 450 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 6 days 18 hours - replication_delay is 450, memory use is 2.00M (peak 2.02M, 0.10% of max, fragmentation 4.29%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [17:27:15] PROBLEM - check_redis on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 446 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 26 keys, up 6 days 18 hours - replication_delay is 446, memory use is 538.35M (peak 706.03M, 6.81% of max, fragmentation 1.04%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [17:27:15] PROBLEM - check_redis_donor_prefs on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 451 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 6 days 18 hours - replication_delay is 451, memory use is 2.00M (peak 2.04M, 0.10% of max, fragmentation 4.30%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [17:27:15] PROBLEM - check_redis on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 447 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 26 keys, up 6 days 18 hours - replication_delay is 447, memory use is 538.33M (peak 706.05M, 6.81% of max, fragmentation 1.04%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [17:28:11] PROBLEM - check_mysql on frdb1007 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql [17:28:49] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-Donation-Processing, 13Patch-For-Review: NCOA Update import not working - https://phabricator.wikimedia.org/T410574#11542720 (10MDemosWMF) @Lars Tried again - here is the error and some screenshots to show form... [17:30:15] PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [17:30:19] PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [17:31:04] Jeff_Green: things ok? ^ [17:31:09] PROBLEM - check_mysql on frdb2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [17:31:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:31:36] i think it must have been another cross-colo-vpn flap [17:32:02] cool [17:32:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [17:32:11] PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql [17:32:15] PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [17:32:19] PROBLEM - check_redis on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 745 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 26 keys, up 6 days 18 hours - replication_delay is 745, memory use is 538.35M (peak 706.03M, 6.81% of max, fragmentation 1.04%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [17:32:19] PROBLEM - check_redis_donor_prefs on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 749 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 6 days 18 hours - replication_delay is 749, memory use is 2.00M (peak 2.02M, 0.10% of max, fragmentation 4.29%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [17:32:20] PROBLEM - check_redis on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 747 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 26 keys, up 6 days 18 hours - replication_delay is 747, memory use is 538.33M (peak 706.05M, 6.81% of max, fragmentation 1.04%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [17:32:21] PROBLEM - check_redis_donor_prefs on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 751 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 6 days 18 hours - replication_delay is 751, memory use is 2.00M (peak 2.04M, 0.10% of max, fragmentation 4.30%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [17:33:11] PROBLEM - check_mysql on frdb1007 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql [17:35:13] PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [17:35:21] PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [17:36:09] PROBLEM - check_mysql on frdb2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [17:36:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:37:09] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [17:37:11] PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql [17:37:15] PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [17:37:17] PROBLEM - check_redis on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 1045 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 26 keys, up 6 days 18 hours - replication_delay is 1045, memory use is 538.35M (peak 706.03M, 6.81% of max, fragmentation 1.04%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [17:37:19] PROBLEM - check_redis_donor_prefs on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 1050 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 6 days 18 hours - replication_delay is 1050, memory use is 2.00M (peak 2.02M, 0.10% of max, fragmentation 4.29%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [17:37:19] PROBLEM - check_redis on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 1048 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 26 keys, up 6 days 18 hours - replication_delay is 1048, memory use is 538.33M (peak 706.05M, 6.81% of max, fragmentation 1.04%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [17:37:21] PROBLEM - check_redis_donor_prefs on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 1052 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 6 days 18 hours - replication_delay is 1052, memory use is 2.00M (peak 2.04M, 0.10% of max, fragmentation 4.30%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [17:38:11] PROBLEM - check_mysql on frdb1007 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql [17:40:13] PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [17:40:15] PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [17:41:09] PROBLEM - check_mysql on frdb2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [17:41:15] PROBLEM - check_mysql on frdb2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:42:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [17:42:11] PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql [17:42:15] PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [17:42:17] PROBLEM - check_redis on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 1345 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 26 keys, up 6 days 18 hours - replication_delay is 1345, memory use is 538.35M (peak 706.03M, 6.81% of max, fragmentation 1.04%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [17:42:19] PROBLEM - check_redis_donor_prefs on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 1349 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 6 days 18 hours - replication_delay is 1349, memory use is 2.00M (peak 2.02M, 0.10% of max, fragmentation 4.29%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [17:42:19] PROBLEM - check_redis on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 1347 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 26 keys, up 6 days 18 hours - replication_delay is 1347, memory use is 538.33M (peak 706.05M, 6.81% of max, fragmentation 1.04%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [17:42:21] PROBLEM - check_redis_donor_prefs on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 1351 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 6 days 18 hours - replication_delay is 1351, memory use is 2.00M (peak 2.04M, 0.10% of max, fragmentation 4.30%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [17:43:11] PROBLEM - check_mysql on frdb1007 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql [17:45:11] RECOVERY - check_mysql on payments2006 is OK: Uptime: 577744 Threads: 6 Questions: 2624745 Slow queries: 0 Opens: 145 Open tables: 139 Queries per second avg: 4.543 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [17:45:15] RECOVERY - check_mysql on payments2005 is OK: Uptime: 579157 Threads: 6 Questions: 2607158 Slow queries: 0 Opens: 145 Open tables: 139 Queries per second avg: 4.501 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [17:46:11] RECOVERY - check_mysql on frdb2004 is OK: Uptime: 575394 Threads: 4 Questions: 68869750 Slow queries: 333 Opens: 150298 Open tables: 1488 Queries per second avg: 119.691 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10271 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [17:46:13] RECOVERY - check_mysql on frdb2005 is OK: Uptime: 577907 Threads: 5 Questions: 69254469 Slow queries: 357 Opens: 150910 Open tables: 1470 Queries per second avg: 119.836 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 10275 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [17:47:11] RECOVERY - check_mysql on frdb2003 is OK: Uptime: 5330505 Threads: 137 Questions: 1930930191 Slow queries: 28448 Opens: 1100240 Open tables: 2944 Queries per second avg: 362.241 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [17:47:11] RECOVERY - check_mysql on frdata2002 is OK: Uptime: 579378 Threads: 4 Questions: 265409 Slow queries: 0 Opens: 135 Open tables: 129 Queries per second avg: 0.458 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql [17:47:11] RESOLVED: [2x] PfwCoreBGPDown: Fundraising Firewall core BGP session down between pfw1-codfw and (null) (10.195.0.248) - group VPN - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DPfwCoreBGPDown [17:47:15] RECOVERY - check_mysql on payments2004 is OK: Uptime: 580091 Threads: 6 Questions: 2616107 Slow queries: 0 Opens: 143 Open tables: 137 Queries per second avg: 4.509 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [17:47:17] RECOVERY - check_redis on frqueue2002 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 27 keys, up 6 days 18 hours - replication_delay is 0, memory use is 533.63M (peak 706.03M, 6.74% of max, fragmentation 1.04%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [17:47:17] RECOVERY - check_redis_donor_prefs on frqueue2002 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 6 days 18 hours - replication_delay is 3, memory use is 2.00M (peak 2.02M, 0.10% of max, fragmentation 4.29%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [17:47:19] RECOVERY - check_redis on frqueue2003 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 27 keys, up 6 days 18 hours - replication_delay is 1, memory use is 533.63M (peak 706.05M, 6.75% of max, fragmentation 1.04%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [17:47:19] RECOVERY - check_redis_donor_prefs on frqueue2003 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 6 days 18 hours - replication_delay is 4, memory use is 2.00M (peak 2.04M, 0.10% of max, fragmentation 4.31%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [17:48:11] RECOVERY - check_mysql on frdb1007 is OK: Uptime: 506888 Threads: 4 Questions: 241884402 Slow queries: 113 Opens: 130503 Open tables: 2224 Queries per second avg: 477.194 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql [18:11:14] PROBLEM - check_mysql on frdb2005 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [18:16:14] RECOVERY - check_mysql on frdb2005 is OK: Uptime: 579708 Threads: 5 Questions: 69784811 Slow queries: 363 Opens: 151742 Open tables: 1470 Queries per second avg: 120.379 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 9712 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [18:36:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3797 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [18:41:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2389 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [18:46:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1220 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [18:51:13] RECOVERY - check_mysql on frdb2005 is OK: Uptime: 581807 Threads: 5 Questions: 74520323 Slow queries: 367 Opens: 153734 Open tables: 1480 Queries per second avg: 128.084 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [18:59:53] (03PS1) 10Lars SG: Ensure {wmf_url.unsubscribe} and other tokens work correctly [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229650 (https://phabricator.wikimedia.org/T415097) [19:02:42] (03PS3) 10Eileen: Upgrade phpsec and add compatibility layer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229246 [19:32:19] 10Fundraising analytics stack: implement google sheets proxy settings - https://phabricator.wikimedia.org/T412844#11543114 (10AStein-WMF) 05Open→03Resolved [19:41:41] (03CR) 10Lars SG: [C:03+2] Upgrade phpsec and add compatibility layer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229246 (owner: 10Eileen) [19:48:59] (03CR) 10Ejegg: [C:03+2] "This looks good! Just curious why the expire action is in API3 while the rest is API4" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1227869 (https://phabricator.wikimedia.org/T412429) (owner: 10Lars SG) [19:57:53] (03PS1) 10Ejegg: Require checksum_invalidator for wmf-civicrm [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229669 [20:07:15] (03Merged) 10jenkins-bot: Upgrade phpsec and add compatibility layer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229246 (owner: 10Eileen) [20:07:55] (03PS8) 10Eileen: maintenance: add SFTPDownload (unifies dlocal + PayPal audit downloaders) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229257 (https://phabricator.wikimedia.org/T415161) [20:07:55] (03PS1) 10Eileen: Switch to phpsec2 [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229671 [20:08:23] (03Merged) 10jenkins-bot: Add checksum invalidator extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1227869 (https://phabricator.wikimedia.org/T412429) (owner: 10Lars SG) [20:08:35] (03CR) 10CI reject: [V:04-1] Switch to phpsec2 [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229671 (owner: 10Eileen) [20:11:30] (03PS1) 10Eileen: Update phpsec to v3 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1229673 [20:11:43] (03CR) 10Eileen: [C:03+2] Update phpsec to v3 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1229673 (owner: 10Eileen) [20:12:11] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1229674 [20:12:21] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1229674 (owner: 10Eileen) [20:13:07] (03CR) 10Lars SG: [C:03+2] Require checksum_invalidator for wmf-civicrm [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229669 (owner: 10Ejegg) [20:21:37] (03PS1) 10Eileen: Opps I added by mistake - not merged upstream [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1229677 [20:22:35] (03CR) 10Wfan: [C:03+2] Opps I added by mistake - not merged upstream [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1229677 (owner: 10Eileen) [20:22:44] !log civicrm upgraded from d137ca87 to 02a558a2 [20:22:46] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:27:54] (03PS1) 10Eileen: Fix my vendor mis-commit [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1229678 [20:28:10] (03CR) 10Eileen: [C:03+2] Fix my vendor mis-commit [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1229678 (owner: 10Eileen) [20:30:10] (03Merged) 10jenkins-bot: Update phpsec to v3 [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1229673 (owner: 10Eileen) [20:36:30] !log config revision changed from db79212a to 8d56295b - disable jobs so I can run settle queue heavily [20:36:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:39:52] !log config revision changed from 8d56295b to 485f8e69 [20:39:54] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:40:26] remind eileen to turn the queues back on in 2 hours :) ^ [21:07:42] (03PS1) 10Cstone: Fix renamed script [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1229690 [21:11:14] (03Merged) 10jenkins-bot: Opps I added by mistake - not merged upstream [wikimedia/fundraising/crm/vendor] - 10https://gerrit.wikimedia.org/r/1229677 (owner: 10Eileen) [21:11:14] (03Merged) 10jenkins-bot: Fix my vendor mis-commit [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1229678 (owner: 10Eileen) [21:30:34] (03CR) 10Damilare Adedoyin: [C:03+2] Fix renamed script [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1229690 (owner: 10Cstone) [21:30:37] (03CR) 10Damilare Adedoyin: [V:03+2 C:03+2] Fix renamed script [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1229690 (owner: 10Cstone) [21:31:15] (03PS2) 10Lars SG: Set up Major Gifts Engagement activity type [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229165 (https://phabricator.wikimedia.org/T386031) [21:31:57] (03PS1) 10Jforrester: build: Drop back-compatibility for testing-access-wrapper, 4+ required for PHP 8.5 [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1229706 [21:32:57] (03CR) 10Lars SG: "It's extending the activity type MajorGiftsEngagement, so that should be OK." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229165 (https://phabricator.wikimedia.org/T386031) (owner: 10Lars SG) [21:35:21] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Donor portal: Only show donations from donors with same email if it is primary - https://phabricator.wikimedia.org/T415225 (10Lars) 03NEW [21:41:21] (03PS1) 10Lars SG: Donor portal: Only show donations from donors with same email if it is primary [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229712 (https://phabricator.wikimedia.org/T415225) [21:42:11] PROBLEM - check_memory on fransw2001 is CRITICAL: CRIT Memory 97% used. Largest process: trino-server-co (1567) = 95.5% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_memory [21:43:46] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: Donor portal: Only show donations from donors with same email if it is primary - https://phabricator.wikimedia.org/T415225#11543468 (10Lars) a:03Lars [21:47:09] PROBLEM - check_memory on fransw2001 is CRITICAL: CRIT Memory 98% used. Largest process: trino-server-co (1567) = 97.4% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_memory [21:47:14] (03PS1) 10Lars SG: Show last donor portal login date in CiviCRM [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229713 (https://phabricator.wikimedia.org/T413112) [21:52:09] PROBLEM - check_memory on fransw2001 is CRITICAL: CRIT Memory 98% used. Largest process: trino-server-co (1567) = 97.4% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_memory [21:57:11] RECOVERY - check_memory on fransw2001 is OK: OK Memory 61% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_memory [22:42:34] 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 06FR-donorrelations, 0725 Dec Q3 Planning: Civi: enable search by merchant reference # for failed donations - https://phabricator.wikimedia.org/T402425#11543542 (10Cstone) fr-tech talked a bit about making a Payment attem... [23:00:17] !log config revision changed from 485f8e69 to 5a1bd9b8 [23:00:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:13:21] 06Fundraising-Backlog: Recurring failure email link improvements - https://phabricator.wikimedia.org/T415229 (10Cstone) 03NEW [23:21:02] (03PS1) 10Lars SG: Add {contact.checksum_value} (without cs=) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229779 (https://phabricator.wikimedia.org/T415092) [23:23:54] (03CR) 10Eileen: [C:03+2] Add {contact.checksum_value} (without cs=) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229779 (https://phabricator.wikimedia.org/T415092) (owner: 10Lars SG) [23:38:45] (03Merged) 10jenkins-bot: Add {contact.checksum_value} (without cs=) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1229779 (https://phabricator.wikimedia.org/T415092) (owner: 10Lars SG) [23:40:18] !log config revision changed from 5a1bd9b8 to 8d6fc61d [23:40:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:44:50] (03PS9) 10Eileen: maintenance: add SFTPDownload (unifies dlocal + PayPal audit downloaders) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1229257 (https://phabricator.wikimedia.org/T415161)