[03:10:04] 10fundraising-tech-ops, 10Infrastructure-Foundations, 10SRE, 10netops: Upgrade pfw to Junos 20+ - https://phabricator.wikimedia.org/T295691 (10Papaul) @Jgreen hello do you think this can be done on May the 16th? [03:11:22] 10fundraising-tech-ops, 10Infrastructure-Foundations, 10SRE, 10netops: Upgrade pfw to Junos 20+ - https://phabricator.wikimedia.org/T295691 (10Papaul) p:05Low→03Medium [03:48:33] (03CR) 10AndyRussG: "Heyy... I see in an e-mail, "Will stop us logging client-side errors for donors coming back from 3DS". Is there a task with more details o" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/779563 (owner: 10Ejegg) [04:48:12] (03CR) 10AndyRussG: "Thanks for this, looking good so far!!! I see this does follow the recommendation in the task description... though... are we sure we don'" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/787512 (https://phabricator.wikimedia.org/T302891) (owner: 10Wfan) [05:13:45] (03CR) 10jerkins-bot: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_35) - 10https://gerrit.wikimedia.org/r/788021 (owner: 10L10n-bot) [15:52:00] hi fr-tech! [15:52:19] Looks like we still need to do something to get the CRM tests passing [15:52:58] Hashar explained that it's a result of the CI images moving to debian bullseye from stretch [15:53:19] and that bullseye has a new mariadb version that doesn't let us connect as root anymore [15:54:03] He wrote a patch to try to swap it to use 'nobody' but it doesn't quite seem to work: https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/787694 [15:56:20] 13:00:26 ERROR 1227 (42000) at line 1: Access denied; you need (at least one of) the SUPER privilege(s) for this operation [15:56:33] hmm, so the mediawiki tests must be creating DBs somehow [15:56:38] let's see how they're doing it [16:05:27] (03CR) 10AndyRussG: "Here are some possibly relevant bits of the console output for tracking down the CI error:" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/787694 (https://phabricator.wikimedia.org/T307178) (owner: 10Hashar) [16:09:54] 10Wikimedia-Fundraising-CiviCRM, 10Continuous-Integration-Infrastructure, 10Patch-For-Review: CiviCRM CI jobs fails when migrating from Stretch to Bullseye - https://phabricator.wikimedia.org/T307178 (10AndyRussG) Thanks so much for all this, @hashar! So, looks like for now we have working CI with the rollba... [16:11:13] (03CR) 10AndyRussG: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/786444 (https://phabricator.wikimedia.org/T286225) (owner: 10Eileen) [16:24:09] (03CR) 10jerkins-bot: [V: 04-1] Remove JP Morgan import [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/786444 (https://phabricator.wikimedia.org/T286225) (owner: 10Eileen) [16:24:54] hi fr-tech ejegg [16:25:32] hiii ejegg cstone fr-tech :) [16:25:39] hi AndyRussG [16:26:04] :) [16:26:26] yeah oops wrt crm tests [16:26:28] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/786444 [16:27:04] however ci was rolled back to stretch already: https://gerrit.wikimedia.org/r/c/integration/config/+/787724/ [16:29:50] 10Wikimedia-Fundraising-CiviCRM, 10Continuous-Integration-Infrastructure, 10Patch-For-Review: CiviCRM CI jobs fails when migrating from Stretch to Bullseye - https://phabricator.wikimedia.org/T307178 (10hashar) That is correct! My ideas is to add a second job to CI which is running with Bullseye and have it... [16:30:22] hmmm not sure what the cause of the current errors are tho: https://integration.wikimedia.org/ci/job/wikimedia-fundraising-civicrm-docker/7934/console [16:30:43] looks like just normal test failures, but I don't know what changed then [16:30:53] maybe an update to buildkit? [16:31:20] see from this line onward: 11:24:02 There were 3 failures: [16:35:40] 10Wikimedia-Fundraising-CiviCRM, 10Continuous-Integration-Infrastructure, 10Patch-For-Review: CiviCRM CI jobs fails when migrating from Stretch to Bullseye - https://phabricator.wikimedia.org/T307178 (10AndyRussG) @hashar thanks! Just to note, actually a previously passing change is now failing. Seems likely... [16:44:47] 10Fundraising Sprint High velocity swallows, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Civi Message Template dynamic content is not populating currency code correctly - https://phabricator.wikimedia.org/T306576 (10AnnWF) 05Open→03In progress a:03AnnWF [16:53:47] (03PS3) 10Wfan: Add check for Adyen modification_reference [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/786390 (https://phabricator.wikimedia.org/T306194) (owner: 10Cstone) [17:06:43] (03CR) 10jerkins-bot: [V: 04-1] Add check for Adyen modification_reference [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/786390 (https://phabricator.wikimedia.org/T306194) (owner: 10Cstone) [17:19:21] (03PS4) 10Wfan: Add check for Adyen modification_reference [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/786390 (https://phabricator.wikimedia.org/T306194) (owner: 10Cstone) [17:28:45] AndyRussG: ahh, that seems to be from a smashpig update [17:30:38] I'll fix those tests [17:31:24] (03CR) 10jerkins-bot: [V: 04-1] Add check for Adyen modification_reference [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/786390 (https://phabricator.wikimedia.org/T306194) (owner: 10Cstone) [17:43:26] (03PS5) 10Wfan: Add check for Adyen modification_reference [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/786390 (https://phabricator.wikimedia.org/T306194) (owner: 10Cstone) [17:45:43] (03PS1) 10Ejegg: Add modification_reference to adyen audit tests [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/788383 [17:58:02] ejegg: did you notice that in 2 of the tests, there's also an expected string that's now a float, looks like? [17:58:05] 11:24:02 - 'fee' => '0.25' [17:58:07] 11:24:02 + 'fee' => 0.25 [18:02:38] ejegg: I have this https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/786390/ with the unittest for modification_reference [18:36:46] AndyRussG: I think the array comparison normally ignores that [18:37:26] but when there's a complete mismatch in another key it dumps all the differences, including the ones that are just type change [18:44:31] (03CR) 10Ejegg: "Maybe we should override get_parent_order_id instead of main_transaction_exists_in_civi ? We should definitely check to see if modificatio" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/786390 (https://phabricator.wikimedia.org/T306194) (owner: 10Cstone) [19:01:58] (03CR) 10Ejegg: "Sorry, I changed my mind, let's keep overriding the main_trans... function. But can you please add a check that the modification_reference" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/786390 (https://phabricator.wikimedia.org/T306194) (owner: 10Cstone) [19:03:17] 10Fundraising-Backlog: Apple Pay first_name can be longer than 64 characters - https://phabricator.wikimedia.org/T295821 (10Cstone) Another one happened, name 169 characters long [19:03:18] wfan sorry to be wishy washy on that [19:03:59] I guess for now it makes sense to search for both if the first doesn't exist [19:04:08] but let's make sure it doesn't crash if that key is missing [19:04:23] and also let's omit that key from the queue message, rather than adding it to the test [19:33:42] OK :) Thanks for the suggestion~ [19:47:10] 10Fundraising-Backlog: Apple Pay first_name can be longer than 64 characters - https://phabricator.wikimedia.org/T295821 (10DStrine) [19:47:12] 10Fundraising-Backlog: Failmail: Field 'first_name' is too long general - https://phabricator.wikimedia.org/T307261 (10DStrine) [19:53:36] (03Abandoned) 10Ejegg: Add modification_reference to adyen audit tests [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/788383 (owner: 10Ejegg) [20:03:05] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/786444 (https://phabricator.wikimedia.org/T286225) (owner: 10Eileen) [20:31:58] (03PS6) 10Wfan: Add check for Adyen modification_reference [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/786390 (https://phabricator.wikimedia.org/T306194) (owner: 10Cstone) [20:38:43] (03PS7) 10Wfan: Add check for Adyen modification_reference [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/786390 (https://phabricator.wikimedia.org/T306194) (owner: 10Cstone) [20:52:00] (03CR) 10jerkins-bot: [V: 04-1] Add check for Adyen modification_reference [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/786390 (https://phabricator.wikimedia.org/T306194) (owner: 10Cstone) [21:03:21] (03PS8) 10Wfan: Add check for Adyen modification_reference [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/786390 (https://phabricator.wikimedia.org/T306194) (owner: 10Cstone) [21:35:29] (03PS9) 10Ejegg: Add check for Adyen modification_reference [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/786390 (https://phabricator.wikimedia.org/T306194) (owner: 10Cstone) [21:35:47] (03CR) 10Ejegg: [C: 03+2] Add check for Adyen modification_reference [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/786390 (https://phabricator.wikimedia.org/T306194) (owner: 10Cstone) [21:48:03] (03Merged) 10jenkins-bot: Add check for Adyen modification_reference [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/786390 (https://phabricator.wikimedia.org/T306194) (owner: 10Cstone) [21:51:23] (03CR) 10Ejegg: "This looks like a good start! The failures are quite odd - I'll look into those stack traces. Ideally we'd have an entry in that dataprovi" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/787512 (https://phabricator.wikimedia.org/T302891) (owner: 10Wfan) [21:52:27] 10Fundraising-Backlog, 10fr-donorservices: Multiple donors showing duplicate billings via Ingenico - https://phabricator.wikimedia.org/T307217 (10DStrine) Hey @EMartin I know you already have an email out on this. @Cstone looked at these and some of them are taking a very long time to settle at ingenico. TLDR... [22:08:15] PROBLEM - check_mysql on payments2003 is CRITICAL: Access denied for user nagios@localhost (using password: YES) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_mysql [22:08:15] PROBLEM - check_payments_wiki on payments2003 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2003.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 638 bytes in 0.040 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_payments_wiki [22:12:48] ^^^ that's me. did a kernel update on the host and mysql is being odd on restart. [22:13:15] PROBLEM - check_mysql on payments2003 is CRITICAL: Access denied for user nagios@localhost (using password: YES) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_mysql [22:13:15] PROBLEM - check_payments_wiki on payments2003 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2003.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 638 bytes in 0.045 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_payments_wiki [22:18:15] RECOVERY - check_mysql on payments2003 is OK: Uptime: 86 Threads: 5 Questions: 229 Slow queries: 0 Opens: 39 Open tables: 35 Queries per second avg: 2.662 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_mysql [22:18:17] RECOVERY - check_payments_wiki on payments2003 is OK: HTTP OK: HTTP/1.1 200 OK - 500 bytes in 0.042 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_payments_wiki [22:24:52] that was bizzare, but sorted. [22:32:13] 10Fundraising-Backlog, 10fr-donorservices: Multiple donors showing duplicate billings via Ingenico - https://phabricator.wikimedia.org/T307217 (10Cstone) There was nothing that stood out as a double charge but some of them are taking a long time to settle. Some data from Ingenico: For CID 3617812: Their Marc... [22:50:01] (03CR) 10Ejegg: "OK, it looks like we're missing a little bit of the request setup. Besides the Request property, we need to set the Output and the Title o" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/787512 (https://phabricator.wikimedia.org/T302891) (owner: 10Wfan) [23:06:48] (03PS9) 10Wfan: Test cases for all available gateway from form chooser [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/787512 (https://phabricator.wikimedia.org/T302891) [23:09:12] (03CR) 10jerkins-bot: [V: 04-1] Test cases for all available gateway from form chooser [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/787512 (https://phabricator.wikimedia.org/T302891) (owner: 10Wfan)