[00:01:35] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Consider whether our triggers are correct regarding handling of trxn_id LIKE 'rfd%' - https://phabricator.wikimedia.org/T350204 (10Eileenmcnaughton) [00:01:49] (03CR) 10CI reject: [V: 04-1] Contribution Import Refactor: inline calls to ImportStatsCollector::getInstance() [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970460 (https://phabricator.wikimedia.org/T240581) (owner: 10Jgleeson) [00:02:00] 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: endowment fiscal & calendar year totals are transposed - https://phabricator.wikimedia.org/T349160 (10Eileenmcnaughton) a:03Eileenmcnaughton [00:02:22] 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: endowment fiscal & calendar year totals are transposed - https://phabricator.wikimedia.org/T349160 (10Eileenmcnaughton) [00:02:29] 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: endowment fiscal & calendar year totals are transposed - https://phabricator.wikimedia.org/T349160 (10Eileenmcnaughton) [00:02:59] (03CR) 10CI reject: [V: 04-1] Contribution Import Refactor: Move main timer naming to fn [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970462 (https://phabricator.wikimedia.org/T240581) (owner: 10Jgleeson) [00:09:38] (03PS1) 10Eileen: Re-run custom field removal - I forgot the civicrm_custom_field delete [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970464 (https://phabricator.wikimedia.org/T347724) [00:10:35] (03CR) 10Eileen: [C: 03+1] "I don't know about the code history- just celebrating removing old code" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970458 (https://phabricator.wikimedia.org/T240581) (owner: 10Jgleeson) [00:16:11] this patch was missed when I removed the fields https://gerrit.wikimedia.org/r/970464 - seems we are getting away with it cos they were already disabled but would be good to get out - it is in a chain but not much of one - not sure if anyone is still working - wfan? [00:16:11] Note I ran it on staging & it went OK [00:16:47] checking [00:17:07] what are civicrm_custom_field? [00:17:15] if that is removable, code looks ok [00:19:30] wfan: yeah so we already rolled out to drop the fields from the wmf_donor table - but I didn't delete the entries in civicrm_custom_field [00:19:50] we don't use the api because of the risk the query will be slow on that table (alhtough it actually wasn't) [00:20:31] ie by doing via sql we can drop all fields in one (possibly slow) sql statement - but if it were slow we would have 20 slow statements [00:31:55] (03CR) 10Wfan: [C: 03+2] "works good locally, and according to Elliott's explanation, "by doing via sql we can drop all fields in one (possibly slow) sql statement " [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970464 (https://phabricator.wikimedia.org/T347724) (owner: 10Eileen) [00:33:49] (03CR) 10Wfan: [C: 03+2] Re-run custom field removal - I forgot the civicrm_custom_field delete (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970464 (https://phabricator.wikimedia.org/T347724) (owner: 10Eileen) [00:33:54] thanks wfan [00:34:04] np, thanks eileen :) [00:48:33] (03CR) 10Eileen: [C: 03+2] "Given this is just a re-run & is blocking a +2d patch of some priority I'm gonna self-+2" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970455 (https://phabricator.wikimedia.org/T345919) (owner: 10Eileen) [00:49:33] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/970468 [00:49:43] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/970468 (owner: 10Eileen) [00:50:50] (03Merged) 10jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/970468 (owner: 10Eileen) [00:51:32] !log civicrm upgraded from 31d53b57 to 6ae3d3fc [00:51:35] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:06:28] 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Civi's preview of receipt: Recurring donations displayed as one-time - https://phabricator.wikimedia.org/T342081 (10Eileenmcnaughton) [01:06:36] 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Civi's preview of receipt: Recurring donations displayed as one-time - https://phabricator.wikimedia.org/T342081 (10Eileenmcnaughton) 05Open→03Resolved [02:31:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14528 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [02:31:12] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14530 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [02:36:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14828 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [02:36:12] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14830 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [02:39:37] ACKNOWLEDGEMENT - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14952 Dwisehaupt known after alter. recovering. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [02:39:37] ACKNOWLEDGEMENT - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14830 Dwisehaupt known after alter. recovering. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [02:39:38] ACKNOWLEDGEMENT - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 8449 Dwisehaupt known after alter. recovering. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [02:39:38] ACKNOWLEDGEMENT - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 14828 Dwisehaupt known after alter. recovering. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [03:20:43] 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog, 10fundraising-tech-ops: CiviCRM point release - https://phabricator.wikimedia.org/T350208 (10Eileenmcnaughton) [03:33:08] (03PS1) 10Eileen: Remove upgrade to mailing_event_queue table [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970477 (https://phabricator.wikimedia.org/T350208) [03:35:46] (03PS1) 10Eileen: CiviCRM 5.68-rc (stock) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970478 (https://phabricator.wikimedia.org/T350208) [03:36:12] RECOVERY - check_mysql on frdb2002 is OK: Uptime: 1762320 Threads: 5 Questions: 378062443 Slow queries: 869 Opens: 4053 Open tables: 1190 Queries per second avg: 214.525 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [03:36:14] (03PS1) 10Eileen: Remove upgrade to mailing_event_queue table [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970479 (https://phabricator.wikimedia.org/T350208) [03:46:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19028 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [03:50:17] 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog, 10fundraising-tech-ops: Fix any rows that do not have the new mailing_id field populated - https://phabricator.wikimedia.org/T350209 (10Eileenmcnaughton) [03:51:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19328 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [03:51:26] (03CR) 10CI reject: [V: 04-1] CiviCRM 5.68-rc (stock) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970478 (https://phabricator.wikimedia.org/T350208) (owner: 10Eileen) [03:51:27] ACKNOWLEDGEMENT - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 19328 Dwisehaupt known replication lag https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [03:52:16] (03CR) 10CI reject: [V: 04-1] Remove upgrade to mailing_event_queue table [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970479 (https://phabricator.wikimedia.org/T350208) (owner: 10Eileen) [03:58:39] (03PS1) 10Eileen: Adjust test for change in html 2 text library [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970481 (https://phabricator.wikimedia.org/T350208) [04:01:12] RECOVERY - check_mysql on frdb2002 is OK: Uptime: 1763821 Threads: 5 Questions: 378063354 Slow queries: 869 Opens: 4053 Open tables: 1190 Queries per second avg: 214.343 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [04:11:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20528 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [04:16:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 20828 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [04:16:12] RECOVERY - check_mysql on frdb1006 is OK: Uptime: 127131 Threads: 13 Questions: 19276800 Slow queries: 167084 Opens: 4827 Open tables: 1788 Queries per second avg: 151.629 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [04:21:12] RECOVERY - check_mysql on frdb2002 is OK: Uptime: 1765020 Threads: 5 Questions: 378064083 Slow queries: 869 Opens: 4053 Open tables: 1190 Queries per second avg: 214.198 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [04:31:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 21728 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [04:36:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22028 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [04:36:14] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22030 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [04:41:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22328 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [04:41:12] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22330 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [04:46:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22628 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [04:46:12] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22631 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [04:51:12] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22930 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [04:51:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 22928 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [04:56:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23228 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [04:56:12] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23230 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [05:01:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23528 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [05:01:12] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23530 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [05:02:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23586 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [05:06:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23828 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [05:06:12] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23830 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [05:07:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 23886 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [05:11:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24128 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [05:11:12] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 7162 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [05:12:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24186 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [05:16:12] RECOVERY - check_mysql on frdb1006 is OK: Uptime: 130731 Threads: 13 Questions: 20940722 Slow queries: 171068 Opens: 4858 Open tables: 1819 Queries per second avg: 160.181 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [05:16:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24428 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [05:17:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24486 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [05:21:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 24728 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [05:22:06] sorry for the spam, these are all known and they are catching up. i have added downtime where needed. [05:23:56] some are being spiky which is annoying. [06:11:12] RECOVERY - check_mysql on frdb2002 is OK: Uptime: 1771620 Threads: 5 Questions: 380281893 Slow queries: 869 Opens: 4054 Open tables: 1191 Queries per second avg: 214.652 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [07:02:56] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_41) - 10https://gerrit.wikimedia.org/r/970529 (owner: 10L10n-bot) [07:57:12] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 1778225 Threads: 6 Questions: 381392986 Slow queries: 871 Opens: 4038 Open tables: 1178 Queries per second avg: 214.479 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [10:51:58] (03CR) 10Jgleeson: Contribution Import Refactor: Move main timer naming to fn (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970462 (https://phabricator.wikimedia.org/T240581) (owner: 10Jgleeson) [10:53:12] (03PS4) 10Jgleeson: Contribution Import Refactor: inline calls to ImportStatsCollector::getInstance() [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970460 (https://phabricator.wikimedia.org/T240581) [10:53:20] (03PS3) 10Jgleeson: Contribution Import Refactor: Move main timer naming to fn [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970462 (https://phabricator.wikimedia.org/T240581) [10:56:00] (03PS5) 10Jgleeson: Contribution Import Refactor: inline calls to ImportStatsCollector::getInstance() [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970460 (https://phabricator.wikimedia.org/T240581) [10:56:22] (03PS4) 10Jgleeson: Contribution Import Refactor: Move main timer naming to fn [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970462 (https://phabricator.wikimedia.org/T240581) [11:11:13] (03CR) 10CI reject: [V: 04-1] Contribution Import Refactor: Move main timer naming to fn [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970462 (https://phabricator.wikimedia.org/T240581) (owner: 10Jgleeson) [11:11:42] (03CR) 10CI reject: [V: 04-1] Contribution Import Refactor: inline calls to ImportStatsCollector::getInstance() [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970460 (https://phabricator.wikimedia.org/T240581) (owner: 10Jgleeson) [11:22:12] (03CR) 10Jgleeson: "Yeah, I think this can go. We're not currently creating any new recurring subscriptions that I can see when testing locally using globalco" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970458 (https://phabricator.wikimedia.org/T240581) (owner: 10Jgleeson) [11:29:40] (03CR) 10Nikerabbit: [V: 03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_41) - 10https://gerrit.wikimedia.org/r/970529 (owner: 10L10n-bot) [11:33:44] (03PS6) 10Jgleeson: Contribution Import Refactor: inline calls to ImportStatsCollector::getInstance() [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970460 (https://phabricator.wikimedia.org/T240581) [11:34:18] (03PS5) 10Jgleeson: Contribution Import Refactor: Move main timer naming to fn [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970462 (https://phabricator.wikimedia.org/T240581) [13:28:56] (03CR) 10Jgleeson: [C: 03+2] "Tested with GPay and ApplePay and both worked as expected! thanks" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/970271 (https://phabricator.wikimedia.org/T346841) (owner: 10Damilare Adedoyin) [13:29:00] (03PS5) 10Ejegg: Contribution Import Refactor: remove old code [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970458 (https://phabricator.wikimedia.org/T240581) (owner: 10Jgleeson) [13:29:11] DI seems to be broken after running setup with this error: Class 'WhichBrowser\Parser' not found [13:29:19] any ideas how to fix [13:29:22] thanks jgleeson [13:29:29] (03CR) 10Ejegg: [C: 03+2] "Safe as houses! Been ages since we got anything new with gateway 'globalcollect'" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970458 (https://phabricator.wikimedia.org/T240581) (owner: 10Jgleeson) [13:30:08] damilare: composer install [13:30:17] on payments [13:30:20] done, no diff [13:30:27] interesting [13:30:49] that's a new lib we pulled in to detect browser headers [13:31:08] (03Merged) 10jenkins-bot: Enable monthly convert on Apple and Google payment methods [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/970271 (https://phabricator.wikimedia.org/T346841) (owner: 10Damilare Adedoyin) [13:31:12] ty ejegg ! [13:31:43] payments branch is on rel1_39 right? [13:31:50] ya should be [13:32:17] what commit are you [13:32:19] on [13:32:23] on payments [13:32:41] 04428d6eb6cc440ee330886e0542288e739eed9f for me [13:32:45] oh my last deploy commit: 87cda4146f8e5e392d703563c8bf72aea0605af9 [13:33:03] let me update that [13:33:39] commit 16acde59111018af15703c48969f6ed9e0d02930 [13:33:41] Author: Elliott Eggleston [13:33:43] Date: Fri Oct 20 10:11:00 2023 -0400 [13:33:45] Update composer.lock with whichbrowser [13:33:47] [13:33:49] Change-Id: I345b2e7b4f568364e8fa8548f521cadcb13b5a80 [13:33:51] commit 87cda4146f8e5e392d703563c8bf72aea0605af9 [13:33:53] Author: Damilare Adedoyin [13:34:04] looks like ejegg pushed up the change right after your commit [13:34:17] unless you've done a new one and I don't have it locally [13:35:23] (03PS7) 10Jgleeson: Contribution Import Refactor: inline calls to ImportStatsCollector::getInstance() [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970460 (https://phabricator.wikimedia.org/T240581) [13:35:29] (03PS6) 10Jgleeson: Contribution Import Refactor: Move main timer naming to fn [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970462 (https://phabricator.wikimedia.org/T240581) [13:35:52] yh I see it after the update [13:36:00] let me reinstall the vendor [13:38:01] damilare: it's fundraising/REL1_39 [13:38:23] there is another branch which is just REL1_39, but that's 'stock' [13:38:58] ohh yeah i'm on fundraising/REL1_39 [13:39:05] I see your commits [13:39:16] probably an issue with composer not installing the package [13:39:18] good stuff - that ought to pull in the new lib when you composer install now [13:39:35] oh huh, did that lock file get up to gerrit OK? [13:40:17] sure looks like it did from my git history [13:40:28] hmm but I wouldn't expect you to see your last commit on the stock REL1_39 branch damilare ? [13:41:11] jgleeson I'm on fundraising/REL1_39 [13:41:16] ah good [13:41:19] did composer install fix it? [13:42:28] no it didn't, ejegg looks this commit only added the lock: I345b2e7b4f568364e8fa8548f521cadcb13b5a80 [13:42:35] we need the composer.json [13:42:37] as well [13:42:51] I had to do a composer require for that specific package [13:43:01] now it's working [13:43:12] I'd push the composer.json patch up [13:44:15] isn't the composer update in DI? [13:44:36] (03PS1) 10Damilare Adedoyin: Update composer json with whichbrowser [core] (fundraising/REL1_39) - 10https://gerrit.wikimedia.org/r/970780 [13:45:30] oh that would make sense because I was doing composer install from the payments root [13:45:43] (03Merged) 10jenkins-bot: Contribution Import Refactor: remove old code [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970458 (https://phabricator.wikimedia.org/T240581) (owner: 10Jgleeson) [13:46:21] it gets added to vendor here https://gerrit.wikimedia.org/r/c/mediawiki/vendor/+/967460 [13:46:35] and added to composer here https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/967266 [13:46:52] composer install on payments should install it due to composer merge [13:47:27] (03Abandoned) 10Damilare Adedoyin: Update composer json with whichbrowser [core] (fundraising/REL1_39) - 10https://gerrit.wikimedia.org/r/970780 (owner: 10Damilare Adedoyin) [13:48:28] hmm, I see it in the DI composer.json but I'm wondering why the composer merge didn't pick it up [13:50:04] I'll need the luck ejegg ha [13:52:43] thanks for spotting that jgleeson, I deleted vendor entirely and did composer install [13:53:05] didn't need to do the composer require [13:53:25] awesome! [13:53:46] I always forget the composer order of things [13:54:21] (03CR) 10Jgleeson: "This looks really useful and straightforward. Could we get a small test to give it a run? Thanks" [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/969196 (https://phabricator.wikimedia.org/T349392) (owner: 10Ejegg) [14:03:23] 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog, 10Patch-For-Review: Enable Monthly Convert for Gpay - https://phabricator.wikimedia.org/T346841 (10Damilare) Hi @Pcoombe, this has been merged. Please can you let us know when would be a good time to have this deployed? [14:04:16] fr-tech, can we hold deploy on payments till pcoombe gives the go ahead ^ [14:09:53] (03CR) 10Jgleeson: [C: 03+2] "This is working great for me locally after updating the custom fields. I can see the 'Venmo user name: @test-venmo-username' on the import" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/967924 (https://phabricator.wikimedia.org/T303422) (owner: 10Wfan) [14:10:58] damilare: sounds fine. we can also cherry pick around commits we don't want to go out in emergencies [14:11:15] but I don't know of anything pressing enough to warrant that [14:11:26] (03PS4) 10Ejegg: Adopt generic PayPal IPN 'rejects' setting for audits [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/969196 (https://phabricator.wikimedia.org/T349392) [14:11:28] cool! [14:11:45] thanks for keeping me honest with the tests jgleeson -- added in PS4 ^^^ [14:11:58] thanks ejegg. looking [14:18:47] (03Merged) 10jenkins-bot: Add venmo user name as external_identifier [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/967924 (https://phabricator.wikimedia.org/T303422) (owner: 10Wfan) [14:24:23] Woah! I'd recommend the Pycharm "interactive lesson" option on the Tip of the Day prompt. It just walked me through the benefits of the local history tool when ctrl+z doesn't cut it and how to play with it in the IDE live. nice [14:24:38] 10Wikimedia-Fundraising-Banners, 10Wikipedia-Android-App-Backlog (Android Release - FY2023-24), 10Wikipedia-iOS-App-Backlog (Design), 10iOS Release FY2023-24: For smaller screens (SE) ensure that the ability to scroll is visible to users - https://phabricator.wikimedia.org/T348695 (10Tsevener) a:03Tsevener [14:24:41] I don't see it on PHPStorm. I might need to update [14:27:24] oh nice [14:27:41] ejegg: so on that patch [14:27:56] are you using the abc.Sequence check as a catch all? [14:27:57] yes? [14:28:08] yeah, anything that we can us 'in' for should be OK [14:28:10] to block ALL values for that field [14:28:30] right, we should be able to block multiple different email addresses for example [14:28:54] ohh I see [14:29:37] my eyes were missing the parentheses [14:29:39] nice! [14:29:43] that's a nice way to do that [14:29:46] thanks! [14:30:15] In the listener it does it slightly differently, casting the config val to array if it's scalar [14:30:22] but that didn't seem like the python way [14:30:43] 10Fundraising-Backlog: SSL error connecting to frdb1003 using python script - https://phabricator.wikimedia.org/T350268 (10Pcoombe) [14:30:56] anyway, now both should be able to handle similar yaml config [14:31:20] (03CR) 10Jgleeson: [C: 03+2] "This looks great and thanks for the test!" [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/969196 (https://phabricator.wikimedia.org/T349392) (owner: 10Ejegg) [14:32:54] (03Merged) 10jenkins-bot: Adopt generic PayPal IPN 'rejects' setting for audits [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/969196 (https://phabricator.wikimedia.org/T349392) (owner: 10Ejegg) [14:36:08] I've been tinkering around with a project in Python to help with quickly generating unit tests. I've had mixed results pasting methods and classes into Chatgpt but with a bit of tinkering I think it could really speed that side of things up. I decided to use Python over PHP to wrap api calls to the OpenAI API and then I went down a ABC rabbit hole when trying to figure out the best way to use [14:36:10] abstraction in Monty Python :) [14:37:41] the dream being you can just feed it a gerrit/gitlab class URL and it gives you a usable starting point with some idea of coverage [14:42:52] (03PS1) 10Ejegg: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/970794 [14:43:10] wow jgleeson, that sounds like that would be a pretty popular tool [14:43:15] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/970794 (owner: 10Ejegg) [14:43:42] (03Merged) 10jenkins-bot: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/970794 (owner: 10Ejegg) [14:49:13] !log fundraising python tools upgraded from 65f101e4 to a4cbbbe7 [14:49:15] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [14:51:24] (03CR) 10Damilare Adedoyin: "Thanks for making the changes wfan, we are really close. Looks like some of the changes led to the payment breaking during testing, I did " [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/963853 (https://phabricator.wikimedia.org/T347606) (owner: 10Wfan) [14:55:09] (03CR) 10Wfan: Add api to remove recuring payment token then delete customer from venmo vault for one-time donor (031 comment) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/963853 (https://phabricator.wikimedia.org/T347606) (owner: 10Wfan) [14:57:27] Jeff_Green or dwisehaupt: I've just made a patch for the puppet repo to add one more setting to the paypal audit config in /etc - it's in my homedir on frpm as ~/puppet/0001-Audit-setting-to-reject-donations-from-PayPal-grant-.patch [14:58:36] ejegg: is there any coordination needed when I roll it out, or good to go anytime? [14:58:53] Jeff_Green: any time is fine [14:59:14] the code to handle it is already live but it doesn't care when the setting is missing [14:59:42] ok cool, I'll get it done in a bit [15:00:07] 10Fundraising Sprint Turtles that are robotic that destroy the whole world with their foot theory, 10Fundraising-Backlog: Adding and storing rml_source data for RML donations reporting - https://phabricator.wikimedia.org/T321163 (10AKanji-WMF) Thank you - keeping in near future, will revisit with you in Q3! [15:04:23] 10Fundraising-Backlog: SSL error connecting to frdb1003 using python script - https://phabricator.wikimedia.org/T350268 (10Pcoombe) p:05Triage→03Unbreak! Bumping to UBN since this blocks us from doing any banner testing [15:07:50] Jeff_Green: that looks maybe connected with the SSL-db-traffic maintenance work, right? ^^^ [15:08:57] ejegg: yeah, looks like. I'm looking at similar for another user-level script too [15:12:47] 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog: Suppress Paypal Grant CID to avoid duplicate db recording - https://phabricator.wikimedia.org/T349392 (10Ejegg) The IPN stuff is done. The audit code is deployed, and there's one setting about to be deployed for that by ops. [15:17:07] 10Fundraising-Backlog, 10fundraising-tech-ops: Modify fundstatgen.py to use mysql connection encryption. - https://phabricator.wikimedia.org/T350067 (10Jgreen) [15:17:09] 10Fundraising-Backlog, 10fundraising-tech-ops: Modify fundstatgen.py to use mysql connection encryption. - https://phabricator.wikimedia.org/T350067 (10Jgreen) a:05Jgreen→03None [15:19:05] 10Fundraising-Backlog, 10fundraising-tech-ops: SSL error connecting to frdb1003 using python script - https://phabricator.wikimedia.org/T350268 (10greg) [15:21:54] (03CR) 10Ejegg: "Observation on old-style function naming" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970460 (https://phabricator.wikimedia.org/T240581) (owner: 10Jgleeson) [15:27:50] lol Halloween's over ^^^ [15:28:40] 10Fundraising-Backlog, 10fundraising-tech-ops: SSL error connecting to frdb1003 using python script - https://phabricator.wikimedia.org/T350268 (10Jgreen) @Pcoombe Sorry--this was indeed caused by the SSL connection requirement. It's an easy fix--just change "frdb1003" to "frdb1003.frack.eqiad.wmnet" in your p... [15:33:15] 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Get CiviProxy API4 patch merged upstream - https://phabricator.wikimedia.org/T349762 (10Ejegg) a:03Ejegg [15:42:51] 10Fundraising-Backlog, 10fundraising-tech-ops: SSL error connecting to frdb1003 using python script - https://phabricator.wikimedia.org/T350268 (10Pcoombe) 05Open→03Resolved a:03Pcoombe Worked perfectly, thanks @Jgreen ! [15:55:07] (03PS3) 10Ejegg: Update email amender extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/968791 (https://phabricator.wikimedia.org/T343899) (owner: 10Eileen) [15:56:29] (03CR) 10Ejegg: [C: 03+2] "Looks good! New API entity permissions seems to prevent submitting the form even if someone enables inputs in browser tools." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/968791 (https://phabricator.wikimedia.org/T343899) (owner: 10Eileen) [15:57:48] (03PS17) 10Wfan: Add api to remove recuring payment token then delete customer from venmo vault for one-time donor [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/963853 (https://phabricator.wikimedia.org/T347606) [16:13:23] (03Merged) 10jenkins-bot: Update email amender extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/968791 (https://phabricator.wikimedia.org/T343899) (owner: 10Eileen) [16:23:34] 10Fundraising-Backlog: Venmo non-native browser flow posing checkout issues for donors - https://phabricator.wikimedia.org/T348699 (10Dwisehaupt) [16:24:24] 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface: Form variant with lock icon next to credit card number label - https://phabricator.wikimedia.org/T348339 (10Dwisehaupt) [16:25:20] 10Fundraising-Backlog: Frontend Import of Fundraiseup Chargeback - https://phabricator.wikimedia.org/T347881 (10Dwisehaupt) [16:25:59] 10Fundraising-Backlog, 10FR-dlocal, 10MediaWiki-extensions-DonationInterface: DLocal: Increment sequence number after processor API call fails - https://phabricator.wikimedia.org/T347599 (10Dwisehaupt) [16:26:35] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Recurring-Donations: Write API action to port recurrings from one processor to another - https://phabricator.wikimedia.org/T340078 (10Ejegg) 05Open→03Declined Going to do this in the db instead [16:26:37] 10Fundraising Sprint This LAN is your LAN, 10Fundraising-Backlog: AMEX rejected transactions through Ingenico are to be ported to Adyen - https://phabricator.wikimedia.org/T338523 (10Ejegg) [16:35:51] 10fundraising-tech-ops: encrypt fundraising database client->server communication - https://phabricator.wikimedia.org/T170321 (10Jgreen) [16:35:54] 10Fundraising-Backlog, 10fundraising-tech-ops: SSL error connecting to frdb1003 using python script - https://phabricator.wikimedia.org/T350268 (10Jgreen) [16:38:41] 10fundraising-tech-ops: encrypt fundraising database client->server communication - https://phabricator.wikimedia.org/T170321 (10Jgreen) [16:39:20] 10fundraising-tech-ops: encrypt fundraising database client->server communication - https://phabricator.wikimedia.org/T170321 (10Jgreen) [16:47:53] (03PS1) 10Damilare Adedoyin: Ensure snooze activity is saved in the specified contact when email has duplicates. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970810 (https://phabricator.wikimedia.org/T349277) [17:24:12] 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog: Enable Monthly Convert for Gpay - https://phabricator.wikimedia.org/T346841 (10Pcoombe) Donatewiki and our current banners have both been updated to not show pre-payment monthly convert for Google Pay. This is fine to deploy now from my pov. Thanks! [17:24:23] 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog, 10MW-1.42-notes (1.42.0-wmf.4; 2023-11-07): Enable Monthly Convert for Apple Pay - https://phabricator.wikimedia.org/T346840 (10Pcoombe) Donatewiki and our current banners have both been updated to not show pre-payment monthly convert for Apple Pay. Th... [17:40:32] PROBLEM - Host frpm1002 is DOWN: PING CRITICAL - Packet loss = 100% [17:42:05] ^^^ known while we are doing this update. [17:43:16] RECOVERY - Host frpm1002 is UP: 3 [18:01:24] 10Fundraising-Backlog: Measure donation processing rate (once we decide what that is) - https://phabricator.wikimedia.org/T342013 (10XenoRyet) p:05High→03Medium [18:51:09] PROBLEM - check_ipsec on fran1001 is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: frban1001_v4 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_ipsec [18:52:09] PROBLEM - check_ipsec on frban2001 is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: fran2001_v4 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2001&service=check_ipsec [18:52:15] PROBLEM - check_ipsec on frban1001 is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: fran1001_v4 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_ipsec [18:52:15] RECOVERY - check_kafkatee on frban1001 is OK: OK: brokers:9 topics:1 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_kafkatee [18:56:15] RECOVERY - check_ipsec on fran1001 is OK: Strongswan OK - 1 ESP OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_ipsec [18:57:09] RECOVERY - check_ipsec on frban2001 is OK: Strongswan OK - 1 ESP OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2001&service=check_ipsec [18:57:15] RECOVERY - check_ipsec on frban1001 is OK: Strongswan OK - 1 ESP OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban1001&service=check_ipsec [19:01:10] 10fundraising-tech-ops: Upgrade Fundraising puppet servers to bookworm and puppetserver 7.x - https://phabricator.wikimedia.org/T347353 (10Jgreen) 05Open→03Resolved a:03Jgreen [19:01:12] 10fundraising-tech-ops: Fundraising environment support for Debian 12 (Bookworm) - https://phabricator.wikimedia.org/T344290 (10Jgreen) [19:02:36] (03PS18) 10Wfan: Add api to remove recuring payment token then delete customer from venmo vault for one-time donor [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/963853 (https://phabricator.wikimedia.org/T347606) [19:03:02] (03CR) 10CI reject: [V: 04-1] Add api to remove recuring payment token then delete customer from venmo vault for one-time donor [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/963853 (https://phabricator.wikimedia.org/T347606) (owner: 10Wfan) [19:22:15] PROBLEM - check_puppetrun on frdb2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 5 minutes ago with 37 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_puppetrun [19:22:15] PROBLEM - check_puppetrun on frdb2003 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 5 minutes ago with 37 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_puppetrun [19:22:15] PROBLEM - check_puppetrun on frbackup2002 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 5 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbackup2002&service=check_puppetrun [19:22:15] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 38 failures. Last run 5 minutes ago with 38 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [19:22:16] PROBLEM - check_puppetrun on frqueue2002 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 5 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_puppetrun [19:22:16] PROBLEM - check_puppetrun on frban2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 5 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2001&service=check_puppetrun [19:22:19] PROBLEM - check_puppetrun on frdata2001 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 5 minutes ago with 31 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_puppetrun [19:23:15] PROBLEM - check_puppetrun on frqueue2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 6 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_puppetrun [19:23:15] PROBLEM - check_puppetrun on frbast2002 is CRITICAL: CRITICAL: Puppet has 29 failures. Last run 6 minutes ago with 29 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast2002&service=check_puppetrun [19:23:19] PROBLEM - check_puppetrun on payments2003 is CRITICAL: CRITICAL: Puppet has 40 failures. Last run 6 minutes ago with 40 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_puppetrun [19:25:09] PROBLEM - check_puppetrun on frlog2001 is CRITICAL: CRITICAL: Puppet has 29 failures. Last run 8 minutes ago with 29 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frlog2001&service=check_puppetrun [19:25:15] PROBLEM - check_puppetrun on frmon2002 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 8 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon2002&service=check_puppetrun [19:25:19] PROBLEM - check_puppetrun on frmx2001 is CRITICAL: CRITICAL: Puppet has 29 failures. Last run 8 minutes ago with 29 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx2001&service=check_puppetrun [19:26:15] PROBLEM - check_puppetrun on frdb2002 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 9 minutes ago with 37 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_puppetrun [19:26:25] PROBLEM - check_puppetrun on frpig2001 is CRITICAL: CRITICAL: Puppet has 35 failures. Last run 9 minutes ago with 35 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [19:27:15] RECOVERY - check_puppetrun on frdb2003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_puppetrun [19:27:15] PROBLEM - check_puppetrun on frdb2001 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 10 minutes ago with 37 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_puppetrun [19:27:15] RECOVERY - check_puppetrun on frbackup2002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbackup2002&service=check_puppetrun [19:27:16] PROBLEM - check_puppetrun on civi2001 is CRITICAL: CRITICAL: Puppet has 38 failures. Last run 10 minutes ago with 38 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [19:27:17] PROBLEM - check_puppetrun on frqueue2002 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 10 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_puppetrun [19:27:17] PROBLEM - check_puppetrun on frban2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 10 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2001&service=check_puppetrun [19:27:19] PROBLEM - check_puppetrun on frdata2001 is CRITICAL: CRITICAL: Puppet has 31 failures. Last run 10 minutes ago with 31 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_puppetrun [19:28:15] PROBLEM - check_puppetrun on frqueue2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 11 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_puppetrun [19:28:15] PROBLEM - check_puppetrun on frbast2002 is CRITICAL: CRITICAL: Puppet has 29 failures. Last run 11 minutes ago with 29 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast2002&service=check_puppetrun [19:28:19] PROBLEM - check_puppetrun on payments2003 is CRITICAL: CRITICAL: Puppet has 40 failures. Last run 11 minutes ago with 40 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_puppetrun [19:30:09] PROBLEM - check_puppetrun on frmon2002 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 13 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon2002&service=check_puppetrun [19:30:10] RECOVERY - check_puppetrun on frlog2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frlog2001&service=check_puppetrun [19:30:19] RECOVERY - check_puppetrun on frmx2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmx2001&service=check_puppetrun [19:31:15] PROBLEM - check_puppetrun on frdb2002 is CRITICAL: CRITICAL: Puppet has 37 failures. Last run 14 minutes ago with 37 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_puppetrun [19:31:25] RECOVERY - check_puppetrun on frpig2001 is OK: OK: Puppet is currently enabled, last run 55 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frpig2001&service=check_puppetrun [19:32:15] RECOVERY - check_puppetrun on frdb2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_puppetrun [19:32:15] RECOVERY - check_puppetrun on civi2001 is OK: OK: Puppet is currently enabled, last run 41 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi2001&service=check_puppetrun [19:32:15] RECOVERY - check_puppetrun on frban2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2001&service=check_puppetrun [19:32:17] RECOVERY - check_puppetrun on frqueue2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_puppetrun [19:32:19] RECOVERY - check_puppetrun on frdata2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_puppetrun [19:33:15] RECOVERY - check_puppetrun on frbast2002 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbast2002&service=check_puppetrun [19:33:15] RECOVERY - check_puppetrun on frqueue2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2001&service=check_puppetrun [19:33:19] RECOVERY - check_puppetrun on payments2003 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_puppetrun [19:34:11] (03PS19) 10Wfan: Add api to remove recuring payment token then delete customer from venmo vault for one-time donor [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/963853 (https://phabricator.wikimedia.org/T347606) [19:35:15] RECOVERY - check_puppetrun on frmon2002 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon2002&service=check_puppetrun [19:35:37] 10fundraising-tech-ops: Fundraising environment support for Debian 12 (Bookworm) - https://phabricator.wikimedia.org/T344290 (10Jgreen) [19:35:39] 10fundraising-tech-ops: Puppet 7.x support in Fundraising environment - https://phabricator.wikimedia.org/T345207 (10Jgreen) 05Resolved→03In progress Reopening this because puppet agents seem to be hitting a lot of timeouts trying to talk to puppetserver. Lots of: ` Could not evaluate: Could not retrieve fi... [19:36:16] RECOVERY - check_puppetrun on frdb2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_puppetrun [19:37:46] (03PS1) 10Cstone: Add example of how to run [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/970851 [19:38:22] (03CR) 10Ejegg: [C: 03+2] Add example of how to run [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/970851 (owner: 10Cstone) [19:38:54] (03Merged) 10jenkins-bot: Add example of how to run [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/970851 (owner: 10Cstone) [19:55:53] 10fundraising-tech-ops: Puppet 7.x support in Fundraising environment - https://phabricator.wikimedia.org/T345207 (10Jgreen) Tripled the RAM allocated to java, and its running a little better for the moment. [19:57:15] PROBLEM - check_puppetrun on pay-lvs1003 is CRITICAL: CRITICAL: Puppet has 32 failures. Last run 5 minutes ago with 32 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1003&service=check_puppetrun [19:57:15] PROBLEM - check_puppetrun on frmon1002 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 6 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon1002&service=check_puppetrun [20:01:15] PROBLEM - check_puppetrun on frauth2002 is CRITICAL: CRITICAL: Puppet has 34 failures. Last run 9 minutes ago with 34 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2002&service=check_puppetrun [20:02:15] RECOVERY - check_puppetrun on pay-lvs1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=pay-lvs1003&service=check_puppetrun [20:02:15] RECOVERY - check_puppetrun on frmon1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon1002&service=check_puppetrun [20:06:15] RECOVERY - check_puppetrun on frauth2002 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2002&service=check_puppetrun [20:10:12] 10Fundraising Sprint Uptown cURL: Enable all payment methods for LATAM pre-test in Q2 - https://phabricator.wikimedia.org/T349857 (10AnnWF) >>! In T349857#9290750, @RKumar_WMF wrote: > All banner links given do not show recurring option, can you please provide? Also, please confirm that recurring is only availab... [20:14:41] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/970857 [20:15:12] ejegg: did you take a look at the different civi version patches I put up [20:15:33] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/970857 (owner: 10Eileen) [20:16:23] (03Merged) 10jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/970857 (owner: 10Eileen) [20:46:21] eileen: I started to, and started looking through release notes [20:46:34] but have mostly been looking at the api4 / authx stuff [20:47:05] anyway, looking back at the feedback on the pull request I guess I need something a bit more configurable than the current patch [20:47:25] still shouldn't need much extra work, now that I know what headers and params to look for [20:48:07] fr-tech I need to pick up my kid, so I'll probably miss a bunch of civi fortnightly [20:52:23] (03PS1) 10Damilare Adedoyin: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/970859 [20:56:22] (03PS2) 10Damilare Adedoyin: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/970859 [20:57:13] (03Abandoned) 10Damilare Adedoyin: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/970859 (owner: 10Damilare Adedoyin) [20:57:43] (03PS1) 10Damilare Adedoyin: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/970862 [20:58:47] /fr-tech, starting the DI deploy, is anyone around to look through this as there are number of deleted/changed files ^ [20:59:41] yeah i guess a dlocal and venmo donation would help test those? I can do that damilare [20:59:55] thanks! [21:05:34] (03CR) 10Damilare Adedoyin: [C: 03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/970862 (owner: 10Damilare Adedoyin) [21:06:18] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/970862 (owner: 10Damilare Adedoyin) [21:08:22] (03PS1) 10Damilare Adedoyin: Update DonationInterface vendor submodule [core] (fundraising/REL1_39) - 10https://gerrit.wikimedia.org/r/970867 [22:00:18] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising-tech-ops: Do a prune on detail data for activity records of type thank you - https://phabricator.wikimedia.org/T325123 (10Dwisehaupt) Post cleanup query: MariaDB [civicrm]> select YEAR(activity_date_time),IF(ISNULL(details),'no text', 'ha... [22:01:26] 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2324Q2 maintenance window - https://phabricator.wikimedia.org/T337579 (10Dwisehaupt) [22:01:54] 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2324Q2 maintenance window - https://phabricator.wikimedia.org/T337579 (10Dwisehaupt) [22:02:38] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Create 2 new stock fields - quantity & ticker - https://phabricator.wikimedia.org/T350313 (10Eileenmcnaughton) [22:06:30] 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2324Q2 maintenance window - https://phabricator.wikimedia.org/T337579 (10Dwisehaupt) [22:07:09] 10Fundraising Sprint Uptown cURL, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Create 2 new stock fields - quantity & ticker - https://phabricator.wikimedia.org/T350313 (10Eileenmcnaughton) [22:18:51] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising-tech-ops: Do a prune on detail data for activity records of type thank you - https://phabricator.wikimedia.org/T325123 (10Eileenmcnaughton) @Dwisehaupt any idea of the impact on db size? It seems like there are about half a million more ro... [22:20:09] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising-tech-ops: Do a prune on detail data for activity records of type thank you - https://phabricator.wikimedia.org/T325123 (10Dwisehaupt) I haven't done the compaction yet. That will be done after the other queries complete. Probably this even... [23:13:25] 10Wikimedia-Fundraising-Banners, 10Wikipedia-Android-App-Backlog (Android Release - FY2023-24), 10Wikipedia-iOS-App-Backlog (Design), 10iOS Release FY2023-24: For smaller screens (SE) ensure that the ability to scroll is visible to users - https://phabricator.wikimedia.org/T348695 (10Tsevener) Hi @cmadeo,... [23:36:32] (03PS2) 10Eileen: CiviCRM 5.68-rc (stock) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970478 (https://phabricator.wikimedia.org/T350208) [23:36:34] (03PS2) 10Eileen: Remove upgrade to mailing_event_queue table [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970479 (https://phabricator.wikimedia.org/T350208) [23:36:36] (03PS2) 10Eileen: Adjust test for change in html 2 text library [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970481 (https://phabricator.wikimedia.org/T350208) [23:48:38] (03PS3) 10Eileen: CiviCRM 5.68-rc (stock) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970478 (https://phabricator.wikimedia.org/T350208) [23:48:40] (03PS3) 10Eileen: Remove upgrade to mailing_event_queue table [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970479 (https://phabricator.wikimedia.org/T350208) [23:48:42] (03PS3) 10Eileen: Adjust test for change in html 2 text library [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970481 (https://phabricator.wikimedia.org/T350208) [23:48:47] ejegg: I think the 5.68 is gtg [23:56:22] (03PS1) 10Eileen: Add 2 new stock fields - quantity & ticker [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/970879 (https://phabricator.wikimedia.org/T350313) [23:56:35] ejegg: also added a patch for a couple more fields I discussed with Melanie eariler [23:58:04] ok, will take a look