[00:01:58] (03PS1) 10Eileen: Add some dev files to gitignore [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/914936 [00:02:04] ejegg: are you ok with ^^ [00:03:13] wellllll, since it's just for us [00:03:15] :) [00:03:29] (03CR) 10Ejegg: [C: 03+2] Add some dev files to gitignore [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/914936 (owner: 10Eileen) [00:05:13] (03PS1) 10Eileen: Run civix.upgrade on thank you tests [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/914937 [00:05:16] thanks ejegg [00:16:07] (03Merged) 10jenkins-bot: Add some dev files to gitignore [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/914936 (owner: 10Eileen) [00:25:25] I'm losing my brain [00:26:09] it turns out the tests in wmf-thankyou\tests\phpunit\Civi\Api4 don't run but the ones in wmf-civicrm\tests\phpunit\Civi\Api4 do - and I can't spot the difference [00:27:59] (03CR) 10Ejegg: [C: 03+2] Run civix.upgrade on thank you tests [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/914937 (owner: 10Eileen) [00:33:37] (03PS1) 10Ejegg: Update SmashPig [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/914944 [00:34:24] (03PS2) 10Ejegg: Update SmashPig [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/914944 [00:34:55] (03CR) 10Ejegg: [C: 03+2] Update SmashPig [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/914944 (owner: 10Ejegg) [00:40:54] (03Merged) 10jenkins-bot: Run civix.upgrade on thank you tests [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/914937 (owner: 10Eileen) [00:47:14] (03Merged) 10jenkins-bot: Update SmashPig [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/914944 (owner: 10Ejegg) [01:09:17] 10Fundraising Sprint Istanbool (not Constantidouble), 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-AutoTY-Email, 10Unplanned-Sprint-Work: Some Thank-you related tests won't run.... - https://phabricator.wikimedia.org/T335933 (10Eileenmcnaughton) [01:15:35] wow eileen, i'm sure that would have taken me a LOT longer to get to the bottom of [01:15:49] it's a mind bender [01:15:58] chatting with Tim on chat on it [01:29:42] !log config revision changed from 26147e89 to 121a864a - disabling populate as it keeps rolling back so prob another overlong row [01:29:43] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:43:52] 10Fundraising-Backlog, 10fundraising-tech-ops, 10fr-donorservices: Expose email-prefs-wiki to public - https://phabricator.wikimedia.org/T335793 (10MBeat33) Great question, @Ejegg other suggestions mypreferences.wikimedia.org myprofile.wikimedia.org donorpreference.wikimedia.org donorsettings.wikimedia.org... [01:57:11] (03PS1) 10Eileen: Limit length for payments_form in backfill [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/914995 [01:57:20] ejegg: another one of those ^^ [02:13:28] (03PS1) 10Wfan: Update trigger [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/915007 (https://phabricator.wikimedia.org/T332074) [02:16:19] (03PS2) 10Ejegg: Limit length for payments_form in backfill [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/914995 (owner: 10Eileen) [02:16:26] (03CR) 10Ejegg: [C: 03+2] Limit length for payments_form in backfill [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/914995 (owner: 10Eileen) [02:16:48] eileen: oh, I added a smashpig update, so that deploy will need a vendor update [02:17:04] I can prep that now [02:18:38] oh hey, it lets us 'git push' for vendor [02:18:44] well that's slightly faster [02:19:04] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/915008 [02:19:06] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/915008 (owner: 10Ejegg) [02:28:26] (03Merged) 10jenkins-bot: Limit length for payments_form in backfill [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/914995 (owner: 10Eileen) [02:28:28] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/915008 (owner: 10Ejegg) [02:33:12] !log civicrm upgraded from b97aaa08 to 05523a9d [02:33:14] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:33:16] ejegg: I pushed it out [02:35:01] !log config revision changed from 121a864a to 5ac52d82 [02:35:01] reenable tracking_populate [02:35:02] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:38:25] thanks eileen ! [02:38:30] ok, I'm outta here [02:38:35] seems to have gotten all slow [02:38:36] have a good one all [02:39:06] hrm? [02:39:16] the populate [02:39:29] I think the server might be busier - not sure [02:39:44] gonna reduce the batch size [02:41:53] (03PS9) 10Ejegg: Ingenico disentanglement: no GC inheritance [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/913961 (https://phabricator.wikimedia.org/T260725) [02:41:55] (03PS2) 10Ejegg: WIP: Delete GlobalCollect [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/913977 (https://phabricator.wikimedia.org/T260725) [02:42:11] !log config revision changed from 5ac52d82 to 7ac11236 reduce batch size, avoid failmail [02:42:12] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:43:33] (03CR) 10CI reject: [V: 04-1] WIP: Delete GlobalCollect [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/913977 (https://phabricator.wikimedia.org/T260725) (owner: 10Ejegg) [03:07:53] (03PS1) 10Eileen: Render-wip [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/915031 [03:07:55] (03PS1) 10Eileen: Fix tests not running [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/915032 (https://phabricator.wikimedia.org/T335933) [03:09:28] (03PS2) 10Eileen: Fix tests not running [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/915032 (https://phabricator.wikimedia.org/T335933) [03:09:30] (03PS2) 10Eileen: Render-wip [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/915031 [03:19:46] (03CR) 10CI reject: [V: 04-1] Fix tests not running [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/915032 (https://phabricator.wikimedia.org/T335933) (owner: 10Eileen) [03:22:07] (03CR) 10CI reject: [V: 04-1] Render-wip [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/915031 (owner: 10Eileen) [03:22:09] (03CR) 10CI reject: [V: 04-1] Fix tests not running [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/915032 (https://phabricator.wikimedia.org/T335933) (owner: 10Eileen) [04:31:13] (03PS18) 10Cstone: Add validate and collect part for direct upi bt [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/903690 (https://phabricator.wikimedia.org/T324294) (owner: 10Wfan) [04:38:58] (03CR) 10Cstone: "Looking good, just a couple of comments." [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/903690 (https://phabricator.wikimedia.org/T324294) (owner: 10Wfan) [04:48:32] (03CR) 10DannyS712: [C: 03+2] build: Updating npm dependencies [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/915000 (owner: 10Libraryupgrader) [05:53:57] 10Fundraising-Backlog, 10Recurring-Donations, 10fr-donorservices, 10FR-Email: Export currency and amount of active recurring donation to Acoustic - https://phabricator.wikimedia.org/T335926 (10MSuijkerbuijk_WMF) Hi @Ejegg I'm not sure I understand what the implications of not exporting Yes/No are, but if t... [06:01:45] 10Fundraising-Backlog, 10fundraising-tech-ops, 10fr-donorservices: Expose email-prefs-wiki to public - https://phabricator.wikimedia.org/T335793 (10MSuijkerbuijk_WMF) Hi @Ejegg I need to understand a bit more the domain structure you are suggesting. Will one main domain then have 1/ a sub domain for the Em... [06:40:35] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_40) - 10https://gerrit.wikimedia.org/r/915217 (owner: 10L10n-bot) [07:34:30] (03CR) 10Nikerabbit: [V: 03+2] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_40) - 10https://gerrit.wikimedia.org/r/915217 (owner: 10L10n-bot) [10:02:11] 10Fundraising-Backlog, 10fundraising-tech-ops: Vanity URL for upcoming Annual Report mailing - https://phabricator.wikimedia.org/T335920 (10RLewis) @Dwisehaupt since its the Annual Report, its a different campaign so we'd want to track it differently, could https://fundraising.wikimedia.org/AnnualReport2122 w... [12:31:17] PROBLEM - check_puppetrun on fran1001 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 14 seconds ago with 8 failures. Failed resources (up to 3 shown): File[/etc/ssh/userkeys/superset],File[/etc/sudoers.d/fundraising_code_update],File[/etc/sudoers.d],File[/home/superset/.ssh/authorized_keys2] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_puppetrun [12:35:23] ACKNOWLEDGEMENT - check_puppetrun on fran1001 is CRITICAL: CRITICAL: Puppet has 8 failures. Last run 14 seconds ago with 8 failures. Failed resources (up to 3 shown): File[/etc/ssh/userkeys/superset],File[/etc/sudoers.d/fundraising_code_update],File[/etc/sudoers.d],File[/home/superset/.ssh/authorized_keys2] Jeff_Green known https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_puppetrun [12:37:13] PROBLEM - check_puppetrun on frav1002 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 6 minutes ago with 9 failures. Failed resources (up to 3 shown): File[/etc/aide/aide.conf],File[/etc/default/aide],File[/etc/ssh/userkeys/aiderator],File[/home/aiderator/.ssh/authorized_keys2] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_puppetrun [12:37:48] ACKNOWLEDGEMENT - check_puppetrun on frav1002 is CRITICAL: CRITICAL: Puppet has 9 failures. Last run 6 minutes ago with 9 failures. Failed resources (up to 3 shown): File[/etc/aide/aide.conf],File[/etc/default/aide],File[/etc/ssh/userkeys/aiderator],File[/home/aiderator/.ssh/authorized_keys2] Jeff_Green known https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_puppetrun [13:17:13] RECOVERY - check_puppetrun on frav1002 is OK: OK: Puppet is currently enabled, last run 44 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_puppetrun [13:31:19] RECOVERY - check_puppetrun on fran1001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_puppetrun [13:47:16] 10fundraising-tech-ops, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:rack/setup/install frav1003 - https://phabricator.wikimedia.org/T334400 (10Jgreen) [13:48:10] 10fundraising-tech-ops, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:rack/setup/install frav1003 - https://phabricator.wikimedia.org/T334400 (10Jgreen) a:05Jclark-ctr→03None [13:48:43] 10fundraising-tech-ops, 10SRE: Q3:rack/setup/install frbast2002, frauth2002 - https://phabricator.wikimedia.org/T334505 (10Jgreen) a:05Jgreen→03None [13:49:25] 10fundraising-tech-ops, 10SRE: Q3:rack/setup/install frmon2002 - https://phabricator.wikimedia.org/T334501 (10Jgreen) a:05Jgreen→03Dwisehaupt [13:49:35] 10fundraising-tech-ops, 10Observability-Alerting: Puppet CA certificate kafka_fundraising_client is about to expire - https://phabricator.wikimedia.org/T335768 (10Jgreen) a:03Jgreen [14:11:38] 10fundraising-tech-ops: Q3:rack/setup/install frbast2002, frauth2002 - https://phabricator.wikimedia.org/T334505 (10jijiki) @RobH, I am not aware if there is anything actionable from any specific team in #SRE, this I am removing the tag. Please re-tag the task if needed [15:09:28] Hi Team - Mariana flagged this task - I feel like this has prob been resolved by now, can I close? [15:09:29] https://phabricator.wikimedia.org/T107378 [15:09:39] checking! [15:10:01] another one from the vaults... [15:10:06] yeah 2015! [15:10:13] yeah there might be some old bad data in there though [15:14:58] this patch seems to touch on the same problem but predates the ticket https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/93135 [15:15:11] another one from the waybackmachine [15:15:23] "Also fixes a currency import error on subscr_signup. This is a very old bug. [15:15:25] The impact is that civicrm_contribution_recur.currency is null for a random [15:15:27] portion of paypal signups. [15:15:29] " [15:18:15] none from this year jgleeson [15:18:32] and looks like 809 in total from like 2011 [15:19:11] (03CR) 10Damilare Adedoyin: [C: 03+1] "Thanks for this ejegg, tested and works. I noticed the BankPaymentProvider is still broken for rtbt/txns that are on the dev list, so I dr" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/913961 (https://phabricator.wikimedia.org/T260725) (owner: 10Ejegg) [15:19:31] last one was in 2013 [15:21:24] ah [15:21:35] feels like a red herring then anilk [15:22:07] ill post that info on the task jgleeson anilk [15:22:13] ty [15:22:21] ...and resolve? :) - thanks everyone for having a look [15:22:49] 10Fundraising-Backlog, 10FR-Paypal, 10Recurring-Donations: Paypal recurring are sometimes added with currency=null - https://phabricator.wikimedia.org/T107378 (10Cstone) Looking in civicrm_contribution_recur: 809 total with currency NULL All from between 2011 and 2013. [15:23:00] hah yeah! [15:23:34] 10Fundraising-Backlog, 10FR-Paypal, 10Recurring-Donations: Paypal recurring are sometimes added with currency=null - https://phabricator.wikimedia.org/T107378 (10Cstone) 05Open→03Resolved a:03Cstone Hasn't happened since 2013 [15:23:52] 🔔 [15:25:28] now only 1000 tasks in the "unscheduled" column ;) [15:25:37] oh, 999! yay [15:27:16] 🔥 [15:43:32] (03CR) 10Jgleeson: [C: 03+1] "This looks like a great way to handle the issue related to the original bug report, but after testing it, I am no longer able to recreate " [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/914016 (https://phabricator.wikimedia.org/T332693) (owner: 10Ejegg) [15:46:38] (03CR) 10Ejegg: "Thanks for the CR! To test it, I added this line to the submit hander in Adyen.js:" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/914016 (https://phabricator.wikimedia.org/T332693) (owner: 10Ejegg) [15:48:50] (03CR) 10Ejegg: Ingenico disentanglement: no GC inheritance (031 comment) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/913961 (https://phabricator.wikimedia.org/T260725) (owner: 10Ejegg) [15:51:41] (03PS1) 10Ejegg: Move test code to test class [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/915731 [15:57:22] (03PS1) 10Ejegg: Remove unused EncodingMangler class [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/915735 [16:08:49] interesting ejegg. that line in the adyen JS didn't trigger the error for me although I do now see the original error after removing the debug=true query param [16:09:28] maybe the actual resizeobserver error is happening first and preventing the second deliberate one [16:11:28] oh huh - remind me how you trigger the 'real' error? [16:13:42] 10Fundraising-Backlog, 10FR-Japan: Make the Japan "V1" payment wiki variant the new control - https://phabricator.wikimedia.org/T329807 (10AKanji-WMF) Flagging next Japan campaigns tentatively slated for Q3 (Jan/Feb). [16:20:09] ejegg: nothing specific. I'm just hitting this URL locally https://paymentstest2.wmcloud.org/index.php?title=Special:AdyenCheckoutGateway&appeal=JimmyQuote&payment_method=cc&recurring=0&uselang=en&language=en¤cy=USD&amount=10&country=US&first_name=Jimmy&last_name=Wales&street_address=1+Montgomery+Street&city=San+Francisco&state_province=CA&postal_code=94104&email=jwales%40example.com [16:20:58] ok, and that error generally happens in Chrome? [16:21:29] it does for me [16:21:48] ejegg: I think I can still review it [16:21:56] now that I can recreate it [16:21:58] cool, thanks! [16:29:26] 10Fundraising Sprint Istanbool (not Constantidouble), 10Fundraising-Backlog: Investigation: Scope Segmentation in Civi & Acoustic - https://phabricator.wikimedia.org/T335376 (10ERoden-WMF) Major Gifts probably == Major Donors defined as 'cumulative giving in calendar year of 10k+' - I've commented on the sh... [16:41:12] PROBLEM - check_puppetrun on frdb1006 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 5 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb [16:41:12] ice=check_puppetrun [16:45:12] PROBLEM - check_puppetrun on frlog2001 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 6 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frl [16:45:12] rvice=check_puppetrun [16:46:18] PROBLEM - check_puppetrun on frdb1006 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 9 seconds ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb [16:46:18] ice=check_puppetrun [16:47:12] PROBLEM - check_puppetrun on frdb1003 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 1 minute ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1 [16:47:12] ce=check_puppetrun [16:48:18] PROBLEM - check_puppetrun on frdb1004 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb [16:48:18] ice=check_puppetrun [16:48:18] PROBLEM - check_puppetrun on frlog1002 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 2 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frl [16:48:18] rvice=check_puppetrun [16:49:31] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Customized Communications Preference field is not working correctly in Civi - https://phabricator.wikimedia.org/T146096 (10MDemosWMF) @RLewis Hi there, for DAF letters we use the First/Last name fields that export from Civi, so I don't have experience wit... [16:50:12] PROBLEM - check_puppetrun on frlog2001 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 3 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frl [16:50:12] rvice=check_puppetrun [16:51:12] PROBLEM - check_puppetrun on frdb2002 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 5 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb [16:51:12] ice=check_puppetrun [16:51:18] PROBLEM - check_puppetrun on frdb1006 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 5 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb [16:51:18] ice=check_puppetrun [16:51:18] PROBLEM - check_puppetrun on fran1001 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 5 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-python3-abba],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host [16:51:18] &service=check_puppetrun [16:52:12] PROBLEM - check_puppetrun on frdb2001 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 6 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb [16:52:12] ice=check_puppetrun [16:52:12] PROBLEM - check_puppetrun on frdb1003 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 6 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb [16:52:12] ice=check_puppetrun [16:52:12] PROBLEM - check_puppetrun on frav1002 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 6 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/buster-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/buster-frack.gpg],File[/etc/apt/preferences.d/buster-frack-perccli],File[/etc/apt/sources.list.d/buster-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&ser [16:52:12] k_puppetrun [16:52:13] PROBLEM - check_puppetrun on frdb2003 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 6 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb [16:52:13] ice=check_puppetrun [16:52:13] PROBLEM - check_puppetrun on frbackup2002 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 6 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host= [16:52:14] 002&service=check_puppetrun [16:53:18] PROBLEM - check_puppetrun on frdb1004 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 7 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb [16:53:18] ice=check_puppetrun [16:53:18] PROBLEM - check_puppetrun on frlog1002 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 7 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frl [16:53:18] rvice=check_puppetrun [16:55:12] RECOVERY - check_puppetrun on frlog2001 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frlog2001&service=check_puppetrun [16:55:22] PROBLEM - check_puppetrun on frdb1005 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 8 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb [16:55:22] ice=check_puppetrun [16:56:12] RECOVERY - check_puppetrun on frdb2002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_puppetrun [16:56:16] RECOVERY - check_puppetrun on frdb1006 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_puppetrun [16:56:16] RECOVERY - check_puppetrun on fran1001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_puppetrun [16:57:12] RECOVERY - check_puppetrun on frav1002 is OK: OK: Puppet is currently enabled, last run 3 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_puppetrun [16:57:12] RECOVERY - check_puppetrun on frdb2001 is OK: OK: Puppet is currently enabled, last run 8 seconds ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_puppetrun [16:57:12] PROBLEM - check_puppetrun on frdb1003 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 11 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frd [16:57:12] vice=check_puppetrun [16:57:13] PROBLEM - check_puppetrun on frbackup2002 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 11 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host [16:57:13] 2002&service=check_puppetrun [16:57:13] PROBLEM - check_puppetrun on frdb2003 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 11 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frd [16:57:13] vice=check_puppetrun [16:58:18] PROBLEM - check_puppetrun on frdb1004 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 12 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frd [16:58:18] vice=check_puppetrun [16:58:18] RECOVERY - check_puppetrun on frlog1002 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frlog1002&service=check_puppetrun [17:00:22] PROBLEM - check_puppetrun on frdb1005 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 13 minutes ago with 6 failures. Failed resources (up to 3 shown): Exec[dearmor /var/lib/apt/keyrings/bullseye-frack.asc],Exec[whack /etc/apt/trusted.gpg.d/bullseye-frack.gpg],File[/etc/apt/preferences.d/bullseye-frack-perccli],File[/etc/apt/sources.list.d/bullseye-frack.list] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frd [17:00:22] vice=check_puppetrun [17:02:12] RECOVERY - check_puppetrun on frdb1003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1003&service=check_puppetrun [17:02:12] RECOVERY - check_puppetrun on frdb2003 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_puppetrun [17:02:14] RECOVERY - check_puppetrun on frbackup2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbackup2002&service=check_puppetrun [17:02:25] ^^those are all known from a puppet change and will be clearing. [17:03:18] RECOVERY - check_puppetrun on frdb1004 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_puppetrun [17:05:22] RECOVERY - check_puppetrun on frdb1005 is OK: OK: Puppet is currently enabled, last run 5 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1005&service=check_puppetrun [17:09:13] (03CR) 10Damilare Adedoyin: [C: 03+1] "Thanks Elliot for all the work on this." [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/913961 (https://phabricator.wikimedia.org/T260725) (owner: 10Ejegg) [17:12:52] 10Fundraising-Backlog, 10fundraising-tech-ops: Vanity URL for upcoming Annual Report mailing - https://phabricator.wikimedia.org/T335920 (10Dwisehaupt) @RLewis I'm assuming you meant 2022 not 2122. You can verify it is working now: https://fundraising.wikimedia.org/AnnualReport2022 [17:13:41] (03CR) 10Damilare Adedoyin: [C: 03+2] Ingenico disentanglement: no GC inheritance [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/913961 (https://phabricator.wikimedia.org/T260725) (owner: 10Ejegg) [17:15:22] (03Merged) 10jenkins-bot: Ingenico disentanglement: no GC inheritance [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/913961 (https://phabricator.wikimedia.org/T260725) (owner: 10Ejegg) [17:19:38] 10Fundraising-Backlog: civicrm notification top blocked if toolbar visiable - https://phabricator.wikimedia.org/T335992 (10AnnWF) [17:19:51] 10Fundraising-Backlog: Take out rtbt from Ingenico Paymentment Method/Submethod - https://phabricator.wikimedia.org/T335993 (10Damilare) [17:20:53] 10Fundraising-Backlog, 10fundraising-tech-ops: Vanity URL for upcoming Annual Report mailing - https://phabricator.wikimedia.org/T335920 (10RLewis) thanks @Dwisehaupt this works great. [17:23:57] 10Fundraising-Backlog, 10fundraising-tech-ops: Vanity URL for upcoming Annual Report mailing - https://phabricator.wikimedia.org/T335920 (10Dwisehaupt) 05Open→03Resolved a:03Dwisehaupt Great. I'll close this request. Let us know if you need anything else. [17:25:27] (03CR) 10Jgleeson: [C: 03+2] "This works great! After figuring out how to recreate the original problem, I tested it with the 'ResizeObserver loop limit exceeded' error" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/914016 (https://phabricator.wikimedia.org/T332693) (owner: 10Ejegg) [17:25:39] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Recurring-Donations, 10fr-donorservices: Update RecurringQueueConsumer to handle donation upgrades - https://phabricator.wikimedia.org/T335827 (10Ejegg) [17:27:09] (03Merged) 10jenkins-bot: Add patterns to ignore in client-side error log [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/914016 (https://phabricator.wikimedia.org/T332693) (owner: 10Ejegg) [17:39:09] thanks jgleeson ! [17:39:23] and thanks damilare [17:40:29] np [18:14:01] 10fundraising-tech-ops, 10DC-Ops, 10SRE, 10ops-eqiad: Q1:rack/setup/install frbast1002, frmon1002, frpig1002 - https://phabricator.wikimedia.org/T319460 (10Dwisehaupt) [18:24:51] (03PS2) 10Wfan: Update trigger [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/915007 (https://phabricator.wikimedia.org/T332074) [18:28:04] (03PS3) 10Wfan: Update trigger [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/915007 (https://phabricator.wikimedia.org/T332074) [18:28:06] (03PS1) 10Wfan: Update the notification box position when toolbar been toggled [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/915816 (https://phabricator.wikimedia.org/T335992) [18:28:43] (03PS2) 10Wfan: Update the notification box position when toolbar been toggled [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/915816 (https://phabricator.wikimedia.org/T335992) [18:30:04] 10Fundraising Sprint Istanbool (not Constantidouble), 10Fundraising-Backlog, 10Patch-For-Review: civicrm notification top blocked if toolbar visiable - https://phabricator.wikimedia.org/T335992 (10AnnWF) a:03AnnWF [18:32:43] 10fundraising-tech-ops: reimage fundraising bastion (frbast) role servers to Bullseye - https://phabricator.wikimedia.org/T333246 (10Dwisehaupt) [18:34:53] 10fundraising-tech-ops: Q3:rack/setup/install frbast2002, frauth2002 - https://phabricator.wikimedia.org/T334505 (10Dwisehaupt) [18:59:45] 10fundraising-tech-ops, 10FR-Email: Update Acoustic Unsubscribe landing page domain to be served by a secure wikimedia.org domain - https://phabricator.wikimedia.org/T336000 (10EWilfong_WMF) [19:21:52] Just continuing some Asana cleanup - cstone are you still testing this? https://phabricator.wikimedia.org/T312877 [19:32:14] 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 10Recurring-Donations, 10fr-donorservices: Create recurring upgrade form - https://phabricator.wikimedia.org/T335795 (10Ejegg) [19:32:49] hmmm anilk i feel like there was something left to be done there [19:35:40] ah yeah, the audit files [19:36:00] did we look at the specific CSV mentioned in th email? [19:36:10] I can do that and respond [19:37:49] 10Fundraising-Backlog: Documentation: Update Payment Platform Providers - https://phabricator.wikimedia.org/T336003 (10AKanji-WMF) [19:45:47] Gut check - I think I'd like to close this: https://phabricator.wikimedia.org/T267946 - while the Amazon Recurring Epic is still open, this particular task is about specific donors, and I think Michael has resolved. I'll ping him and then close? [19:47:03] sounds good anilk [19:49:06] 10Fundraising-Backlog, 10FR-Amazon, 10Recurring-Donations, 10fr-donorservices: What to do about Amazon recurring donations - https://phabricator.wikimedia.org/T267946 (10AKanji-WMF) Hi @MBeat33 - are you okay if I close this? We'll still be reviewing the overarching issue at {T107391}... [19:54:54] 10Fundraising-Backlog, 10FR-Amazon, 10Recurring-Donations, 10fr-donorservices: What to do about Amazon recurring donations - https://phabricator.wikimedia.org/T267946 (10MBeat33) For sure, @AKanji-WMF fine to close [19:55:30] 10Fundraising-Backlog, 10FR-Amazon, 10Recurring-Donations, 10fr-donorservices: What to do about Amazon recurring donations - https://phabricator.wikimedia.org/T267946 (10AKanji-WMF) 05Open→03Resolved a:03AKanji-WMF [20:00:07] 10Fundraising Sprint Amazing grep, 10Fundraising Sprint Bridge over troubled Wifi, 10Fundraising-Backlog, 10FR-Japan, and 3 others: Japan Form Variations for Testing for Q3 - https://phabricator.wikimedia.org/T322793 (10AKanji-WMF) Given the Japan campaign is over, and this updated task: {T329807} - can we... [20:00:52] (03PS1) 10Wfan: Update Upgrader [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/915833 (https://phabricator.wikimedia.org/T332074) [20:05:51] 10Fundraising Sprint Amazing grep, 10Fundraising Sprint Bridge over troubled Wifi, 10Fundraising-Backlog, 10FR-Japan, and 3 others: Japan Form Variations for Testing for Q3 - https://phabricator.wikimedia.org/T322793 (10Ejegg) Sure, I think we were only leaving this open in case people wanted us to finish... [20:06:00] 10Fundraising Sprint Amazing grep, 10Fundraising Sprint Bridge over troubled Wifi, 10Fundraising-Backlog, 10FR-Japan, and 3 others: Japan Form Variations for Testing for Q3 - https://phabricator.wikimedia.org/T322793 (10Ejegg) 05Open→03Resolved [20:07:39] 🔔 [20:07:45] Yay to resolving! [20:12:14] 10Fundraising-Backlog, 10Recurring-Donations, 10Thank-You-Page, 10fr-donorservices: Create thank you pages for recurring upgrade - https://phabricator.wikimedia.org/T335881 (10Ejegg) [22:12:43] (03CR) 10Eileen: "Can we do this elsewhere? We update this code regularly & would have to keep repatching it if we merge this (we could put in our wmf-civic" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/915816 (https://phabricator.wikimedia.org/T335992) (owner: 10Wfan) [22:13:40] (03CR) 10Eileen: "or if we think it affects other users we can maybe upstream it - ie if it's not just our theme combo & the same is relevant to our dmaster" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/915816 (https://phabricator.wikimedia.org/T335992) (owner: 10Wfan) [22:21:39] 10Fundraising Sprint Istanbool (not Constantidouble), 10Fundraising-Backlog: Investigation: Scope Segmentation in Civi & Acoustic - https://phabricator.wikimedia.org/T335376 (10Eileenmcnaughton) I'll take another look at the spreadsheet but I just want to articulate that I'm hoping we can use triggers to updat... [22:27:59] (03PS1) 10Eileen: WIP Add donor_segment_id field [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/915854 [22:32:09] (03PS3) 10Wfan: Update the notification box position when toolbar been toggled [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/915816 (https://phabricator.wikimedia.org/T335992) [22:44:15] it's always a fun time debugging when you have to resort to spl_object_id... [23:10:02] (03PS3) 10Ejegg: WIP: Delete GlobalCollect [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/913977 (https://phabricator.wikimedia.org/T260725) [23:11:49] (03CR) 10CI reject: [V: 04-1] WIP: Delete GlobalCollect [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/913977 (https://phabricator.wikimedia.org/T260725) (owner: 10Ejegg) [23:12:03] (03PS4) 10Wfan: Update trigger [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/915007 (https://phabricator.wikimedia.org/T332074) [23:12:53] well, down to 7 errors and 12 failures, anyway. and getting the same results locally, which is nice.