[00:52:09] PROBLEM - check_disk on frmon1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4914 MB (70% inode=87%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 1940 MB (94% inode=99%): /home 12350 MB (99% inode=99%): /tmp 12350 MB (99% inode=99%): /var 25615 MB (96% inode=99%): /srv 87453 MB (10% inode=99%): /var/tmp 422 MB (99% inode=99%): /var/log 262051 MB [00:52:09] e=99%): /var/log/audit 422 MB (99% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon1002&service=check_disk [00:57:09] RECOVERY - check_disk on frmon1002 is OK: DISK OK - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4914 MB (70% inode=87%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 1940 MB (94% inode=99%): /home 12350 MB (99% inode=99%): /tmp 12350 MB (99% inode=99%): /var 25615 MB (96% inode=99%): /srv 487729 MB (60% inode=99%): /var/tmp 422 MB (99% inode=99%): /var/log 262035 MB (98% inode [00:57:09] ar/log/audit 422 MB (99% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frmon1002&service=check_disk [01:04:04] (03PS1) 10Eileen: Try upgrading phpunit higher... [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/946641 [01:13:54] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/946641 (owner: 10Eileen) [01:27:10] (03CR) 10Eileen: [C: 03+2] "Cool - I didn't know we have this set up!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/942043 (owner: 10Wfan) [01:28:09] looks like phpunit is passing right up to version 9.6 - https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/946641 - I think it's best to get on the latest that works (I hit a conflict on something else hence I tried it) [06:04:42] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_39) - 10https://gerrit.wikimedia.org/r/946720 (owner: 10L10n-bot) [12:49:51] (03CR) 10Damilare Adedoyin: [C: 03+2] "Thanks for making the changes, I understand both Paypal and Venmo providers share same logic, hence same tests, but we might have to move " [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/939796 (https://phabricator.wikimedia.org/T334430) (owner: 10Wfan) [12:50:25] (03Merged) 10jenkins-bot: Complete technical work to enable Venmo through Braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/939796 (https://phabricator.wikimedia.org/T334430) (owner: 10Wfan) [12:50:41] (03Merged) 10jenkins-bot: Add refund script for braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/942513 (https://phabricator.wikimedia.org/T342788) (owner: 10Wfan) [12:50:44] (03Merged) 10jenkins-bot: Unit test for audit parse venmo [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/942532 (https://phabricator.wikimedia.org/T340801) (owner: 10Wfan) [12:50:49] (03Merged) 10jenkins-bot: Unit Test for Braintree Refund [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/942718 (https://phabricator.wikimedia.org/T342788) (owner: 10Wfan) [13:39:03] 10fundraising-tech-ops, 10SRE, 10decommission-hardware, 10ops-codfw: decommission frbast2001.frack.codfw.wmnet - https://phabricator.wikimedia.org/T340156 (10Jhancock.wm) a:05Papaul→03Jhancock.wm [14:35:15] 10fundraising-tech-ops, 10SRE, 10decommission-hardware, 10ops-codfw: decommission frmon2001.frack.codfw.wmnet - https://phabricator.wikimedia.org/T342694 (10Jhancock.wm) [14:37:59] 10fundraising-tech-ops, 10SRE, 10decommission-hardware, 10ops-codfw: decommission frmon2001.frack.codfw.wmnet - https://phabricator.wikimedia.org/T342694 (10Jhancock.wm) @Papaul here are the ports on the switches Fasw-c8a: 16 Fasw-c8b: 16 mgmt: 18 [14:38:28] 10fundraising-tech-ops, 10SRE, 10decommission-hardware, 10ops-codfw: decommission frbast2001.frack.codfw.wmnet - https://phabricator.wikimedia.org/T340156 (10Jhancock.wm) [14:39:49] 10fundraising-tech-ops, 10SRE, 10decommission-hardware, 10ops-codfw: decommission frbast2001.frack.codfw.wmnet - https://phabricator.wikimedia.org/T340156 (10Jhancock.wm) @Papaul ports are as follows Fasw-c8a: 15 Fasw-c8b: 15 mgmt: 16 [15:08:46] (03CR) 10Jgleeson: "Yeah I agree with Dami. Can we split the Venmo and Paypal payment tests out into their own respective test suites please. thanks for all t" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/939796 (https://phabricator.wikimedia.org/T334430) (owner: 10Wfan) [15:20:39] 10Fundraising Sprint Owner of a Lonely Product, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10Patch-For-Review: Before x months not working in segmentation - https://phabricator.wikimedia.org/T342943 (10jgleeson) The SQL is making me dizzy 😆 [15:47:31] Thanks damilare and jgleeson, so here is the last piece for venmo technical work ticket : https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/946588 to add venmo as new instrument [15:57:40] np wfan, thanks for all the work on this! [16:11:16] PROBLEM - check_memory on fran1001 is CRITICAL: CRIT Memory 99% used. Largest process: python3 (1058796) = 93.5% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_memory [16:16:16] PROBLEM - check_memory on fran1001 is CRITICAL: CRIT Memory 99% used. Largest process: python3 (1058796) = 93.6% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_memory [16:21:16] RECOVERY - check_memory on fran1001 is OK: OK Memory 6% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran1001&service=check_memory [16:26:36] (03PS1) 10Damilare Adedoyin: WIP: Use Financial Type= "Recurring Gifts" for all first recurring contribution and "Recurring Gifts - Cash" for subsequent donations. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/946969 (https://phabricator.wikimedia.org/T341370) [16:27:28] 10Fundraising-Backlog: Disabling Gift Sources - https://phabricator.wikimedia.org/T343735 (10NNichols) UPDATE: We've decided that we would like to use **Online Gift** as the gift source for all gifts coming through a banner, email, website, etc. I've created the option in civi [16:29:27] 10Wikimedia-Fundraising-Banners, 10Wikipedia-Android-App-Backlog (Android Release - FY2023-24): India fundraising messages in Android app - https://phabricator.wikimedia.org/T335918 (10JTannerWMF) p:05Triage→03Medium [16:38:28] 10Fundraising-Backlog: Changing coding to gifts made on the Endowment website - https://phabricator.wikimedia.org/T343756 (10NNichols) [16:40:15] (03CR) 10CI reject: [V: 04-1] WIP: Use Financial Type= "Recurring Gifts" for all first recurring contribution and "Recurring Gifts - Cash" for subsequent donations. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/946969 (https://phabricator.wikimedia.org/T341370) (owner: 10Damilare Adedoyin) [16:47:51] 10Fundraising-Backlog: New Custom Fields - Donor Specified - https://phabricator.wikimedia.org/T343830 (10NNichols) [17:45:27] (03PS2) 10Damilare Adedoyin: WIP: Use Financial Type= "Recurring Gifts" for all first recurring contribution and "Recurring Gifts - Cash" for subsequent donations. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/946969 (https://phabricator.wikimedia.org/T341370) [18:01:45] (03CR) 10CI reject: [V: 04-1] WIP: Use Financial Type= "Recurring Gifts" for all first recurring contribution and "Recurring Gifts - Cash" for subsequent donations. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/946969 (https://phabricator.wikimedia.org/T341370) (owner: 10Damilare Adedoyin) [18:26:11] (03CR) 10Jgleeson: [C: 04-1] "I'm getting failures on the new tests. I think I need to update the triggers I have locally? Could you explain how pls" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/943648 (owner: 10Eileen) [18:29:50] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog, 10fundraising-tech-ops, 10Patch-For-Review, 10Python3-Porting: modernize DjangoBannerStats to python3 - https://phabricator.wikimedia.org/T301905 (10CodeReviewBot) jgleeson closed https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests... [18:36:52] 10Wikimedia-Fundraising-Banners, 10Wikipedia-iOS-App-Backlog, 10iOS Release FY2023-24: India fundraising messages in iOS app - https://phabricator.wikimedia.org/T335919 (10JTannerWMF) p:05Triage→03Medium [18:37:16] 10Wikimedia-Fundraising-Banners, 10Wikipedia-iOS-App-Backlog, 10iOS Release FY2023-24: India fundraising messages in iOS app - https://phabricator.wikimedia.org/T335919 (10JTannerWMF) p:05Medium→03High [19:32:20] (03PS3) 10Damilare Adedoyin: WIP: Use Financial Type= "Recurring Gifts" for all first recurring contribution and "Recurring Gifts - Cash" for subsequent donations. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/946969 (https://phabricator.wikimedia.org/T341370) [19:47:28] (03CR) 10CI reject: [V: 04-1] WIP: Use Financial Type= "Recurring Gifts" for all first recurring contribution and "Recurring Gifts - Cash" for subsequent donations. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/946969 (https://phabricator.wikimedia.org/T341370) (owner: 10Damilare Adedoyin) [20:08:31] (03CR) 10Jgleeson: "Following up with the optimization we talked about on tech-talk" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/946588 (https://phabricator.wikimedia.org/T334430) (owner: 10Wfan) [20:11:37] (03CR) 10Jgleeson: Add venmo as instrument (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/946588 (https://phabricator.wikimedia.org/T334430) (owner: 10Wfan) [20:15:14] (03PS4) 10Damilare Adedoyin: WIP: Use Financial Type= "Recurring Gifts" for all first recurring contribution and "Recurring Gifts - Cash" for subsequent donations. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/946969 (https://phabricator.wikimedia.org/T341370) [20:30:12] (03PS2) 10Wfan: Add venmo as instrument [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/946588 (https://phabricator.wikimedia.org/T334430) [20:31:19] (03PS5) 10Damilare Adedoyin: WIP: Use Financial Type= "Recurring Gifts" for all first recurring contribution and "Recurring Gifts - Cash" for subsequent donations. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/946969 (https://phabricator.wikimedia.org/T341370) [20:40:51] (03PS6) 10Damilare Adedoyin: Use Financial Type= "Recurring Gifts" for all first recurring contribution and "Recurring Gifts - Cash" for subsequent donations. [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/946969 (https://phabricator.wikimedia.org/T341370) [20:59:26] (03CR) 10Eileen: "Hmm - this doesn't change the triggers - so you must already have them in your master but have done a git pull & not got the new ones" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/943648 (owner: 10Eileen) [21:00:11] (03PS3) 10Wfan: Add venmo as instrument [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/946588 (https://phabricator.wikimedia.org/T334430) [21:04:34] If anyone wants to jump in on the segmentation discussion with Katie / Joseph it is here https://meet.google.com/qgj-qjpn-fzd?authuser=0 at 15 past (10 mins) [22:16:33] 10Fundraising-Backlog: Acoustic Click-tracking - SSL cert for links.email.wikimedia.org - https://phabricator.wikimedia.org/T318188 (10AKanji-WMF) Sounds like this has been resolved, after a convo with @JMando - @KHaggard can you please confirm? [23:08:14] PROBLEM - check_puppetrun on frnetmon1001 is CRITICAL: CRITICAL: Puppet has 6 failures. Last run 6 minutes ago with 6 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frnetmon1001&service=check_puppetrun [23:08:20] PROBLEM - check_puppetrun on frlog1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 6 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frlog1002&service=check_puppetrun [23:08:58] ^^ delayed alerts from the frpm reboot [23:09:13] may see some more be expect these to clear on their own in a sec. [23:11:24] PROBLEM - check_puppetrun on frauth1002 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 9 minutes ago with 10 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth1002&service=check_puppetrun [23:13:14] RECOVERY - check_puppetrun on frnetmon1001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frnetmon1001&service=check_puppetrun [23:13:20] PROBLEM - check_puppetrun on frlog1002 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 11 minutes ago with 1 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frlog1002&service=check_puppetrun [23:16:24] PROBLEM - check_puppetrun on frauth1002 is CRITICAL: CRITICAL: Puppet has 10 failures. Last run 14 minutes ago with 10 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth1002&service=check_puppetrun [23:18:20] RECOVERY - check_puppetrun on frlog1002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frlog1002&service=check_puppetrun [23:21:20] RECOVERY - check_puppetrun on frauth1002 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth1002&service=check_puppetrun [23:39:19] anilk: re the segmentation - we are into the long tail part of it. There are 4 parts to the long tail [23:39:19] 1) new requirements that might come in from stakeholders (finessing the criteria) [23:39:19] 2) updating acoustic - planned for around Jan [23:39:19] 3) adding additional fields for Joseph - if we do this this year we would probably want to start work on this at least one sprint before the next maintenance window, taking into account any feedback from Nora [23:39:19] 3) preparing for year end roll over. This doesn't actually need to be done before sometime next year [23:39:20] So in terms of sprints - there probably isn't anything on us at the moment but aspects of 1 could arise ad hoc & we need to make some prioritisation decisions around 3 - against other work. I'll probably wrap up the current phabs this sprint / push the one that is for year end roll over out of the sprint since I think we'd be better to focus on that around March next year