[01:22:10] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Paypal:1) 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [01:37:10] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [01:45:16] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1420 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [01:47:10] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Paypal:1) 1 [=1], check_endpoints_warning (Paypal:1) 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [01:50:10] PROBLEM - check_disk on frdev1001 is CRITICAL: DISK CRITICAL - free space: /dev 257922 MB (100% inode=99%): /run 47450 MB (91% inode=99%): / 20561 MB (38% inode=95%): /dev/shm 257941 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 257941 MB (100% inode=99%): /boot 152 MB (61% inode=99%): /srv 320373 MB (8% inode=99%): /run/user/615 51588 MB (100% inode=99%): /run/user/3609 51588 MB (100% inode=99%): /run/user/12807 5 [01:50:10] 100% inode=99%): /run/user/3649 51588 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_disk [01:50:16] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1720 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [01:52:10] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [01:55:10] RECOVERY - check_disk on frdev1001 is OK: DISK OK - free space: /dev 257922 MB (100% inode=99%): /run 47442 MB (91% inode=99%): / 20561 MB (38% inode=95%): /dev/shm 257941 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /sys/fs/cgroup 257941 MB (100% inode=99%): /boot 152 MB (61% inode=99%): /srv 438136 MB (11% inode=99%): /run/user/615 51588 MB (100% inode=99%): /run/user/3609 51588 MB (100% inode=99%): /run/user/12807 51588 MB (1 [01:55:10] =99%): /run/user/3649 51588 MB (100% inode=99%): /run/user/573 51588 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_disk [01:55:16] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2020 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [01:58:18] eileen: is there a reason to keep the contribution_tracking table in dev_drupal on frdev1001? [01:58:34] not saying we need to clear it now, but thinking if we needed to clear space where we might do so. [01:59:05] (the right answer is to migrate to frdev1002 and frdb1006) [02:00:16] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 2000605 Threads: 23 Questions: 88548238 Slow queries: 656265 Opens: 340622203 Flush tables: 1 Open tables: 200 Queries per second avg: 44.260 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [02:13:34] (03CR) 10Eileen: Add silverpop_export column for upgrade activities (031 comment) [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/919256 (https://phabricator.wikimedia.org/T335824) (owner: 10Ejegg) [02:24:18] 10Fundraising-Backlog: Error upon event import (Stripe) - https://phabricator.wikimedia.org/T336895 (10JSimmons) @Eileenmcnaughton I found a small issue with a few gifts that were included on the most recent Stripe import. There are two entries under the payment details section. The first line is the correct gif... [02:31:05] (03CR) 10Eileen: "This is a better subquery - the INNER join" [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/919256 (https://phabricator.wikimedia.org/T335824) (owner: 10Ejegg) [02:55:24] (03PS5) 10Ejegg: Add silverpop_export column for upgrade activities [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/919256 (https://phabricator.wikimedia.org/T335824) [02:55:26] (03PS4) 10Ejegg: Add eligibility for recurring upgrade to export [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/920360 (https://phabricator.wikimedia.org/T335824) [02:55:31] (03CR) 10CI reject: [V: 04-1] Add eligibility for recurring upgrade to export [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/920360 (https://phabricator.wikimedia.org/T335824) (owner: 10Ejegg) [02:57:01] (03CR) 10Ejegg: "Thanks for the review, and thanks for suggesting that really nice count(*) > 0 subquery!" [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/919256 (https://phabricator.wikimedia.org/T335824) (owner: 10Ejegg) [02:58:01] (03PS6) 10Ejegg: Add silverpop_export column for upgrade activities [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/919256 (https://phabricator.wikimedia.org/T335824) [02:58:02] (03PS5) 10Ejegg: Add eligibility for recurring upgrade to export [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/920360 (https://phabricator.wikimedia.org/T335824) [02:58:07] (03CR) 10CI reject: [V: 04-1] Add eligibility for recurring upgrade to export [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/920360 (https://phabricator.wikimedia.org/T335824) (owner: 10Ejegg) [02:58:40] ^^ Still the spurious V-1 telling me to rebase a patch that's right on top of its parent [02:58:59] oh derp I used the wrong one [02:59:48] (03PS7) 10Ejegg: Add silverpop_export column for upgrade activities [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/919256 (https://phabricator.wikimedia.org/T335824) [02:59:50] (03PS6) 10Ejegg: Add eligibility for recurring upgrade to export [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/920360 (https://phabricator.wikimedia.org/T335824) [02:59:55] (03CR) 10CI reject: [V: 04-1] Add eligibility for recurring upgrade to export [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/920360 (https://phabricator.wikimedia.org/T335824) (owner: 10Ejegg) [03:00:02] ok, PS7 has the INNER JOIN [03:00:28] gotta run, thanks again eileen ! [03:06:28] (03CR) 10Eileen: [C: 03+2] "This seems right now - exactly zero contacts have this activity type id so far so it didn't find any..." [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/919256 (https://phabricator.wikimedia.org/T335824) (owner: 10Ejegg) [03:10:51] (03Merged) 10jenkins-bot: Add silverpop_export column for upgrade activities [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/919256 (https://phabricator.wikimedia.org/T335824) (owner: 10Ejegg) [03:14:54] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/920808 [03:15:19] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/920808 (owner: 10Eileen) [03:16:29] (03Merged) 10jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/920808 (owner: 10Eileen) [04:42:10] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Paypal:5) 5 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [04:47:10] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Paypal:4) 4 [=1], paypal_transaction_failure 15 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [04:52:10] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Paypal:3) 3 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [04:57:10] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Paypal:2) 2 [=1], check_endpoints_warning (Paypal:1) 1 [=1], paypal_transaction_failure 12 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [05:02:10] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Paypal:4) 4 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [05:07:10] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Paypal:3) 3 [=1], check_endpoints_warning (Paypal:1) 1 [=1] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [05:12:10] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [05:42:10] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Paypal:5) 5 [=1], check_endpoints_warning (Paypal:1) 1 [=1], paypal_transaction_failure 18 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [05:47:10] PROBLEM - check_log_messages on frav1002 is CRITICAL: CRITICAL: check_endpoints_critical (Paypal:3) 3 [=1], paypal_transaction_failure 12 [=10] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [05:52:10] RECOVERY - check_log_messages on frav1002 is OK: OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frav1002&service=check_log_messages [06:04:24] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_40) - 10https://gerrit.wikimedia.org/r/920855 (owner: 10L10n-bot) [08:14:26] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/921001 (owner: 10L10n-bot) [08:44:20] (03CR) 10Raimond Spekking: [C: 03+2] "false positive" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/921001 (owner: 10L10n-bot) [19:22:18] 10Fundraising-Backlog, 10WMF-General-or-Unknown: Replace deprecated Hooks::runWithoutAbort in CentralNotice wmf production code - https://phabricator.wikimedia.org/T336967 (10Umherirrender) [19:22:34] 10Fundraising-Backlog, 10WMF-General-or-Unknown: Replace deprecated Hooks::runWithoutAbort in CentralNotice wmf production code - https://phabricator.wikimedia.org/T336967 (10Umherirrender) [20:03:24] (03PS1) 10Eileen: Update endowment text to live [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/921092 [20:07:18] PROBLEM - check_nginx on frdata2001 is CRITICAL: PROCS CRITICAL: 0 processes with command name nginx https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_nginx [20:07:20] PROBLEM - check_puppetrun on frdata2001 is CRITICAL: CRITICAL: Puppet has 1 failures. Last run 5 minutes ago with 1 failures. Failed resources (up to 3 shown): Service[nginx] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_puppetrun [20:07:22] PROBLEM - check_ssl on frdata2001 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_ssl [20:07:24] PROBLEM - check_ssl_fr on frdata2001 is CRITICAL: SSL CRITICAL - failed to connect or SSL handshake:Connection refused https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_ssl_fr [20:11:55] ^^ those are fixed. needed to sync a cert across that hadn't happened yet. [20:12:20] RECOVERY - check_nginx on frdata2001 is OK: PROCS OK: 49 processes with command name nginx https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_nginx [20:12:20] RECOVERY - check_puppetrun on frdata2001 is OK: OK: Puppet is currently enabled, last run 2 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_puppetrun [20:12:22] RECOVERY - check_ssl on frdata2001 is OK: SSL OK - Certificate fundraising.wikimedia.org valid until 2023-08-09 23:55:16 +0000 (expires in 83 days) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_ssl [20:12:24] RECOVERY - check_ssl_fr on frdata2001 is OK: SSL OK - Certificate frdata.wikimedia.org valid until 2023-07-18 12:01:27 +0000 (expires in 60 days) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_ssl_fr [20:24:59] 10Fundraising-Backlog, 10fundraising-tech-ops, 10fr-donorservices: Expose email-prefs-wiki to public - https://phabricator.wikimedia.org/T335793 (10Dwisehaupt) OK. I have updated the LE certificate config and all the apache and nginx configs. The new setup is in place on https://donorpreferences.wikimedia.or... [20:47:06] 10Fundraising Sprint Jealousy Driven Development, 10Fundraising-Backlog, 10Patch-For-Review, 10Unplanned-Sprint-Work: Individual matching import issue - https://phabricator.wikimedia.org/T336609 (10MDemosWMF) Hi @Eileenmcnaughton - just tried another Individual import and only the first 5 rows imported out... [20:52:35] 10Fundraising Sprint Jealousy Driven Development, 10Fundraising-Backlog, 10Patch-For-Review, 10Unplanned-Sprint-Work: Individual matching import issue - https://phabricator.wikimedia.org/T336609 (10MDemosWMF) Update - I tried again and it imported the rest except for row 7 and row 36. It might have gotten... [21:48:52] 10Fundraising-Backlog, 10fundraising-tech-ops, 10fr-donorservices: Expose email-prefs-wiki to public - https://phabricator.wikimedia.org/T335793 (10Dwisehaupt) [22:07:44] 10Fundraising-Backlog, 10fundraising-tech-ops, 10fr-donorservices: Expose email-prefs-wiki to public - https://phabricator.wikimedia.org/T335793 (10Dwisehaupt)