[00:00:16] PROBLEM - check_payments_wiki on payments1007 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1007.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 317 bytes in 0.016 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_payments_wiki [00:01:16] PROBLEM - check_payments_wiki on payments1006 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1006.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 317 bytes in 0.019 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_payments_wiki [00:01:16] PROBLEM - check_payments_wiki on payments2002 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2002.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 317 bytes in 0.016 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2002&service=check_payments_wiki [00:02:26] PROBLEM - check_payments_wiki on payments1005 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1005.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 317 bytes in 0.018 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_payments_wiki [00:02:55] ACKNOWLEDGEMENT - check_payments_wiki on payments1005 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1005.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 317 bytes in 0.018 second response time Dwisehaupt known https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_payments_wiki [00:02:55] ACKNOWLEDGEMENT - check_payments_wiki on payments1006 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1006.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 317 bytes in 0.019 second response time Dwisehaupt known https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_payments_wiki [00:02:55] ACKNOWLEDGEMENT - check_payments_wiki on payments1007 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments1007.frack.eqiad.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 317 bytes in 0.016 second response time Dwisehaupt known https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_payments_wiki [00:02:56] ACKNOWLEDGEMENT - check_payments_wiki on payments2001 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2001.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 317 bytes in 0.015 second response time Dwisehaupt known https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_payments_wiki [00:02:56] ACKNOWLEDGEMENT - check_payments_wiki on payments2002 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2002.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 317 bytes in 0.016 second response time Dwisehaupt known https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2002&service=check_payments_wiki [00:02:56] ACKNOWLEDGEMENT - check_payments_wiki on payments2003 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string OK not found on https://payments2003.frack.codfw.wmnet:443https://payments.wikimedia.org/index.php/Special:SystemStatus - 317 bytes in 0.015 second response time Dwisehaupt known https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_payments_wiki [00:15:16] RECOVERY - check_payments_wiki on payments1007 is OK: HTTP OK: HTTP/1.1 200 OK - 529 bytes in 0.043 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_payments_wiki [00:16:16] RECOVERY - check_payments_wiki on payments1006 is OK: HTTP OK: HTTP/1.1 200 OK - 529 bytes in 0.042 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1006&service=check_payments_wiki [00:16:16] RECOVERY - check_payments_wiki on payments2002 is OK: HTTP OK: HTTP/1.1 200 OK - 529 bytes in 0.040 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2002&service=check_payments_wiki [00:17:16] RECOVERY - check_payments_wiki on payments1005 is OK: HTTP OK: HTTP/1.1 200 OK - 529 bytes in 0.052 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1005&service=check_payments_wiki [00:18:16] RECOVERY - check_payments_wiki on payments2003 is OK: HTTP OK: HTTP/1.1 200 OK - 529 bytes in 0.037 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_payments_wiki [00:18:16] RECOVERY - check_payments_wiki on payments2001 is OK: HTTP OK: HTTP/1.1 200 OK - 529 bytes in 0.040 second response time https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_payments_wiki [01:23:09] 10Fundraising-Backlog: Enable Venmo on live - https://phabricator.wikimedia.org/T344287 (10AnnWF) 1: update the config to new braintree account (updated and deploy localsettings/SmashPig/local-config/braintree/main.yaml on frpm1002, while still go to old account, see listner/SmashPig/local-config/braintree/main.... [02:13:21] (03PS1) 10Eileen: Redis fix - only remove contacts from sub-keys if they are in the main key [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/950060 (https://phabricator.wikimedia.org/T247347) [02:14:14] 10Fundraising-Backlog, 10fr-donorservices: Permission changed for DR agents - https://phabricator.wikimedia.org/T343899 (10Eileenmcnaughton) {F37558981} [02:14:17] (03CR) 10CI reject: [V: 04-1] Redis fix - only remove contacts from sub-keys if they are in the main key [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/950060 (https://phabricator.wikimedia.org/T247347) (owner: 10Eileen) [02:16:43] 10Fundraising-Backlog: Enable Venmo on live - https://phabricator.wikimedia.org/T344287 (10AnnWF) [02:20:27] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/950060 (https://phabricator.wikimedia.org/T247347) (owner: 10Eileen) [02:52:05] (03CR) 10Eileen: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/950060 (https://phabricator.wikimedia.org/T247347) (owner: 10Eileen) [03:10:32] (03PS4) 10Wfan: Check each recurring if have cancel date before charge [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/910615 (https://phabricator.wikimedia.org/T335152) [03:11:33] 10Fundraising Tech - Chaos Crew: Apple Pay Payment Processing Certificate Expires September 12, 2023 - https://phabricator.wikimedia.org/T344288 (10AnnWF) a:03AnnWF [03:16:52] wfan: buildkit should be fixed now [03:25:08] (03CR) 10CI reject: [V: 04-1] Check each recurring if have cancel date before charge [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/910615 (https://phabricator.wikimedia.org/T335152) (owner: 10Wfan) [03:29:58] 10Fundraising Sprint CAPS LOCK CULTS, 10Fundraising Sprint Dampness, 10Fundraising Sprint 🐲 Phab the Magic Dragon, 10Fundraising-Backlog, and 3 others: Delete more deleted contacts? - https://phabricator.wikimedia.org/T247347 (10Eileenmcnaughton) Summary of situation (emailed to fr-tech & copied here) Hi... [09:51:08] 10Fundraising-Backlog, 10MediaWiki-extensions-DonationInterface, 10Upstream: Accept GNU Taler for donations - https://phabricator.wikimedia.org/T157529 (10Nemo_bis) I've just talked with [Grothoff](https://grothoff.org/christian/) at Chaos Communication Camp about it. Apparently payments will start being pos... [09:52:08] I've updated the task on Taler support https://phabricator.wikimedia.org/T157529#9101358 [12:45:17] 10Fundraising-Backlog, 10fundraising-tech-ops: Streamline Fundraising datacenter failover procedure - https://phabricator.wikimedia.org/T344380 (10Jgreen) [13:33:49] 10Fundraising-Backlog, 10fundraising-tech-ops: Streamline Fundraising datacenter failover procedure - https://phabricator.wikimedia.org/T344380 (10Jgreen) [14:17:14] 10Fundraising-Backlog, 10FR-Docker: Docker dev setup: try out and set up stuff for silverpop export - https://phabricator.wikimedia.org/T289526 (10jgleeson) 05Open→03Resolved [14:19:05] 10Fundraising Sprint Corrugated super slide, 10Fundraising-Backlog, 10FR-Docker: Docker dev setup: Set up queue consumers - https://phabricator.wikimedia.org/T268686 (10jgleeson) 05Open→03Resolved [14:19:07] 10Fundraising-Backlog, 10FR-Docker, 10Epic: Epic: feature parity with vagrant on docker - https://phabricator.wikimedia.org/T262971 (10jgleeson) [14:19:36] 10Fundraising Sprint Yellow hornets of kindness and healing, 10Fundraising Sprint Zeitgeistbusters, 10Fundraising-Backlog, 10FR-Docker: Docker dev setup: Fundraising tools > silverpop export - https://phabricator.wikimedia.org/T269708 (10jgleeson) 05Open→03Resolved [14:19:38] 10Fundraising-Backlog, 10FR-Docker, 10Epic: Epic: feature parity with vagrant on docker - https://phabricator.wikimedia.org/T262971 (10jgleeson) [14:20:10] 10Fundraising-Backlog, 10FR-Docker: Investigate requirements and setup options for self-contained, quick-and-simple-to-run FR applications on Docker - https://phabricator.wikimedia.org/T274341 (10jgleeson) 05Open→03Resolved [14:20:30] 10Fundraising-Backlog, 10FR-Docker: Update main page wiki text - https://phabricator.wikimedia.org/T274841 (10jgleeson) 05Open→03Resolved [14:20:32] 10Fundraising-Backlog, 10FR-Docker, 10Epic: Epic: feature parity with vagrant on docker - https://phabricator.wikimedia.org/T262971 (10jgleeson) [14:21:33] 10Fundraising-Backlog, 10FR-Docker: Fix Docker performance on Mac - https://phabricator.wikimedia.org/T274339 (10jgleeson) a:03jgleeson [14:23:17] 10Fundraising-Backlog, 10FR-Docker: Docker dev setup: replace setup.sh with something better - https://phabricator.wikimedia.org/T278728 (10jgleeson) a:03jgleeson [16:30:41] 10Fundraising-Backlog, 10fundraising-tech-ops: Streamline Fundraising datacenter failover procedure - https://phabricator.wikimedia.org/T344380 (10Jgreen) [16:36:40] 10Fundraising Sprint 🐲 Phab the Magic Dragon, 10Fundraising-Backlog, 10Wikipedia-iOS-App-Backlog, 10iOS Release FY2023-24: As a donor I want to be able to make an easy and convenient donation via Apple Pay in the Wikipedia app - https://phabricator.wikimedia.org/T288285 (10JTannerWMF) [17:02:10] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Adyen SEPA Direct debit unsettled transactions June - August 2023 - https://phabricator.wikimedia.org/T344203 (10EMartin) Adyen has reverted to auto-settling SEPA on their end going forward. It seems my request to add direct debit in France in June was s... [17:04:47] ^ adyen issue resolved fr-tech [17:04:54] Christine will be happy! [17:22:03] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Adyen SEPA Direct debit unsettled transactions June - August 2023 - https://phabricator.wikimedia.org/T344203 (10EMartin) 05Open→03Resolved [18:05:10] PROBLEM - Host pay-lvs1003 is DOWN: PING CRITICAL - Packet loss = 100% [18:06:38] ^^ that's the reboot for openssl upgrade. [18:07:14] RECOVERY - Host pay-lvs1003 is UP: PING OK - Packet loss = 0%, RTA = 0.42 ms [19:25:00] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising-tech-ops: Install `coworker` and `cv` on prod - https://phabricator.wikimedia.org/T324611 (10Dwisehaupt) [19:27:21] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fundraising-tech-ops: Install `coworker` and `cv` on prod - https://phabricator.wikimedia.org/T324611 (10Dwisehaupt) An update to the systemd::service manifest was rolled out to support maintenance mode. This cleared the blocker for coworker and it ha... [19:36:15] 10Fundraising-Backlog, 10fundraising-tech-ops: Streamline Fundraising datacenter failover procedure - https://phabricator.wikimedia.org/T344380 (10Jgreen) [20:25:47] 10Fundraising-Backlog, 10fundraising-tech-ops: Set up Community CiviCRM instance. - https://phabricator.wikimedia.org/T343486 (10Dwisehaupt) @Qgil Thanks for that information. I'll look into the CloudVPS request process so that we can get started independently. There will probably be follow on questions from us. [21:17:10] 10Wikimedia-Fundraising-Banners, 10Wikipedia-iOS-App-Backlog: Netherlands fundraising messages for iOS app - https://phabricator.wikimedia.org/T344516 (10spatton) [21:20:03] 10Wikimedia-Fundraising-Banners, 10Wikipedia-iOS-App-Backlog: Netherlands fundraising messages for Android app - https://phabricator.wikimedia.org/T344519 (10spatton) [21:20:55] 10Wikimedia-Fundraising-Banners, 10Wikipedia-Android-App-Backlog: Netherlands fundraising messages for Android app - https://phabricator.wikimedia.org/T344519 (10spatton) [22:07:12] PROBLEM - check_puppetrun on frbackup2002 is CRITICAL: CRITICAL: Puppet has 21 failures. Last run 5 minutes ago with 21 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbackup2002&service=check_puppetrun [22:08:10] ^^^ that's expected since i rebooted frpm2001 [22:10:12] PROBLEM - check_puppetrun on frlog2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 8 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frlog2001&service=check_puppetrun [22:11:16] same there. [22:11:18] PROBLEM - check_puppetrun on frauth2002 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 9 minutes ago with 13 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2002&service=check_puppetrun [22:12:12] PROBLEM - check_puppetrun on frbackup2002 is CRITICAL: CRITICAL: Puppet has 21 failures. Last run 10 minutes ago with 21 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbackup2002&service=check_puppetrun [22:15:12] PROBLEM - check_puppetrun on frlog2001 is CRITICAL: CRITICAL: Puppet has 30 failures. Last run 13 minutes ago with 30 failures. Failed resources (up to 3 shown): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frlog2001&service=check_puppetrun [22:16:18] PROBLEM - check_puppetrun on frauth2002 is CRITICAL: CRITICAL: Puppet has 13 failures. Last run 14 minutes ago with 13 failures. Failed resources (up to 3 shown): File[/usr/local/bin/package_update_check],File[/etc/vim/vimrc.local],File[/usr/local/bin/yubikey_otp_filter],File[/bin/nopeshell] https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2002&service=check_puppetrun [22:17:12] RECOVERY - check_puppetrun on frbackup2002 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frbackup2002&service=check_puppetrun [22:19:38] PROBLEM - Host frbast2002 is DOWN: PING CRITICAL - Packet loss = 100% [22:20:12] RECOVERY - check_puppetrun on frlog2001 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frlog2001&service=check_puppetrun [22:21:18] RECOVERY - check_puppetrun on frauth2002 is OK: OK: Puppet is currently enabled, last run 4 minutes ago with 0 failures https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frauth2002&service=check_puppetrun [22:23:10] RECOVERY - Host frbast2002 is UP: PING OK - Packet loss = 0%, RTA = 30.29 ms [22:46:07] ACKNOWLEDGEMENT - Host frqueue2001 is DOWN: PING CRITICAL - Packet loss = 100% Dwisehaupt kernel upgrade [22:46:07] ACKNOWLEDGEMENT - Host frqueue2002 is DOWN: PING CRITICAL - Packet loss = 100% Dwisehaupt kernel upgrade