[00:13:59] (03PS1) 10Eileen: CiviCRM submodule update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809726 [00:14:13] (03CR) 10Eileen: [C: 03+2] CiviCRM submodule update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809726 (owner: 10Eileen) [00:16:20] this is the delete global collect that seems agreed https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/809294 [00:23:03] oh sure, let's do that too [00:23:09] (03PS2) 10Ejegg: Remove recurring_global_collect [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809294 (owner: 10Eileen) [00:23:15] (03CR) 10Ejegg: [C: 03+2] Remove recurring_global_collect [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809294 (owner: 10Eileen) [00:23:43] There might be some things in the shared recurring.inc that are now unused, but I didn't track any down just yet [00:27:23] (03Merged) 10jenkins-bot: CiviCRM submodule update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809726 (owner: 10Eileen) [00:29:24] no I didn't spot them [00:29:49] Here is an etherpad for the outage (copy & paste - not sure if any tweaks needed) https://etherpad.wikimedia.org/p/outage-2022-Jun-Civi-upgrade [00:30:09] thanks! [00:31:26] I don't think we will need to turn campaigns off? My understanding is they are off [00:36:42] (03Merged) 10jenkins-bot: Remove recurring_global_collect [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809294 (owner: 10Eileen) [00:38:25] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into HEAD [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/809731 [00:38:30] ejegg: quite a lot in that merge-commit [00:40:07] (03PS1) 10Eileen: Revert "Remove engage role for now" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809733 [00:40:36] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into HEAD [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/809731 (owner: 10Eileen) [00:42:27] howdy. [00:44:54] i'll prep a diff to disable the p-c jobs. [00:45:51] (03PS1) 10Eileen: Remove flush, move geocoders to post action [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809734 [00:47:33] (03CR) 10Eileen: [C: 03+2] "recheck" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/809731 (owner: 10Eileen) [00:48:46] (03CR) 10CI reject: [V: 04-1] Remove flush, move geocoders to post action [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809734 (owner: 10Eileen) [00:51:50] hmmm... we don't need to take down payments for this upgrade, correct? since any donations that may come in will be queued but not processed. [00:52:33] dwisehaupt: right, as far as I can tell neither payments nor campaigns [00:53:45] cool. i'm prepping that diff too. [00:54:54] ok - the code is merged [01:00:04] ok. pushing up the change for maint mode. i'll hand roll the puppet run on civi. [01:00:20] (03PS2) 10Eileen: Remove flush, move geocoders to post action [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809734 [01:03:40] (03CR) 10CI reject: [V: 04-1] Remove flush, move geocoders to post action [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809734 (owner: 10Eileen) [01:05:38] (03PS3) 10Eileen: Remove flush, move geocoders to post action [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809734 [01:06:13] ok. got thwarted by some odd stray hidden character in my shell but p-c should now be down. [01:06:21] lol [01:06:41] ok - so we should get these sql running as quickly as possible - how do you want to do it [01:07:10] fire them up one shell per db? [01:07:26] one shell per sql you mean? [01:07:40] sorry, one shell per table. [01:08:04] since there were some that did an update and then an alter on the same table. [01:08:30] like lines 29 and 31 on your etherpad from earlier. [01:08:44] yep - sounds good [01:08:52] are you gonna fire them all off? [01:08:59] (03CR) 10CI reject: [V: 04-1] Remove flush, move geocoders to post action [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809734 (owner: 10Eileen) [01:09:01] i can do that if you would like. [01:09:07] cool - [01:09:13] ok. i'll get them going. [01:10:07] and just to verify, all of these tables are in the civicrm db. not the drupal db. yes? [01:10:18] yep [01:10:25] coolthx [01:10:59] (03CR) 10Ejegg: [C: 03+2] Revert "Remove engage role for now" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809733 (owner: 10Eileen) [01:14:53] ok. they are all started or queued. [01:15:26] the query on 29 finished quickly, so i went ahead and queued up it's follow on at 31. [01:15:51] as a safety gap, replication is stopped on frdb2002 while these finish up. [01:19:05] eileen: the query on line 31 didn't complete. it exited with the warning: failed to load FOREIGN KEY constraints [01:19:21] hmm ok - [01:19:42] I'll look but you can do the others [01:20:41] oh weird [01:20:50] hmm that query is one I did after it was slow on frdev - there might be an error in it as I didn't actually run that one on prod [01:21:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: Slave IO: No Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [01:21:16] ok. the others are rolling. i'm marking them as done in the etherpad as the complete. [01:22:30] I really can't see what is wrong with that one [01:22:33] i've put in the downtime for that alert there. [01:23:34] (03Merged) 10jenkins-bot: Revert "Remove engage role for now" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809733 (owner: 10Eileen) [01:24:45] I'll try the email on direct on prod if you don't have it running dwisehaupt - but I'll just to one field [01:25:25] similar warning for the query on line 25. [01:25:47] eileen: sure. go ahead. [01:26:11] for reference. i have a screen session with a window per query and letting them run until they exit. [01:26:23] dwisehaupt: looks like it had passed despite the error - when I re-ran it was instant [01:28:43] the only fk on email is contact_id - so I'm querying to see if any rows are in breach of that [01:28:44] select * FROM civicrm_email LEFT JOIN civicrm_contact c ON c.id = contact_id WHERE c.id IS NULL; [01:29:11] yeah, warnings don't necessarily mean failure. i'm happy with it if you are. just want to be certain. [01:29:30] nope [01:29:37] no rows in breach [01:33:14] ok. down to 2 queries running. [01:33:28] i just started the one on line 11. [01:34:37] !log civicrm upgraded from 3cb5e6dd to f48fe112 [01:34:41] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:34:46] I just pushed out the code [01:35:12] these are running faster than they did on staging :-) [01:35:21] also running in paralell is MUCH quicker [01:35:33] yeah. also much newer hardware. :) [01:35:38] nice [01:35:49] it good a good three hours sequentially on staging [01:36:27] the upgrade script still does the same to a lot of tables - but they are smaller and none leapt out as slow [01:41:41] ok. so civicrm_activity is about 170G so this alter will take a bit. [01:41:59] ok - it took maybe forty mins on staging [01:42:08] yeah. that sounds about what i was thinking. [01:42:18] i'm gonna grab some more water and stretch. [01:42:24] cool [01:42:35] brb [01:50:13] PROBLEM - check_mysql on frdev1001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1280 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [02:07:25] eileen: ok. the last query just finished and all good. [02:07:42] nice [02:07:44] ok - I'll start the upgrade script [02:19:47] I hit an error on the extension upgrade but it still seems to have worked [02:20:08] log table sql timeout? [02:24:39] I pasted the error at tthe bottom of the etherpad - trying to access a table before it existed [02:24:57] I'm just diffing the triggers now [02:27:25] (03PS1) 10Eileen: CiviCRM 5.51 upgrade - triggers update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809738 [02:27:34] ejegg: ^^ triggers update [02:27:34] looking [02:32:12] huh, getting some unknown columns [02:32:43] i'm gonna go grab a plate of dinner. back in a few. [02:33:16] 'note' on line 1120 and 1288, event_tz_bak on 1931 1933 1935, no table civicrm_grant on line 1979 + 2 more [02:33:29] also table civicrm.civicrm_oauth_systoken [02:33:43] and a couple custom fields [02:33:53] right, those are just prod I guess [02:35:10] yeah that seems OK [02:35:13] ok, all those tables do seem to exist on prod [02:35:44] anyway, syntax is fine [02:36:01] (03CR) 10Ejegg: [C: 03+2] CiviCRM 5.51 upgrade - triggers update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809738 (owner: 10Eileen) [02:41:52] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/809739 [02:42:04] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/809739 (owner: 10Eileen) [02:44:57] back [02:49:23] just let me know when the triggers are deployed and i'll apply them. [02:57:15] dwisehaupt: it's deployed [02:57:25] cool. rolling them out. [02:58:01] !log civicrm upgraded from f48fe112 to 9e5a5310 [02:58:04] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [02:59:04] just the triggers.mysql file, correct? [03:03:58] yep & the new trigger will be on civicrm_search_display [03:04:47] cool. just working through the diff now. [03:07:54] hmmm.. i don't see one on civicrm_search_display in the diff. pasting the ones i see in the etherpad [03:09:34] sorry search_segment [03:09:35] https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/809739/-1..1/drupal/sites/all/modules/wmf_civicrm/scripts/triggers.mysql [03:09:55] ah yes. those are there. [03:09:58] all good then. [03:12:20] sorry, for some reason i was having issues navigating gerrit to get that view. thanks for the link. [03:14:53] we can start bringing it back up then [03:17:08] I ran a slow-start donation & that worked [03:19:19] I like the way our etherpad has us testing the UI before the UI is re-enabled :-) [03:19:30] ok. i'll start rolling back the downtime first. [03:20:19] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 208639 Threads: 13 Questions: 3689888 Slow queries: 225021 Opens: 35641505 Flush tables: 1 Open tables: 200 Queries per second avg: 17.685 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4490 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [03:21:36] ok. civi is out of maint mode but the p-c jobs are still disabled. feel free to test. [03:25:17] when you give me the all clear i can restart the queues and then it'll just be a matter of keeping an eye on the replication lag catching up. [03:26:18] go for it - there is an issue on the mailing screen - but I don't think that stops us going live [03:27:48] eileen: i'm getting a 500 error trying to preview the EOY message [03:27:58] yep - me too [03:28:11] not sure why - I just tested sk & that works OK [03:28:13] oh that's the error you just mentioned to dwisehaupt ? [03:28:20] yeah [03:28:29] I even created our first data segment - https://civicrm.wikimedia.org/civicrm/admin/search#/list?tab=segment [03:28:32] ok. i'll fire up the queues then. [03:28:37] yep [03:28:48] let's get it back up & then deal with the preview [03:30:08] heh: Enter 'ego' to proceed [03:30:46] ok. p-c jobs are re-enabled. i'll restart replication on frdb2002 [03:36:24] oh that is a weird error to pop up now [03:36:25] HP Fatal error: Declaration of CRM_Contact_Import_MetadataTrait::getHeaderPatterns() must be compatible with CRM_Import_Parser::getHeaderPatterns(): array in /srv/org.wikimedia.civicrm/civicrm/CRM/Contact/Import/Parser/Contact.php on line 27 [03:36:34] hey eileen dwisehaupt how's stuff going? I just got back from evening kid transport, am available if I can do anything [03:36:53] hmm [03:37:02] AndyRussG: we are bring it back up but some unexpected error confusing me [03:37:25] hmmm in what context? [03:38:17] If I update everything to master and rebuild will I have closest possible local environment? [03:39:48] hmm - so I had an error that is above. I just disabled the targetsmart import extension & the error disappeared - that makes sense because a) we only use that once a year at most and b) it is not enabled in our dev and c) it will need updating before we use it again [03:40:10] but now previews are not found - I'm gonna try another round of cache refreshes [03:40:58] but it's also OK for them to be broken overnight [03:44:11] if frdb2002 follows frdb2001, the replication should clear there in about 2 hours. [03:46:12] hmmm oki [03:46:42] so the ExampleData::get call is returning nothing even with just the name like workflow/endowment_thank_you/% clause [03:47:21] i guess the actual thank yous are going out fine though [03:47:30] yep they will be [03:47:36] i'm going to step away for a bit to help with dogbath. back in a bit to check in. [03:47:38] yeah - it's broken locally too now - I didn't test after that last re-apply of stuff [03:50:49] ejegg: I think I can notify people it's up now though - it's just that one thing [03:51:48] ahh shoot, I didn't test that one with the rest of the civicrm patches either [03:52:09] ok, sounds fine to fix tomorrow then [03:52:28] yeah it was working recently - but before I got the tests working [03:52:53] ok, I should head to bed. Thanks, eileen and dwisehaupt ! [03:59:28] hmm - still working on staging [04:05:56] oh weird, the ingenico audit parser failed with 'Class 'Civi\Api4\Service\Spec\Provider\CustomGroupCreationSpecProvider' not found in CachedCiviContainer->getSpecGathererService()' [04:06:08] why does it need that? [04:06:26] I guess that can be for tomorrow too [04:12:55] hmmm [04:21:00] replication still churning away and everything else looking ok on our end. i've removed the downtime portions in icinga for all the hosts that have recovered. [04:24:45] ug - I don't like that one much [04:28:33] (03PS1) 10Eileen: Revert "Temporary hack to wrangle versions" [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/809744 [04:30:35] AndyRussG: can you check ^^ [04:31:08] I have no idea how that is present - so I'm gonna do some more checks but I think that getting back in again is the problem [04:36:40] It must have been a rebase or something - I am just checking if that is why I had to hack the tests to pass [04:39:17] I'm also trying re-doing a 'clean' update (ie re-download the tarball & then just apply settings.location.php & the message template reduced patch - without the upgrade stuff (script has run), the test hack and the revert - to see if that passes - if it does that would be clean [04:43:51] eileen: one sec [04:45:34] (03CR) 10AndyRussG: [C: 03+2] Revert "Temporary hack to wrangle versions" [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/809744 (owner: 10Eileen) [04:45:42] eileen: done :) [04:47:38] thanks AndyRussG [04:48:13] it's weird because the way I do those updates is I check in the tarball so I messed something up [04:51:56] (03PS1) 10AndyRussG: Fix GatewayChooser pass-through '0' value params [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/809746 (https://phabricator.wikimedia.org/T310895) [04:52:07] eileen: don't worry :) [04:52:16] also I'm going to sleep in a few, so last call from the +2 bar [04:52:26] :-) [04:53:48] (03CR) 10CI reject: [V: 04-1] Fix GatewayChooser pass-through '0' value params [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/809746 (https://phabricator.wikimedia.org/T310895) (owner: 10AndyRussG) [04:54:09] (03PS1) 10Eileen: Civi-stock [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/809747 [04:54:11] (03PS1) 10Eileen: Re-add settings_location.php [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/809748 (https://phabricator.wikimedia.org/T303883) [04:54:13] (03PS1) 10Eileen: Re-add WMF code to support our templates [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/809749 (https://phabricator.wikimedia.org/T303883) [04:54:21] 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising-Backlog, 10MW-1.39-notes (1.39.0-wmf.17; 2022-06-20), 10Patch-For-Review: Some parameters dropped by Special:GatewayChooser for recurring donations - https://phabricator.wikimedia.org/T310895 (... [04:57:33] (03Merged) 10jenkins-bot: Revert "Temporary hack to wrangle versions" [wikimedia/fundraising/crm/civicrm] - 10https://gerrit.wikimedia.org/r/809744 (owner: 10Eileen) [04:59:18] (03PS1) 10Eileen: Submodule update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809750 [05:00:30] (03CR) 10Eileen: [C: 03+2] Submodule update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809750 (owner: 10Eileen) [05:01:12] (03PS2) 10AndyRussG: Fix GatewayChooser pass-through '0' value params [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/809746 (https://phabricator.wikimedia.org/T310895) [05:15:57] (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/809751 [05:16:27] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/809751 (owner: 10Eileen) [05:21:13] RECOVERY - check_mysql on frdb2002 is OK: Uptime: 218587 Threads: 10 Questions: 3683569 Slow queries: 96 Opens: 2494 Flush tables: 1 Open tables: 888 Queries per second avg: 16.851 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [05:23:01] !log civicrm upgraded from 9e5a5310 to 55bc690b [05:23:05] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [05:30:42] ok. frdb2002 is all caught up. i'm going to head to bed now. [05:30:46] have a good one eileen. [08:30:01] 10Fundraising-Backlog, 10DNS, 10Infrastructure-Foundations, 10Mail, and 3 others: Consider if to support BIMI for wiki mail - https://phabricator.wikimedia.org/T311685 (10greg) The email team in fundraising has interest in this topic as well. [08:35:32] 10Fundraising-Backlog, 10DNS, 10Infrastructure-Foundations, 10Mail, and 3 others: Consider if to support BIMI for wiki mail - https://phabricator.wikimedia.org/T311685 (10jcrespo) Probably related: T211404 T167337 [11:32:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [11:32:16] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [11:35:16] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [11:36:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [11:37:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [11:37:16] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [11:40:16] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [11:41:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [11:42:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [11:42:16] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [11:45:16] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [11:46:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [11:47:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [11:47:16] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [11:50:16] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [11:51:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [11:52:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [11:52:16] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [11:55:16] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [11:56:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [11:56:50] howdy. checking on those alerts. [11:57:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [11:57:14] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [11:59:11] hmmm... [11:59:23] fr-tech: anyone online? [12:00:16] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [12:01:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [12:01:28] i'm going to ack these alerts. [12:02:12] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [12:02:16] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [12:05:16] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [12:06:12] PROBLEM - check_mysql on frdb2002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [12:06:16] ACKNOWLEDGEMENT - check_mysql on frdb1002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) Dwisehaupt investigating replication error - The acknowledgement expires at: 2022-07-01 16:04:36. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [12:06:17] ACKNOWLEDGEMENT - check_mysql on frdb2001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) Dwisehaupt investigating replication error - The acknowledgement expires at: 2022-07-01 16:04:36. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [12:06:17] ACKNOWLEDGEMENT - check_mysql on frdb2002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) Dwisehaupt investigating replication error - The acknowledgement expires at: 2022-07-01 16:04:36. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [12:06:17] ACKNOWLEDGEMENT - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) Dwisehaupt investigating replication error - The acknowledgement expires at: 2022-07-01 16:04:36. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [13:06:14] RECOVERY - check_mysql on frdb2002 is OK: Uptime: 246487 Threads: 11 Questions: 4317134 Slow queries: 139 Opens: 2556 Flush tables: 1 Open tables: 946 Queries per second avg: 17.514 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [13:12:08] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 246470 Threads: 10 Questions: 4306929 Slow queries: 135 Opens: 2599 Flush tables: 1 Open tables: 988 Queries per second avg: 17.474 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [13:20:14] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 244639 Threads: 13 Questions: 4541337 Slow queries: 265313 Opens: 42416468 Flush tables: 1 Open tables: 200 Queries per second avg: 18.563 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [13:22:14] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 245099 Threads: 11 Questions: 4198089 Slow queries: 134 Opens: 36724732 Flush tables: 1 Open tables: 200 Queries per second avg: 17.128 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2501 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [13:48:57] (03PS20) 10Jgleeson: Tests: Front-end calls to charge one-time donation via Braintree [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/805906 (https://phabricator.wikimedia.org/T303419) [13:49:01] (03PS1) 10Jgleeson: Add in fallbacks for Braintree result mapping. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/809992 [13:59:28] 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Braintree-Integration: Network configuration for Braintree integration - https://phabricator.wikimedia.org/T303457 (10Jgreen) >>! In T303457#8038683, @Damilare wrote: > @Jgreen and @Dwisehaupt the list o... [14:25:30] 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Braintree-Integration: Network configuration for Braintree integration - https://phabricator.wikimedia.org/T303457 (10Jgreen) [14:35:08] 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Braintree-Integration: Network configuration for Braintree integration - https://phabricator.wikimedia.org/T303457 (10Jgreen) [14:36:43] 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Braintree-Integration: configure endpoint monitoring for Braintree API hosts - https://phabricator.wikimedia.org/T311727 (10Jgreen) [14:41:25] howdy fr-tech! [14:42:25] hi jgleeson fr-tech [14:43:09] hows it goin cstone ? [14:43:54] was hoping to work on the deck today but our neighbors are here cleaning up their place for the 4th [14:44:14] so weedwhacking is strong [14:44:30] how is your garden? [14:45:35] ha nice! [14:45:44] the garden is still a WIP [14:46:13] I have turf and soil looking from the window at me that I'm gonna lay later tonight [14:46:33] this is the last bit [14:46:36] I hope [14:46:38] nice [14:48:29] what work you doin on your deck? [14:49:27] oh i just work out there haha [14:49:32] when I have the tree background :P [14:50:30] programming work :P [14:51:36] ah gotcha. nice tree background :) [14:53:38] but the lots here are long and skinny so their place is like 10 ft from ours [14:54:34] I had to google weed whacker [14:54:41] we call them things strimmers [14:54:45] side trimmers [14:54:55] oooh [14:54:55] I need one actually, mine isn't so good [14:56:31] I'm trying to come up with a plan to find the cash to get one of these things https://www.amazon.co.uk/Flymo-EasiLife-Robotic-Lawn-Mower/dp/B08W8D4JS6/ref=sr_1_3?keywords=robotic+lawnmower&qid=1656600936&sr=8-3 [14:57:08] haha right? if I ever somehow get control of the yard here I want one too [14:57:34] although the dirt being 90% sand helps the grass not grow too crazily [14:57:45] ah [15:01:07] around here it's getting trendy to have your yard be a mix of wildflowers and weeds, I can get behind that b/c I hate mowing! [15:02:36] nice we do have some large clover patches but I think wed need to bring in real dirt to help with the really sandy parts [15:02:44] help it grow there [15:04:51] we have the same problem for a good chunk of our yard, the only thing that likes to grow there is crabgrass [15:05:46] Jeff_Green: you just reminded me of the time Roxy's grandmother complimented our garden saying "This garden is great for the kids, it's like a wild fairy garden". We had weeds everywhere [15:06:00] ha! [15:06:51] people actually put up signs here saying "No Mow May" and more info about supporting pollinators, can we just make that "No Mow May-Oct"? [15:07:03] ooh nice yeah ^^ [15:09:42] I have gotten my dad to not cut down all the baby trees that are growing in/at the edge of our rip rap [15:09:52] im hoping to use that chaos to help with any high water [15:19:54] (03PS1) 10Damilare Adedoyin: Accomodate fluent api [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/810015 [15:21:01] (03CR) 10CI reject: [V: 04-1] Accomodate fluent api [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/810015 (owner: 10Damilare Adedoyin) [15:28:16] (03PS2) 10Damilare Adedoyin: Accomodate fluent api [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/810015 [15:31:03] (03PS1) 10Cstone: WIP: Add Braintree to payment processors [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/810020 [15:31:05] (03PS1) 10Cstone: WIP: Add Braintree as a payment processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/810021 [15:31:15] what happened here [15:31:37] (03Abandoned) 10Cstone: WIP: Add Braintree as a payment processor [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/810021 (owner: 10Cstone) [15:32:29] lol [15:32:36] gerrit blip [15:32:48] no it was a git commit -amend vers --amend chaos [15:33:23] self sabotage haha [15:33:55] (03PS2) 10Cstone: WIP: Add Braintree to payment processors [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/810020 [16:08:43] (03CR) 10Jgleeson: [C: 03+2] "LGTM!" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/810015 (owner: 10Damilare Adedoyin) [16:09:27] (03Merged) 10jenkins-bot: Accomodate fluent api [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/810015 (owner: 10Damilare Adedoyin) [16:42:21] fr-tech I'm gonna miss alerts retro as it's dinner time here and my mum is over. back in a bit [17:13:36] 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Braintree-Integration: Network configuration for Braintree integration - https://phabricator.wikimedia.org/T303457 (10Jgreen) @Damilare & @Ejegg: FYI payments staging gets the production network config,... [17:35:58] 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Braintree-Integration: Network configuration for Braintree integration - https://phabricator.wikimedia.org/T303457 (10Jgreen) [17:56:55] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Thank you mailer activity creation is broken - https://phabricator.wikimedia.org/T311762 (10Ejegg) [17:59:27] 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Braintree-Integration: configure endpoint monitoring for Braintree API hosts - https://phabricator.wikimedia.org/T311727 (10Jgreen) [18:01:11] 10Fundraising-Backlog, 10DNS, 10Infrastructure-Foundations, 10Mail, and 3 others: Consider if to support BIMI for wiki mail - https://phabricator.wikimedia.org/T311685 (10ssingh) p:05Triage→03Medium [18:36:03] 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Braintree-Integration: Network configuration for Braintree integration - https://phabricator.wikimedia.org/T303457 (10Jgreen) p:05Triage→03Medium [18:36:28] 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Braintree-Integration: configure endpoint monitoring for Braintree API hosts - https://phabricator.wikimedia.org/T311727 (10Jgreen) p:05Triage→03Medium a:03Jgreen [19:02:47] 10fundraising-tech-ops, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install frdb1005, frdev1003 - https://phabricator.wikimedia.org/T306935 (10Jgreen) [19:16:26] hi fr-tech, seems my VPS host is having issues [19:16:41] so I've got no backscroll and missed pings for the last hour or so [19:17:44] From what I can see on backscroll, I don't think you missed anything important. [19:45:59] 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Braintree-Integration: configure endpoint monitoring for Braintree API hosts - https://phabricator.wikimedia.org/T311727 (10Jgreen) [20:53:24] (03CR) 10Ejegg: [C: 03+2] Fix GatewayChooser pass-through '0' value params [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/809746 (https://phabricator.wikimedia.org/T310895) (owner: 10AndyRussG) [20:55:02] (03CR) 10Ejegg: "Want to undo that country_fields change?" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/808790 (https://phabricator.wikimedia.org/T310895) (owner: 10AndyRussG) [20:55:50] (03CR) 10AndyRussG: Test pass-through of additional params for GatewayChoosre (031 comment) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/808790 (https://phabricator.wikimedia.org/T310895) (owner: 10AndyRussG) [21:04:06] (03PS3) 10Cstone: Add Braintree to payment processors [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/810020 [21:06:42] (03PS2) 10AndyRussG: Test pass-through of additional params for GatewayChoosre [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/808790 (https://phabricator.wikimedia.org/T310895) [21:20:02] (03Merged) 10jenkins-bot: Fix GatewayChooser pass-through '0' value params [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/809746 (https://phabricator.wikimedia.org/T310895) (owner: 10AndyRussG) [21:28:23] huh that took a while to CI its way along... almost 30 min! [21:32:27] recurring payment charged woo [21:32:43] yaayyy [21:37:22] (03PS1) 10Cstone: Add Paypal as a payment method in recurring Smashpig [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/810105 (https://phabricator.wikimedia.org/T303425) [21:41:19] (03PS2) 10Cstone: WIP: Add recurring for Braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/809048 (https://phabricator.wikimedia.org/T303425) [21:41:33] (03CR) 10CI reject: [V: 04-1] WIP: Add recurring for Braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/809048 (https://phabricator.wikimedia.org/T303425) (owner: 10Cstone) [21:42:23] (03PS3) 10Ejegg: Test pass-through of additional params for GatewayChoosre [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/808790 (https://phabricator.wikimedia.org/T310895) (owner: 10AndyRussG) [21:42:28] (03CR) 10Ejegg: [C: 03+2] Test pass-through of additional params for GatewayChoosre [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/808790 (https://phabricator.wikimedia.org/T310895) (owner: 10AndyRussG) [21:46:57] (03PS3) 10Cstone: WIP: Add recurring for Braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/809048 (https://phabricator.wikimedia.org/T303425) [21:47:37] (03CR) 10CI reject: [V: 04-1] WIP: Add recurring for Braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/809048 (https://phabricator.wikimedia.org/T303425) (owner: 10Cstone) [21:50:57] thx ejegg :) [21:51:21] alsop another oops on the speling in the commit message ;p [21:51:38] or maybe it was French [21:51:46] GatewayChoosrée [21:59:14] (03Merged) 10jenkins-bot: Test pass-through of additional params for GatewayChoosre [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/808790 (https://phabricator.wikimedia.org/T310895) (owner: 10AndyRussG) [22:04:35] 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Braintree-Integration: Network configuration for Braintree integration - https://phabricator.wikimedia.org/T303457 (10Damilare) @Jgreen thanks for all the work on this front. No we wouldn't be needing th... [22:09:25] well I don't see any flood of emails so I guess the import went ok :-) - eyener we will need to figure out how to get people going on the ability to create Data Segments - without David :-( [22:17:45] (03CR) 10Wfan: [C: 03+2] "Looks good, and now I can remove my payment_instrument_id from the test cases 😊" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/810020 (owner: 10Cstone) [22:24:40] (03PS1) 10Ejegg: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/810116 [22:24:47] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/810116 (owner: 10Ejegg) [22:25:36] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [extensions/DonationInterface] (deployment) - 10https://gerrit.wikimedia.org/r/810116 (owner: 10Ejegg) [22:28:12] 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add Braintree audit parser to SmashPig - https://phabricator.wikimedia.org/T310757 (10AnnWF) a:03AnnWF [22:28:14] ejegg: thx for deploying! [22:28:27] 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Add Braintree audit parser to SmashPig - https://phabricator.wikimedia.org/T310757 (10AnnWF) https://gerrit.wikimedia.org/r/c/wikimedia/fu... [22:28:53] (03PS4) 10Eileen: Remove flush, move geocoders to post action [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809734 [22:29:11] (03PS4) 10Eileen: Remove system.flush that should be obsolete now [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809332 [22:30:36] (03Merged) 10jenkins-bot: Add Braintree to payment processors [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/810020 (owner: 10Cstone) [22:31:54] (03CR) 10CI reject: [V: 04-1] Remove flush, move geocoders to post action [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809734 (owner: 10Eileen) [22:32:40] (03CR) 10CI reject: [V: 04-1] Remove system.flush that should be obsolete now [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/809332 (owner: 10Eileen) [22:37:40] (03PS1) 10Ejegg: Merge branch 'REL1_35' into fundraising/REL1_35 [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/810120 [22:38:14] (03PS1) 10Ejegg: Update DonationInterface submodule [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/810121 [22:38:44] (03CR) 10Ejegg: [C: 03+2] Update DonationInterface submodule [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/810121 (owner: 10Ejegg) [22:49:35] (03CR) 10Ejegg: [C: 03+2] Merge branch 'REL1_35' into fundraising/REL1_35 [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/810120 (owner: 10Ejegg) [22:49:44] oh derp, I thought I had already +2ed that one [23:00:45] (03Merged) 10jenkins-bot: Merge branch 'REL1_35' into fundraising/REL1_35 [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/810120 (owner: 10Ejegg) [23:00:50] (03Merged) 10jenkins-bot: Update DonationInterface submodule [core] (fundraising/REL1_35) - 10https://gerrit.wikimedia.org/r/810121 (owner: 10Ejegg) [23:59:16] oh hmm, i just tried ctrl-c at the start of 'fundraising_code_update -p payments-wiki=rollback' [23:59:22] because I didn't mean to roll back [23:59:29] just was using shell history [23:59:57] and now I'm not sure if it's still refreshing or frozen or what