[09:25:54] (03PS9) 10Damilare Adedoyin: Setup 3DS Flow [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1060129 (https://phabricator.wikimedia.org/T367786) [09:34:02] (03PS2) 10Damilare Adedoyin: Gravy ACH (Trustly) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1063242 (https://phabricator.wikimedia.org/T371810) [09:42:31] (03PS3) 10Damilare Adedoyin: Gravy ACH (Trustly) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1063242 (https://phabricator.wikimedia.org/T371810) [09:44:30] (03CR) 10CI reject: [V:04-1] Gravy ACH (Trustly) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1063242 (https://phabricator.wikimedia.org/T371810) (owner: 10Damilare Adedoyin) [09:44:34] (03PS10) 10Damilare Adedoyin: Setup 3DS flow [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1060421 (https://phabricator.wikimedia.org/T367786) [09:45:48] (03PS4) 10Damilare Adedoyin: Gravy ACH (Trustly) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1063242 (https://phabricator.wikimedia.org/T371810) [09:46:53] (03PS11) 10Damilare Adedoyin: Setup 3DS flow [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1060421 (https://phabricator.wikimedia.org/T367786) [13:04:20] (03PS3) 10Damilare Adedoyin: Ensure Gravy recurring card payments can be charged via Civi SmashPig recurring charge job [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1063191 (https://phabricator.wikimedia.org/T372559) [13:04:57] (03CR) 10CI reject: [V:04-1] Ensure Gravy recurring card payments can be charged via Civi SmashPig recurring charge job [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1063191 (https://phabricator.wikimedia.org/T372559) (owner: 10Damilare Adedoyin) [13:16:00] (03PS4) 10Damilare Adedoyin: Ensure Gravy recurring card payments can be charged via Civi SmashPig recurring charge job [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1063191 (https://phabricator.wikimedia.org/T372559) [13:17:05] (03CR) 10CI reject: [V:04-1] Ensure Gravy recurring card payments can be charged via Civi SmashPig recurring charge job [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1063191 (https://phabricator.wikimedia.org/T372559) (owner: 10Damilare Adedoyin) [13:19:37] (03PS5) 10Damilare Adedoyin: Ensure Gravy recurring card payments can be charged via Civi SmashPig recurring charge job [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1063191 (https://phabricator.wikimedia.org/T372559) [13:56:47] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: CentralNotice selenium tests failing - https://phabricator.wikimedia.org/T360899#10073331 (10zeljkofilipin) @AnnWF The test is checking if a banner with the `id` `centralnotice_testbanner` is displayed at https://en... [13:58:27] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 2 others: Request additional mgmt IP range for frack servers - https://phabricator.wikimedia.org/T370164#10073333 (10Papaul) @Dwisehaupt all working thank you. [14:03:54] (03CR) 10Damilare Adedoyin: "Thanks cstone for the CR, I found the cause for the error response and fixed. Kindly review again." [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1063191 (https://phabricator.wikimedia.org/T372559) (owner: 10Damilare Adedoyin) [14:08:27] (03PS11) 10Damilare Adedoyin: Gravy ACH (Trustly) [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1063208 (https://phabricator.wikimedia.org/T371810) [14:09:02] (03PS37) 10Damilare Adedoyin: Setup listener for transaction event message. [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1056531 (https://phabricator.wikimedia.org/T370797) [14:09:02] (03PS18) 10Damilare Adedoyin: ProcessCaptureRequestJob for authorization messages [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1057937 (https://phabricator.wikimedia.org/T370797) [14:09:02] (03PS12) 10Damilare Adedoyin: Record capture job [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1058580 (https://phabricator.wikimedia.org/T370797) [14:09:03] (03PS14) 10Damilare Adedoyin: Setup Refunds method and maintenance script [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1059078 (https://phabricator.wikimedia.org/T367808) [14:09:03] (03PS9) 10Damilare Adedoyin: Refund job [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1059356 (https://phabricator.wikimedia.org/T370797) [14:09:06] (03PS12) 10Damilare Adedoyin: Handle Report execution webhook message [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1059932 (https://phabricator.wikimedia.org/T370797) [14:09:09] (03PS10) 10Damilare Adedoyin: Setup 3DS Flow [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1060129 (https://phabricator.wikimedia.org/T367786) [14:12:32] (03PS5) 10Damilare Adedoyin: Gravy ACH (Trustly) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1063242 (https://phabricator.wikimedia.org/T371810) [14:12:43] (03PS12) 10Damilare Adedoyin: Setup 3DS flow [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1060421 (https://phabricator.wikimedia.org/T367786) [15:36:16] 06Fundraising-Backlog, 10FR-donorservices: Manually settled transactions didn't reach Civi - https://phabricator.wikimedia.org/T372480#10073965 (10AMJohnson) Just noting that for transaction # 207338904.1. This transaction had to be refunded so it can be removed from the 'needs to be in Civi list'. :) [16:16:51] fr-tech do these new fields look good for the gravy task? https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1063259 [16:21:23] 03Fundraising Sprint: powerfulFunctionPleaseOnlyCallIfAbsolutelyNecessary(), 06Fundraising-Backlog: Gravy test rollout plan - https://phabricator.wikimedia.org/T372469#10074210 (10jgleeson) Test links spreadsheet has been started [[ https://docs.google.com/spreadsheets/d/18qmz3GaPp8SA8U0vk-uftZsRBaoichWPn9JkL1... [16:21:27] 03Fundraising Sprint: powerfulFunctionPleaseOnlyCallIfAbsolutelyNecessary(), 06Fundraising-Backlog: Gravy test rollout plan - https://phabricator.wikimedia.org/T372469#10074211 (10jgleeson) [16:21:46] checking ejegg [16:50:44] (03CR) 10Jgleeson: [C:03+2] "LGTM!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1063259 (https://phabricator.wikimedia.org/T372468) (owner: 10Ejegg) [16:51:05] also thanks for the code review ejegg on the audit stuff [16:51:05] ejegg: I just went throught the new fields patch and compared with the notes on the sheet. Looks like you missed the bankend processor field [16:51:24] ahh jgleeson just reviewed [16:51:41] damilare: this one? https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1063079 [16:52:00] oops wrong link, this one https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1063058 [16:52:22] ohh different patch [16:52:27] cool thanks! [16:52:32] LGTM then! [17:08:50] Fr-tech I won't be able to make the India meeting in 20 mins [17:08:52] 03Fundraising Sprint: powerfulFunctionPleaseOnlyCallIfAbsolutelyNecessary(), 06Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2425Q1 maintenance window - https://phabricator.wikimedia.org/T337582#10074547 (10Dwisehaupt) [17:09:59] (03Merged) 10jenkins-bot: Add new fields for Gravy audit [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1063259 (https://phabricator.wikimedia.org/T372468) (owner: 10Ejegg) [17:17:28] Actually I'll try and join from the car [17:29:32] thanks jgleeson|dinner ! [17:29:37] I'll merge that to deploy [17:30:24] dwisehaupt: we'll have a new field sync to apply soon [17:31:11] ok. i should be able to get to it in a little. doing the kernel updates and reboots for a few hosts right now. [17:33:37] PROBLEM - Host frdb1005 is DOWN: PING CRITICAL - Packet loss = 100% [17:35:13] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1063856 [17:35:23] RECOVERY - Host frdb1005 is UP: PING OK - Packet loss = 0%, RTA = 0.49 ms [17:36:13] PROBLEM - check_mysql on frdb1006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [17:40:04] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1063856 (owner: 10Ejegg) [17:41:06] (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1063856 (owner: 10Ejegg) [17:41:11] PROBLEM - check_mysql on frdb1006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [17:43:21] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: CentralNotice selenium tests failing - https://phabricator.wikimedia.org/T360899#10074699 (10AnnWF) >>! In T360899#10073331, @zeljkofilipin wrote: > @AnnWF The test is checking if a banner with the `id` `centralnoti... [17:46:13] PROBLEM - check_mysql on frdb1006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [17:51:13] PROBLEM - check_mysql on frdb1006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [17:56:13] PROBLEM - check_mysql on frdb1006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [18:01:11] PROBLEM - check_mysql on frdb1006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [18:06:11] PROBLEM - check_mysql on frdb1006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [18:11:11] PROBLEM - check_mysql on frdb1006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [18:16:13] PROBLEM - check_mysql on frdb1006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [18:21:13] PROBLEM - check_mysql on frdb1006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [18:22:28] !log fundraising civicrm upgraded from 56521963 to fd01c939 [18:22:29] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [18:23:14] ok dwisehaupt the field script is deployed [18:23:34] but I think we want Civi to be in maintenance mode again to create them [18:23:42] i'm looking at frdb1006. it should have re-established replication. [18:23:53] ok [18:26:08] give me a few mins and i'll have it back in maint mode. do we want frpigs in maint for it also? [18:26:15] PROBLEM - check_mysql on frdb1006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [18:31:12] RECOVERY - check_mysql on frdb1006 is OK: Uptime: 2667697 Threads: 10 Questions: 155460640 Slow queries: 3141400 Opens: 5872 Open tables: 1914 Queries per second avg: 58.275 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [18:34:05] ejegg: ok. i'm ready to put it in maint mode when you are ready. [18:45:07] want to do it now dwisehaupt? [18:47:20] sure. [18:47:24] running puppet. [18:48:32] ok. pending queue consumer is still running but we are all set. [18:48:40] great, that one is OK [18:49:26] oh i should run under screen... [18:50:12] ok, running [18:50:34] let's check to make sure it's adding all in one alter [18:56:37] yep, seems to be doing that [19:31:02] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: CentralNotice selenium tests failing - https://phabricator.wikimedia.org/T360899#10075234 (10AnnWF) @Jdlrobson Hi Jon, I see you recently update the site en.wikipedia.beta.wmflabs.org, could you answer the question... [19:35:36] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: CentralNotice selenium tests failing - https://phabricator.wikimedia.org/T360899#10075241 (10Pcoombe) It's not a code issue. The beta cluster centralnotice campaigns are configured at https://meta.wikimedia.beta.wmf... [19:37:24] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: CentralNotice selenium tests failing - https://phabricator.wikimedia.org/T360899#10075247 (10AnnWF) >>! In T360899#10075241, @Pcoombe wrote: > It's not a code issue. The beta cluster centralnotice campaigns are conf... [19:39:40] (03PS4) 10Jgleeson: Add Gravy Audit parser [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1062119 [19:42:50] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 13Patch-For-Review: Consolidate error handling in queues - https://phabricator.wikimedia.org/T365418#10075271 (10XenoRyet) [19:42:54] (03CR) 10Jgleeson: "I've tentatively pushed up a change to convert the amounts from cents to dollars, but I see you noted we need to be mindful of other curre" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1062119 (owner: 10Jgleeson) [19:42:58] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Switch Acoustic mapping to use the API to define the columns - https://phabricator.wikimedia.org/T368470#10075274 (10XenoRyet) [19:43:16] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 07Browser-Support-Apple-Safari: Safari crashes on some click handling events for banners that move the centralNotice element outside of the the main body - https://phabricator.wikimedia.org/T242895#10075276 (10Izno) [19:43:18] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Uptick in Dutch donors experiencing issues donating - https://phabricator.wikimedia.org/T372571#10075275 (10EMartin) @AMJohnson Adyen is saying they cannot troubleshoot and think the matter is something on our end since it isn't cont... [19:43:30] 03Fundraising Sprint: powerfulFunctionPleaseOnlyCallIfAbsolutelyNecessary(), 06Fundraising-Backlog: 2024 Sprint O - Priorities - https://phabricator.wikimedia.org/T371308#10075283 (10AKanji-WMF) 05Open→03Resolved a:03AKanji-WMF [20:03:56] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-donorservices: Settled Adyen transaction not in Civi - https://phabricator.wikimedia.org/T372670#10075417 (10AKanji-WMF) p:05Triage→03Medium [20:09:25] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Uptick in Dutch donors experiencing issues donating - https://phabricator.wikimedia.org/T372571#10075444 (10AKanji-WMF) p:05Triage→03High [20:11:59] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice: CentralNotice should catch banner errors and log them to its own channel - https://phabricator.wikimedia.org/T361680#10075458 (10AKanji-WMF) p:05High→03Medium [20:13:01] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: ACH address setting N/A in civi instead of empty data - https://phabricator.wikimedia.org/T372482#10075464 (10AKanji-WMF) [20:13:46] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-donorservices: Manually settled transactions didn't reach Civi - https://phabricator.wikimedia.org/T372480#10075472 (10AKanji-WMF) [20:27:18] rejoined at the right time [20:31:34] wow, that table alter is still chugging [20:33:31] 14Fundraising Sprint: justWork(), 03Fundraising Sprint: powerfulFunctionPleaseOnlyCallIfAbsolutelyNecessary(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, and 2 others: Make email custom fields editable from contact summary, upstream patch - https://phabricator.wikimedia.org/T366356#10075527 (10... [20:34:49] 401* [20:36:22] I just moved the phab this https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1058727 is with back to review [20:36:38] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Civi-Dedupe, 13Patch-For-Review: Fail mail on dedupe - https://phabricator.wikimedia.org/T363965#10075534 (10Eileenmcnaughton) Ah no - there is a patch that needs reviewing for this https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1... [20:37:10] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1350 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [20:38:14] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1416 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [21:04:19] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: ACH address setting N/A in civi instead of empty data - https://phabricator.wikimedia.org/T372482#10075660 (10AnnWF) a:03AnnWF [21:36:43] ejegg: if you can face it this one is still hanging around https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/1058727 [21:39:27] looking [21:41:08] oh that's still on top of 5.76 [21:45:43] ejegg: is it ? - doesn't show that way to me [21:46:15] you mean 1058727? [21:46:41] oh huh, i wonder why my local install complains about the db in an unexpected version whn I pull it down [21:46:44] ok, nvm [21:46:46] will review [21:56:27] thanks ejegg - I know that one is a bit of a dog - but it will allow us to re-enable that job [21:56:43] ejegg: do you know how many alters are in the script? i'm assuming it's one per field for the table and log table. [21:58:45] judging by the replication lag metrics, it looks like it took about 60 mins for the first alter and each successive one has been around 20mins or so. [22:00:15] dwisehaupt: there's one big alter adding all the fields [22:00:36] oh. i wonder if the engine has divided them up. [22:00:56] since right now it's running: ALTER TABLE `civicrm`.log_wmf_contribution_extra ADD `interchange_fee` decimal(20,2) [22:06:19] yeah - that will be after the main alter [22:11:30] ah yes. i see it in the slow log now. the main query completed in 5065.507762. now it's running through the log table alters. [22:12:01] looks like we are on the last one, scheme_fee [22:14:24] it'll probably wrap up in 11 minutes. [22:31:41] 06Fundraising-Backlog: New Yubikey setup for Runjini Murthy - https://phabricator.wikimedia.org/T372837 (10RMurthy) 03NEW [22:38:38] 06Fundraising-Backlog, 10fundraising-tech-ops: New Yubikey setup for Runjini Murthy - https://phabricator.wikimedia.org/T372837#10075843 (10Dwisehaupt) Key has been updated and merged. It will be on the hosts with the next puppet run. ` commit 2691c4211925f8ffd147d5e0090e7c993b39bed3 Author: Dallas Wiseha... [22:38:49] 06Fundraising-Backlog, 10fundraising-tech-ops: New Yubikey setup for Runjini Murthy - https://phabricator.wikimedia.org/T372837#10075847 (10Dwisehaupt) 05Open→03Resolved [22:41:13] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4814 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [22:42:13] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4870 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [22:43:26] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 4936 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [22:46:13] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5115 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [22:47:09] PROBLEM - check_mysql on frdb2001 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3881 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [22:48:19] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 3947 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [22:48:38] woo, it's done! [22:49:29] my downtime expired. re-setting it. [23:09:24] (03PS1) 10Ejegg: Update triggers for new contribution_extra fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1063917 [23:09:38] dwisehaupt: does that look good? ^^ [23:09:47] or any other fr-tech that's around... [23:10:52] looking. [23:11:27] that looks like the right fields to me. [23:11:41] want me to +2 it? [23:11:57] (03CR) 10Eileen: [C:03+2] Update triggers for new contribution_extra fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1063917 (owner: 10Ejegg) [23:12:10] ha. she beat me to it. [23:12:30] :) [23:13:03] (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1063921 [23:13:06] (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1063921 (owner: 10Ejegg) [23:13:20] (03CR) 10Ejegg: [V:03+2] Update triggers for new contribution_extra fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1063917 (owner: 10Ejegg) [23:13:34] ok, I just manually submitted that one [23:14:33] (03CR) 10Ejegg: [V:03+2 C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1063921 (owner: 10Ejegg) [23:15:52] !log fundraising civicrm upgraded from fd01c939 to 1022abf1 [23:15:53] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [23:16:12] dwisehaupt: ok, the new triggers file is deployed to civi. Want to apply it to the db? [23:16:34] cool. i'll start the process. [23:17:58] diffs match the gerrit one. [23:18:07] thanks! [23:18:09] we should be good. [23:18:16] ready for me to pull us out of maint mode? [23:25:53] yep! [23:25:57] thanks dwisehaupt [23:26:32] ok. doing so. [23:29:40] ok. verified it's back up and i can run queries. [23:31:54] rockin [23:42:55] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 13Patch-For-Review: Uptick in Dutch donors experiencing issues donating - https://phabricator.wikimedia.org/T372571#10075947 (10EMartin) Hi All concerned. Adyen just opened a ticked seemingly about our issue: {F57281968} [23:52:09] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 269500 Threads: 4 Questions: 6618254 Slow queries: 124 Opens: 2352 Open tables: 1018 Queries per second avg: 24.557 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [23:53:15] RECOVERY - check_mysql on frdb1004 is OK: Uptime: 2699749 Threads: 5 Questions: 153003883 Slow queries: 1241 Opens: 5763 Open tables: 1028 Queries per second avg: 56.673 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql