[00:00:04] oh, just output another . [00:00:09] I guess it's still going [00:00:12] and I can abort later [00:35:00] yeah. iirc it has some smarter breakpoints set now so that it won't leave git in a weird state. [00:53:17] !log updated payments-wiki from ef53c82e to 78dee85e [00:53:20] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [01:23:15] 10Fundraising Sprint Kermit hopping principle, 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising-Backlog, 10MW-1.39-notes (1.39.0-wmf.17; 2022-06-20): Some parameters dropped by Special:GatewayChooser for recurring donations - https://phabricator.wikimedia.org/T310895 (10AndyRussG) @HNordee... [03:02:06] ejegg|away: just saw the mailing errors phab [03:02:18] we can turn off that thank you tracking I think [03:03:06] will we get bounce detection otherwise? [04:42:37] no - not until we solve it - but if it's not going out? [10:35:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [10:36:13] PROBLEM - check_mysql on frdb2002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [10:37:11] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [10:37:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [10:40:15] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [10:41:13] PROBLEM - check_mysql on frdb2002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [10:42:13] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [10:42:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [10:45:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [10:46:13] PROBLEM - check_mysql on frdb2002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [10:47:07] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [10:47:13] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [10:50:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [10:51:13] PROBLEM - check_mysql on frdb2002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [10:52:13] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [10:52:15] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [10:55:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [10:56:13] PROBLEM - check_mysql on frdb2002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [10:57:08] PROBLEM - check_mysql on frdb2001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [10:57:17] PROBLEM - check_mysql on frdb1002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [10:59:54] i'm investigating these ^^^ [11:00:17] PROBLEM - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [11:01:13] PROBLEM - check_mysql on frdb2002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [11:01:28] ACKNOWLEDGEMENT - check_mysql on frdb1002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) Dwisehaupt investigating mysql replication issue https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [11:01:28] ACKNOWLEDGEMENT - check_mysql on frdb2001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) Dwisehaupt investigating mysql replication issue https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [11:01:28] ACKNOWLEDGEMENT - check_mysql on frdb2002 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) Dwisehaupt investigating mysql replication issue https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [11:01:28] ACKNOWLEDGEMENT - check_mysql on frdev1001 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) Dwisehaupt investigating mysql replication issue https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [11:16:13] RECOVERY - check_mysql on frdb2002 is OK: Uptime: 326287 Threads: 12 Questions: 5288842 Slow queries: 171 Opens: 2704 Flush tables: 1 Open tables: 996 Queries per second avg: 16.209 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2002&service=check_mysql [11:17:07] RECOVERY - check_mysql on frdb2001 is OK: Uptime: 325970 Threads: 10 Questions: 5277350 Slow queries: 169 Opens: 2715 Flush tables: 1 Open tables: 1005 Queries per second avg: 16.189 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2001&service=check_mysql [11:17:13] RECOVERY - check_mysql on frdb1002 is OK: Uptime: 323999 Threads: 11 Questions: 5150242 Slow queries: 164 Opens: 44819320 Flush tables: 1 Open tables: 303 Queries per second avg: 15.895 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2217 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1002&service=check_mysql [12:46:21] 10Fundraising Sprint Localhost Hospitality Studies, 10Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Braintree-Integration: configure endpoint monitoring for Braintree API hosts - https://phabricator.wikimedia.org/T311727 (10Jgreen) a:05Jgreen→03None [13:20:48] 10Fundraising-Backlog, 10fundraising-tech-ops, 10DBA: Civicrm mariadb replication breaks on foreign key constraint - https://phabricator.wikimedia.org/T311838 (10Jgreen) p:05Triage→03High [13:24:08] 10fundraising-tech-ops, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:(Need By: TBD) rack/setup/install frlog1002 - https://phabricator.wikimedia.org/T306839 (10Cmjohnson) [13:24:38] 10fundraising-tech-ops, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install frdb1005, frdev1003 - https://phabricator.wikimedia.org/T306935 (10Cmjohnson) [13:35:15] RECOVERY - check_mysql on frdev1001 is OK: Uptime: 331939 Threads: 18 Questions: 5801713 Slow queries: 360686 Opens: 54129821 Flush tables: 1 Open tables: 200 Queries per second avg: 17.478 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdev1001&service=check_mysql [13:41:09] 10fundraising-tech-ops, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install frdb1005, frdb1006 (dev database) - https://phabricator.wikimedia.org/T306935 (10Jgreen) [13:43:02] 10fundraising-tech-ops, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install frdb1005, frdb1006 (dev database) - https://phabricator.wikimedia.org/T306935 (10Jgreen) [13:50:15] 10fundraising-tech-ops, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install frdb1005, frdb1006 (dev database) - https://phabricator.wikimedia.org/T306935 (10Jgreen) I renamed what was originally "frdev1003" on this task to "frdb1006" because that better describes the server's role. [13:58:05] 10Fundraising-Backlog, 10fundraising-tech-ops, 10DBA: Civicrm mariadb replication breaks on foreign key constraint - https://phabricator.wikimedia.org/T311838 (10Jgreen) [14:14:50] 10fundraising-tech-ops, 10DC-Ops, 10SRE, 10ops-eqiad, 10Patch-For-Review: (Need By: TBD) rack/setup/install frdb1005, frdb1006 (dev database) - https://phabricator.wikimedia.org/T306935 (10Jgreen) [14:55:53] 10fundraising-tech-ops: Fundraising access request for Sarah Guseman (Endowment Assistant - Contractor) - https://phabricator.wikimedia.org/T292212 (10JAdams) 05Resolved→03Open Hi @Lgruwell-WMF! Would you please be able to give approval for Sarah Guseman, Endowment Assistant, to gain access to Fundraising da... [15:13:07] (03CR) 10Damilare Adedoyin: [C: 03+2] Add in fallbacks for Braintree result mapping. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/809992 (owner: 10Jgleeson) [15:16:13] (03Merged) 10jenkins-bot: Add in fallbacks for Braintree result mapping. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/809992 (owner: 10Jgleeson) [15:34:16] (03PS21) 10Damilare Adedoyin: Tests: Front-end calls to charge one-time donation via Braintree [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/805906 (https://phabricator.wikimedia.org/T303419) (owner: 10Jgleeson) [16:10:50] (03CR) 10Damilare Adedoyin: [C: 03+2] Tests: Front-end calls to charge one-time donation via Braintree [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/805906 (https://phabricator.wikimedia.org/T303419) (owner: 10Jgleeson) [16:25:50] (03Merged) 10jenkins-bot: Tests: Front-end calls to charge one-time donation via Braintree [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/805906 (https://phabricator.wikimedia.org/T303419) (owner: 10Jgleeson) [16:31:54] 10Fundraising-Backlog, 10fundraising-tech-ops: Vendor to Contractor Civi access - https://phabricator.wikimedia.org/T311548 (10Dwisehaupt) Engage roles removed from user account and granted CiviCRM Admin role. [16:56:12] 10Fundraising-Backlog, 10fundraising-tech-ops: Vendor to Contractor Civi access - https://phabricator.wikimedia.org/T311548 (10Dwisehaupt) Certificate has been reissued and sent via email. Password sent via SMS. The existing certificate will work while she takes the time to install the new one. [17:00:22] thanks for the review damilare ! [17:00:33] np [17:00:39] I think that's out sprint goal right? ready for the one-hour test [17:00:53] merged, not yet deployed [17:01:13] We can deploy it all next week. Probably best to do that Tuesday when everyone is around [17:03:09] yep that sounds about right. [19:02:24] (03PS1) 10Cstone: WIP: Add Braintree recurring [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/810394 (https://phabricator.wikimedia.org/T303425) [19:13:47] (03PS4) 10Cstone: WIP: Add recurring for Braintree [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/809048 (https://phabricator.wikimedia.org/T303425) [20:08:32] 10fundraising-tech-ops: Fundraising access request for Sarah Guseman (Endowment Assistant - Contractor) - https://phabricator.wikimedia.org/T292212 (10Lgruwell-WMF) Approved [20:12:22] 10fundraising-tech-ops: Fundraising access request for Sarah Guseman (Endowment Assistant - Contractor) - https://phabricator.wikimedia.org/T292212 (10Dwisehaupt) [20:13:58] 10fundraising-tech-ops: Fundraising access request for Sarah Guseman (Endowment Assistant - Contractor) - https://phabricator.wikimedia.org/T292212 (10Dwisehaupt) [20:14:31] 10fundraising-tech-ops: Fundraising access request for Sarah Guseman (Endowment Assistant - Contractor) - https://phabricator.wikimedia.org/T292212 (10Dwisehaupt) @JAdams I have added superset to the the list of services above. Are there other bits that Sarah needs at this time? [20:21:32] 10fundraising-tech-ops, 10DC-Ops, 10SRE, 10ops-eqiad: (Need By: TBD) rack/setup/install frdb1005, frdb1006 (dev database) - https://phabricator.wikimedia.org/T306935 (10Cmjohnson) 05Open→03Resolved @Jgreen updated frdev1003 to frdb1006, thank you for fixing dns. The on-site work has been completed, i... [20:24:06] 10fundraising-tech-ops: Fundraising access request for Sarah Guseman (Endowment Assistant - Contractor) - https://phabricator.wikimedia.org/T292212 (10Dwisehaupt) [20:24:41] 10fundraising-tech-ops: Fundraising access request for Sarah Guseman (Endowment Assistant - Contractor) - https://phabricator.wikimedia.org/T292212 (10Dwisehaupt) Superset account created and password set. Password sent via SMS and instructions on how to change it sent via email. [20:39:11] 10fundraising-tech-ops, 10DC-Ops, 10SRE, 10ops-eqiad: Q3:(Need By: TBD) rack/setup/install frlog1002 - https://phabricator.wikimedia.org/T306839 (10Cmjohnson) @Jgreen i don't seem to have the template directory or 10.in file in my DNS repo to make changes for you. If you can update frlog1002's dns then you... [21:47:37] 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Thank you mailer activity creation is broken - https://phabricator.wikimedia.org/T311762 (10Eileenmcnaughton) This appearst to be the same issue as https://phabricator.wikimedia.org/T311838 (different symptom) [21:47:55] 10Fundraising-Backlog, 10fundraising-tech-ops, 10DBA: Civicrm mariadb replication breaks on foreign key constraint - https://phabricator.wikimedia.org/T311838 (10Eileenmcnaughton) This is a variant of https://phabricator.wikimedia.org/T311762 I believe [22:39:47] eileen: yeah. the thing that is weird about the query in T311838 is that it breaks replication. even though when we run the query after it runs fine. [22:39:47] T311838: Civicrm mariadb replication breaks on foreign key constraint - https://phabricator.wikimedia.org/T311838 [22:40:18] yeah that's super weird - has it recurred? I wonder if it relates to the warnings on upgrade [22:40:25] and there are plenty of other instances of that UPDATE where it succeeds perfectly fine. [22:41:09] it happened yesterday at 1130 UTC and again today at 1045 UTC iirc. [22:52:34] sorry, those timestamps with dates would be would be 2022-06-30 1130 UTC and 2022-07-01 1045 UTC [22:52:57] since days don't make sense with a dateline