[00:07:15] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/925096 [00:07:29] (03CR) 10Eileen: [C: 03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/925096 (owner: 10Eileen) [00:08:16] (03Merged) 10jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/925096 (owner: 10Eileen) [00:11:22] !log civicrm upgraded from 885208ca to 3819d6d1 [00:11:25] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [00:48:39] thanks eileen ! [00:48:56] no more as yet.... [00:49:08] hmm, i should have logged something [00:49:56] when do you go on sabatical? [00:52:35] the start of July [00:52:55] I should put that in the calendar! [00:54:13] ah dang i duplicated the silverpop_missing_countries table [00:56:16] and... there are 161 contacts with duplicate employer relationships [00:56:19] boo [00:56:49] i forget if we have a recipe for that written down somewhere [00:59:18] (03PS1) 10Eileen: Do not merge - locally generated triggers file [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/925143 [01:01:22] maybe on one of the old phabs? [01:02:10] https://phabricator.wikimedia.org/T306822 is still open [01:12:25] (03CR) 10CI reject: [V: 04-1] Do not merge - locally generated triggers file [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/925143 (owner: 10Eileen) [01:21:34] (03PS1) 10Eileen: Declare source in Benevity file set up [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/925163 [01:22:17] ejegg: I feel like we could get rid of that missing countries table - surely a one-off update on those contacts cos there are barely any involved [01:22:25] eileen: so again I have forgotten what to prioritize in cleaning up duplicate relationships [01:22:32] should we just always keep the newest? [01:22:44] and update the contact.employer_id to match? [01:22:54] yeah - can you give me an example? [01:23:15] cid 30955186 [01:23:33] the newer relationship is not the one matching the employer_id [01:24:58] looks like maybe manually added there? [01:25:13] Altered by: Dr. Nora Kathie Nichols [01:26:07] hmm - yeah - I guess we have to resolve that in acoustic - on the employer_id - t since we can't rule out 2 legit employers [01:27:03] ok, so we can add a join on employer_id to the acoustic script [01:27:18] let's see how many we have with duplicates of the same employer [01:27:26] But on that contact - the employer_id feels wrong... - cos the other was more recent [01:27:42] yeah, I'm surprised it didn't update [01:28:00] do you have to manually select 'is current employer' ? [01:28:57] or is that default? [01:30:56] I think on merge on the latest code it would 'probably' update .... [01:31:11] but it would take the value from the more recent contact [01:31:19] I did put out a slack ping on it [01:31:35] oh wait... we DO have the ON er.contact_id_a = c.id AND er.contact_id_b = c.employer_id AND er.is_active=1 AND er.relationship_type_id=( [01:31:48] or rather [01:31:57] er.contact_id_b = c.employer_id [01:32:21] OK, never mind, I don't actually see any more dupes with the same employer [01:32:31] I will just drop that doubled table and restart the rebuild [01:34:09] sorry for the confusion eileen [01:34:23] oh cool [01:49:15] So we did have a duplicate wmf employer relationship on megan's record [01:52:48] i deleted it, but now I want to know how it got there [01:53:34] hmm, there have been some merges [01:58:01] but those two relationship seem to have existed from the start of logging [01:58:14] i wonder why they didn't cause an error before [01:59:13] (03PS3) 10Eileen: Move our transaction handling class to extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/922945 [01:59:15] (03PS3) 10Eileen: Minor extraction [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/922947 [01:59:17] (03PS2) 10Eileen: Declare field names in one function only [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/924179 [01:59:19] (03PS2) 10Eileen: Do not merge - locally generated triggers file [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/925143 [01:59:21] (03PS1) 10Eileen: Transfer year-field calculated data to the field declaration [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/925200 [01:59:23] (03PS1) 10Eileen: Add WMF Donor api & test [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/925201 [02:00:15] ejegg: this is a teensy one on a test - doesn't actually do what I hoped but I think it is right anywayhttps://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/925163 [02:00:15] ooh nice [02:00:30] (nice to have WMF Donor API) [02:00:46] Yeah - I wanted to be able to see what would be generated for a given contact [02:00:55] (03CR) 10Ejegg: [C: 03+2] Declare source in Benevity file set up [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/925163 (owner: 10Eileen) [02:01:06] also trying to re-org so I can just get some fields & see the sql [02:01:30] (03Abandoned) 10Eileen: Add conditional to help with tests [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/924178 (owner: 10Eileen) [02:02:33] this one is bigger (would disrupt your thought process) but gets one namespace cleanup in https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/922945 [02:03:59] I hate the way it tags anything marked work in progress as having a merge conflict [02:04:14] oh huh [02:04:52] yeah this had no conflict - https://gerrit.wikimedia.org/r/c/wikimedia/fundraising/crm/+/925200 - untill I marked as work-in-progress [02:06:03] that tools repo bug where it marked everything as merge conflict really messed me up today [02:06:16] we had overridden a V-1 thinking it was spurious [02:06:24] but it turned out the patch actually failed tests! [02:06:45] ouch [02:06:47] so I had to fix some additional stuff to add those new recurring upgrade fields before I could kick off the rebuild [02:07:07] and missed the opportunity to do it today [02:10:10] well, that transaction move looks pretty harmless [02:10:30] cool - I have to shoot out for a bit [02:10:42] i guess we will have bootstrapped civi wherever we use it [02:10:48] ok, have a good rest of your day [02:10:48] yep [02:10:56] i'll be done mine soon [02:11:16] cya [02:13:54] (03Merged) 10jenkins-bot: Declare source in Benevity file set up [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/925163 (owner: 10Eileen) [02:24:49] (03CR) 10Ejegg: [C: 03+2] Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/924997 (owner: 10Ejegg) [02:25:26] (03Merged) 10jenkins-bot: Merge branch 'master' into deploy [wikimedia/fundraising/tools] (deploy) - 10https://gerrit.wikimedia.org/r/924997 (owner: 10Ejegg) [02:25:44] (03CR) 10Ejegg: [C: 03+2] "Cool, this looks safe enough!" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/922945 (owner: 10Eileen) [02:37:35] i'm assuming the new export is still running. [02:38:43] yarp. cool. [02:39:01] (03Merged) 10jenkins-bot: Move our transaction handling class to extension [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/922945 (owner: 10Eileen) [02:41:36] dwisehaupt: yep, new export is running on new server but won't actually send the files tonight. old export will run on old server and export tonight's files [02:45:09] sounds good [02:45:21] gonna reboot my lappy for kernel upgrades. [03:02:14] ah dang dwisehaupt the old export failed [03:02:17] let's see why [03:03:15] ohh, access denied [03:04:02] dwisehaupt: oh shoot, can you please revert the /etc/silverpop_export.yaml changes for now? [03:04:22] I probably shouldn't have used the nautical term to cancel my request... [03:16:41] (03PS4) 10Ejegg: Minor extraction [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/922947 (owner: 10Eileen) [03:16:45] (03CR) 10Ejegg: [C: 03+2] Minor extraction [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/922947 (owner: 10Eileen) [03:30:17] (03Merged) 10jenkins-bot: Minor extraction [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/922947 (owner: 10Eileen) [04:14:30] sorry, was planning college visits. i'll revert it. [04:18:22] ok. all reverted. [04:18:30] sorry for the delay. [04:27:12] email sent with info on the job that is running. [05:59:13] (03PS3) 10Eileen: Declare field names in one function only [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/924179 [06:21:19] (03CR) 10CI reject: [V: 04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_40) - 10https://gerrit.wikimedia.org/r/925391 (owner: 10L10n-bot) [10:44:40] 10Fundraising-Backlog: Formatting issue in 'Renew Your Recurring' email sent from Civi - https://phabricator.wikimedia.org/T337919 (10krobinson) [10:48:25] 10Fundraising-Backlog: Formatting issue in 'Renew Your Recurring' email sent from Civi - https://phabricator.wikimedia.org/T337919 (10krobinson) [12:51:14] RECOVERY - check_mysql on frdb1006 is OK: Uptime: 474 Threads: 4 Questions: 682396 Slow queries: 0 Opens: 547 Open tables: 540 Queries per second avg: 1439.654 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [14:34:54] (03CR) 10Ejegg: "Thanks, C. Scott! We ended up doing this in another patch when we went to 1.39" [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/711006 (owner: 10C. Scott Ananian) [14:38:31] (03PS2) 10Ejegg: Update extension for wikimedia/remex 2.3.2 (included in 1.37.0-wmf18) [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/711006 (owner: 10C. Scott Ananian) [14:39:42] (03CR) 10Ejegg: "Oh look, we still had the transitional code in there. I've rebased this one and edited it to include removal of the transitional code. If " [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/711006 (owner: 10C. Scott Ananian) [14:52:14] (03PS1) 10Ejegg: Remove stray semicolon [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/925827 [14:52:42] (03CR) 10Ejegg: [C: 03+2] Remove stray semicolon [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/925827 (owner: 10Ejegg) [14:57:22] (03Merged) 10jenkins-bot: Remove stray semicolon [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/925827 (owner: 10Ejegg) [15:01:14] (03PS40) 10Damilare Adedoyin: IP Velocity Filter: Increase count after first run [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/859168 (https://phabricator.wikimedia.org/T323544) (owner: 10AndyRussG) [15:02:55] (03CR) 10CI reject: [V: 04-1] IP Velocity Filter: Increase count after first run [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/859168 (https://phabricator.wikimedia.org/T323544) (owner: 10AndyRussG) [15:07:49] confusing - first and latest recurring dates are swapped [15:44:01] (03PS1) 10Ejegg: Explicitly specify insert column order [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/925848 [15:44:07] (03CR) 10CI reject: [V: 04-1] Explicitly specify insert column order [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/925848 (owner: 10Ejegg) [15:44:58] grrrr NO JENKINS IT DOES NOT NEED A REBASE [15:45:08] (03CR) 10Ejegg: "recheck" [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/925848 (owner: 10Ejegg) [15:45:14] (03CR) 10CI reject: [V: 04-1] Explicitly specify insert column order [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/925848 (owner: 10Ejegg) [15:50:21] (03CR) 10Ejegg: "recheck" [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/925848 (owner: 10Ejegg) [15:53:05] jgleeson: that gitlab change seems fine to me. Want to make the merge request to the releng/dev-images repo? [15:55:43] sure. thanks ejegg [16:16:04] (03CR) 10Ejegg: "recheck" [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/925848 (owner: 10Ejegg) [16:17:35] (03PS2) 10Ejegg: Explicitly specify insert column order [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/925848 [16:17:44] (03CR) 10CI reject: [V: 04-1] Explicitly specify insert column order [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/925848 (owner: 10Ejegg) [16:17:57] (03CR) 10Ejegg: "recheck" [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/925848 (owner: 10Ejegg) [16:22:44] (03CR) 10Ejegg: "recheck" [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/925848 (owner: 10Ejegg) [16:25:56] fr-tech that's finally got a V+2 ^^^ [16:26:09] fixes the mixup between first and last recurring donation date [16:29:33] and hopefully makes future mixups less likely [17:02:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [17:03:52] Susan Orlean is also funny on Twitter [17:07:13] RECOVERY - check_mysql on frdb2003 is OK: Uptime: 79480 Threads: 4 Questions: 40923547 Slow queries: 18 Opens: 1940 Open tables: 1798 Queries per second avg: 514.891 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [17:12:56] ejegg: i just rolled out that config change and did the puppet run on civi1001. should be all good now. [17:13:55] thanks! I'll try running it now [17:15:22] actually, I'll try it from civi1002 since that's where we'll move the job to [17:15:41] ah, probably need to wait for the puppet run there [17:15:47] still seeing the old hostname [17:23:07] i'll force it there. [17:23:48] ok. all set. [17:58:18] PROBLEM - check_mysql on payments2001 is CRITICAL: slave query error: Access denied: you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_mysql [17:58:18] PROBLEM - check_mysql on payments2003 is CRITICAL: slave query error: Access denied: you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_mysql [18:00:14] PROBLEM - check_mysql on payments1007 is CRITICAL: slave query error: Access denied: you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_mysql [18:00:14] PROBLEM - check_mysql on payments1008 is CRITICAL: slave query error: Access denied: you need (at least one of) the SUPER, SLAVE MONITOR privilege(s) for this operation https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_mysql [18:02:38] ^^^ this is known and being worked. [18:03:16] RECOVERY - check_mysql on payments2001 is OK: Uptime: 1207564 Threads: 5 Questions: 2490074 Slow queries: 0 Opens: 100 Open tables: 94 Queries per second avg: 2.062 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2001&service=check_mysql [18:03:16] RECOVERY - check_mysql on payments2003 is OK: Uptime: 1208360 Threads: 5 Questions: 2491393 Slow queries: 0 Opens: 100 Open tables: 94 Queries per second avg: 2.061 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2003&service=check_mysql [18:05:16] RECOVERY - check_mysql on payments1007 is OK: Uptime: 688443 Threads: 11 Questions: 2259673 Slow queries: 0 Opens: 106 Open tables: 100 Queries per second avg: 3.282 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1007&service=check_mysql [18:05:16] RECOVERY - check_mysql on payments1008 is OK: Uptime: 1190113 Threads: 4 Questions: 1540419 Slow queries: 0 Opens: 95 Open tables: 89 Queries per second avg: 1.294 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_mysql [19:19:35] why am I only now finding out about chrome tab groups [19:19:58] they have colours too [19:21:13] https://blog.google/products/chrome/manage-tabs-with-google-chrome/ [19:21:18] since 2020 apparently [19:37:08] maybe that just means you had a reasonable amount of tabs jgleeson and didnt have to go to groups [19:37:10] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [19:37:45] yep that was it cstone :) [19:38:17] way too much lightning with this current thunderstorm [19:42:10] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [19:47:10] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [19:48:38] wfan hmm, the export.py script didn't seem to create a new checksums file for me this time [19:48:43] let's see why [19:49:31] Ahh, I'm running the older code there [19:51:59] !log fundraising python tools upgraded from 72570bdd to 759d4c89 [19:52:00] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [19:52:10] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Yes Slave SQL: No Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [19:52:19] darn it, I'll have to write the whole 8gb main file again [19:57:10] RECOVERY - check_mysql on frdb2003 is OK: Uptime: 89680 Threads: 4 Questions: 43545223 Slow queries: 36 Opens: 1972 Open tables: 1817 Queries per second avg: 485.562 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [20:30:19] 10Fundraising-Backlog, 10fundraising-tech-ops: migrate Silverpop export job from civi1001/frdev1001 to civi1002+frdb1006 - https://phabricator.wikimedia.org/T337498 (10Ejegg) [20:35:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4835 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12350 MB (99% inode=99%): /tmp 12350 MB (99% inode=99%): /var 2558 MB (9% inode=80%): /srv 109995 MB (37% inode=99%): /var/tmp 422 MB (99% inode=99%): /var/log 23542 MB (88% [20:35:10] %): /var/log/audit 422 MB (99% inode=99%): /run/user/3649 6392 MB (100% inode=99%): /run/user/573 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [20:40:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4835 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12350 MB (99% inode=99%): /tmp 12350 MB (99% inode=99%): /var 1592 MB (6% inode=80%): /srv 109995 MB (37% inode=99%): /var/tmp 422 MB (99% inode=99%): /var/log 23534 MB (88% [20:40:10] %): /var/log/audit 422 MB (99% inode=99%): /run/user/3649 6392 MB (100% inode=99%): /run/user/573 6392 MB (100% inode=99%): /run/user/571 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [20:42:44] 10Fundraising-Backlog: Matching import date error - https://phabricator.wikimedia.org/T337995 (10MDemosWMF) [20:45:10] PROBLEM - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4835 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12350 MB (99% inode=99%): /tmp 12350 MB (99% inode=99%): /var 1592 MB (6% inode=80%): /srv 109998 MB (37% inode=99%): /var/tmp 422 MB (99% inode=99%): /var/log 23518 MB (88% [20:45:10] %): /var/log/audit 422 MB (99% inode=99%): /run/user/3649 6392 MB (100% inode=99%): /run/user/573 6392 MB (100% inode=99%): /run/user/571 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [20:45:26] ACKNOWLEDGEMENT - check_disk on civi1002 is CRITICAL: DISK CRITICAL - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4835 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12350 MB (99% inode=99%): /tmp 12350 MB (99% inode=99%): /var 1592 MB (6% inode=80%): /srv 109998 MB (37% inode=99%): /var/tmp 422 MB (99% inode=99%): /var/log 23518 [20:45:26] inode=99%): /var/log/audit 422 MB (99% inode=99%): /run/user/3649 6392 MB (100% inode=99%): /run/user/573 6392 MB (100% inode=99%): /run/user/571 6392 MB (100% inode=99%): Jeff_Green known. one-time rebuild. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [20:50:10] Hi ejegg ! Sorry for the delay in getting back to you, I was in a couple long meetings. How are things going with the export stuff, anything I can help with at this point? [20:51:54] katers_: we've got the CSVs generated, and I tried adding the new server's address to the allow list, but it's still giving me an authentication error trying to upload [20:52:06] I'll poke around a bit and make sure I used the right IP address [20:52:18] and that the new server has the right password [20:53:47] Ok, sounds good. thanks for the update! [20:57:33] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Change recurring rebilling logic from 'monthly' to 'on demand' in India through Dlocal - https://phabricator.wikimedia.org/T336893 (10spatton) Thanks @EMartin, the pre-test for this has been [[ https://app.asana.com/0/1125967519298615/1204738783815804 | s... [21:03:32] 10Fundraising Tech - Chaos Crew, 10Fundraising-Backlog: Change recurring rebilling logic from 'monthly' to 'on demand' in India through Dlocal - https://phabricator.wikimedia.org/T336893 (10EMartin) @spatton Thank you, Sam! [21:05:25] 10Fundraising-Backlog: Fix India form text cosmetics before pretest - https://phabricator.wikimedia.org/T337520 (10EMartin) @AKanji-WMF Hi Anil, the pretest for India will be 13 June, can this be completed before then? [21:08:15] (03CR) 10Wfan: Explicitly specify insert column order (031 comment) [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/925848 (owner: 10Ejegg) [21:12:27] (03PS3) 10Ejegg: Explicitly specify insert column order [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/925848 [21:12:53] (03CR) 10Ejegg: Explicitly specify insert column order (031 comment) [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/925848 (owner: 10Ejegg) [21:16:53] (03CR) 10Wfan: [C: 03+2] Explicitly specify insert column order [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/925848 (owner: 10Ejegg) [21:18:42] (03Merged) 10jenkins-bot: Explicitly specify insert column order [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/925848 (owner: 10Ejegg) [21:23:41] thanks wfan ! [21:24:56] np [21:39:19] ooh, that rendering bug in the 'recurring charge failed' email looks pretty bad [21:39:34] and eileen's not around for a while [21:40:02] T337919 [21:40:02] T337919: Formatting issue in 'Renew Your Recurring' email sent from Civi - https://phabricator.wikimedia.org/T337919 [21:40:16] says 'formatting issue' but it's basically sending all the HTML escaped it looks like? [21:48:09] heh, eileen, are you stealth-monitoring even what it says you're not in the channel? [21:48:22] lol [21:48:25] i just mentioned your name in conjunction with T337919 [21:48:26] T337919: Formatting issue in 'Renew Your Recurring' email sent from Civi - https://phabricator.wikimedia.org/T337919 [21:48:39] ah no I missed it [21:48:56] from the timing I figure that has to do with the email swapover [21:49:05] err, the rendering changes [21:49:08] ouch - but only that one [21:49:16] ? [21:49:27] I guess just the failed recurring one, yeah [21:50:01] we must handle it slightly differently after we render or something [21:50:04] let's see [21:50:52] ejegg: I think this [21:50:53] $value['msg_text'] = $value['html']; [21:51:05] looks like change to msg_html ? [21:51:19] cos later I see [21:51:20] 'html' => $email['msg_html'] ?? NULL, [21:51:20] 'text' => $email['msg_text'] ?? NULL, [21:51:39] & yeah it sends via civi not our weird thing from the looks [21:52:27] hmm? looking [21:52:45] that's in FailureEmail::render() [21:53:05] & the second is in FailureEmail::send() [22:03:30] Is there any way to test sending that one? [22:03:43] yeah - should be possible from the recurring [22:04:15] I couldn't find the link but maybe was failing at ui [22:04:32] side question - did the guy renew? interested on the impact on renewal rate :-) [22:04:51] cstone: do you have a recurring contribution - it should be there as an action on it? [22:05:15] I have 5 [22:05:17] Hah [22:05:27] The typo method [22:05:43] well it def got his attention [22:06:07] Omg I see it now [22:06:20] I was looking at the individual recurring charge not the big charge [22:06:27] yep [22:07:11] hah gmail still like formats it [22:07:31] btw in answer to ejegg question - I use irc cloud so only get some backscroll without log trawling - in this case none cos my laptop went flat but it seems to give 'some' [23:10:10] RECOVERY - check_disk on civi1002 is OK: DISK OK - free space: /dev 31939 MB (100% inode=99%): /run 6391 MB (99% inode=99%): / 4835 MB (68% inode=86%): /dev/shm 31962 MB (100% inode=99%): /run/lock 5 MB (100% inode=99%): /boot 319 MB (75% inode=99%): /home 12350 MB (99% inode=99%): /tmp 12350 MB (99% inode=99%): /var 9976 MB (37% inode=80%): /srv 109996 MB (37% inode=99%): /var/tmp 422 MB (99% inode=99%): /var/log 23514 MB (88% inode=99% [23:10:10] og/audit 422 MB (99% inode=99%): /run/user/3649 6392 MB (100% inode=99%): /run/user/573 6392 MB (100% inode=99%): https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_disk [23:18:17] eileen: is the failure ty email using a different flow than the normal ty email? [23:18:35] i didnt dive into the code yet I should probably just do that hah [23:18:47] cstone: yep sorry I can put up the patch if you want - I didn't cos I thought ejegg|food would & then I could +2 [23:18:59] ah iwas just trying to get the failure email locally [23:20:35] (03PS1) 10Eileen: Fix for mis-rendered html in failure email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/926039 (https://phabricator.wikimedia.org/T337919) [23:20:52] is there a way you got to see it locally? [23:20:55] cstone: ^^ - it is what I referred to 90 mins ago in irc log [23:20:58] i get the change makes sense but for the future [23:21:03] no - that is just from the code [23:21:22] so does mailcatcher hide the stray tags? [23:23:08] its not even showing up in mailchatcher [23:23:16] sorry so out of the loop on this i see now you updated it when I was gone [23:33:45] (03CR) 10CI reject: [V: 04-1] Fix for mis-rendered html in failure email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/926039 (https://phabricator.wikimedia.org/T337919) (owner: 10Eileen) [23:36:10] (03PS2) 10Eileen: Fix for mis-rendered html in failure email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/926039 (https://phabricator.wikimedia.org/T337919) [23:37:16] (03PS3) 10Eileen: Fix for mis-rendered html in failure email [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/926039 (https://phabricator.wikimedia.org/T337919) [23:43:40] 10Fundraising Sprint Keep ya HEAD(1) up, 10Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10fr-donorservices: Wrap merges in a transaction - https://phabricator.wikimedia.org/T335382 (10AnnWF)