[00:08:12] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1294 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [00:09:09] ACKNOWLEDGEMENT - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1294 Dwisehaupt known lag from large alter. - T374144 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [00:26:12] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2372 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [00:31:10] PROBLEM - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2672 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [00:33:17] ACKNOWLEDGEMENT - check_mysql on frdb1006 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 2672 Dwisehaupt known lag from large alter. - T374144 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [00:36:53] (03PS1) 10Eileen: Update triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089902 [00:38:58] (03Abandoned) 10Eileen: Update triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089902 (owner: 10Eileen) [00:39:30] (03PS1) 10Eileen: Trigger update [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089904 [00:47:24] (03PS2) 10Eileen: Update triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089904 [00:47:45] (03PS1) 10Eileen: Update triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089905 [00:48:56] ok - after a bit of a fight the triggers look ok - do you wanna hit them cstone [01:21:42] (03CR) 10Dwisehaupt: [C:03+2] Update triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089905 (owner: 10Eileen) [01:24:07] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1089917 [01:24:28] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1089917 (owner: 10Eileen) [01:36:15] Eileen I got stuck in traffic did you get everything you needed for the triggers [01:40:10] RECOVERY - check_coworker on civi1002 is OK: PROCS OK: 1 process with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_coworker [01:43:16] RECOVERY - check_mysql on frdb1004 is OK: Uptime: 383063 Threads: 4 Questions: 31922889 Slow queries: 177 Opens: 2407 Open tables: 1054 Queries per second avg: 83.335 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [01:46:14] RECOVERY - check_mysql on frdb1006 is OK: Uptime: 3058041 Threads: 12 Questions: 314108556 Slow queries: 3517059 Opens: 6169 Open tables: 1825 Queries per second avg: 102.715 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1006&service=check_mysql [02:43:58] (03PS1) 10Eileen: Add phone data fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089935 (https://phabricator.wikimedia.org/T376686) [02:44:00] (03PS1) 10Eileen: Move addContributionTrackingIfMissing to queue consumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089936 (https://phabricator.wikimedia.org/T376686) [02:44:01] (03PS1) 10Eileen: Move contribution tracking generate to the consumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089937 [02:44:01] (03PS1) 10Eileen: Simplify contribution tracking lookup [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089938 (https://phabricator.wikimedia.org/T376686) [02:45:56] (03PS2) 10Eileen: Move addContributionTrackingIfMissing to queue consumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089936 (https://phabricator.wikimedia.org/T376686) [02:45:56] (03PS2) 10Eileen: Move contribution tracking generate to the consumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089937 [02:45:57] (03PS2) 10Eileen: Simplify contribution tracking lookup [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089938 (https://phabricator.wikimedia.org/T376686) [02:45:57] (03PS2) 10Eileen: Add phone data fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089935 (https://phabricator.wikimedia.org/T376686) [02:47:04] (03CR) 10CI reject: [V:04-1] Add phone data fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089935 (https://phabricator.wikimedia.org/T376686) (owner: 10Eileen) [02:47:10] (03CR) 10CI reject: [V:04-1] Move addContributionTrackingIfMissing to queue consumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089936 (https://phabricator.wikimedia.org/T376686) (owner: 10Eileen) [02:53:45] 06Fundraising-Backlog, 10fundraising-tech-ops: FR-Tech FY2425Q2 maintenance window (Nov 11-15th, 2024) - https://phabricator.wikimedia.org/T337583#10311001 (10Dwisehaupt) [03:03:38] (03Abandoned) 10Eileen: Update triggers [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089904 (owner: 10Eileen) [03:06:55] (03CR) 10CI reject: [V:04-1] Move addContributionTrackingIfMissing to queue consumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089936 (https://phabricator.wikimedia.org/T376686) (owner: 10Eileen) [03:07:35] (03CR) 10CI reject: [V:04-1] Add phone data fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089935 (https://phabricator.wikimedia.org/T376686) (owner: 10Eileen) [03:08:43] (03PS3) 10Eileen: Move addContributionTrackingIfMissing to queue consumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089936 (https://phabricator.wikimedia.org/T376686) [03:08:43] (03PS3) 10Eileen: Move contribution tracking generate to the consumer [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089937 [03:08:43] (03PS3) 10Eileen: Simplify contribution tracking lookup [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089938 (https://phabricator.wikimedia.org/T376686) [03:08:43] (03PS3) 10Eileen: Add phone data fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089935 (https://phabricator.wikimedia.org/T376686) [03:08:44] (03PS1) 10Eileen: Move another function to only caller [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089942 (https://phabricator.wikimedia.org/T376686) [03:08:46] (03PS1) 10Eileen: qip [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089943 [03:22:50] (03PS2) 10Eileen: Move another function to only caller [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089942 (https://phabricator.wikimedia.org/T376686) [03:22:50] (03PS1) 10Eileen: Clean up existing phone save behaviour [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089945 (https://phabricator.wikimedia.org/T376686) [03:22:52] (03PS1) 10Eileen: Add new phone data fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089946 (https://phabricator.wikimedia.org/T376686) [03:23:10] (03Abandoned) 10Eileen: Add phone data fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089935 (https://phabricator.wikimedia.org/T376686) (owner: 10Eileen) [03:23:18] (03Abandoned) 10Eileen: qip [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089943 (owner: 10Eileen) [05:02:52] (03PS1) 10Eileen: Populate new phone fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089969 (https://phabricator.wikimedia.org/T376686) [05:02:52] (03PS1) 10Eileen: Minor clean up - call function on class rather than deprecated function directly [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089970 [05:02:52] (03PS1) 10Eileen: Simplify function - the current code calls layers of caching [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089971 [05:19:21] (03CR) 10CI reject: [V:04-1] Populate new phone fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089969 (https://phabricator.wikimedia.org/T376686) (owner: 10Eileen) [05:19:53] (03CR) 10CI reject: [V:04-1] Simplify function - the current code calls layers of caching [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089971 (owner: 10Eileen) [05:20:51] (03CR) 10CI reject: [V:04-1] Minor clean up - call function on class rather than deprecated function directly [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089970 (owner: 10Eileen) [05:33:45] (03PS2) 10Eileen: Minor clean up - call function on class rather than deprecated function directly [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089970 [05:33:45] (03PS2) 10Eileen: Simplify function - the current code calls layers of caching [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089971 [05:33:46] (03PS2) 10Eileen: Add new phone data fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089946 (https://phabricator.wikimedia.org/T376686) [05:33:46] (03PS2) 10Eileen: Populate new phone fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089969 (https://phabricator.wikimedia.org/T376686) [05:49:27] (03PS3) 10Eileen: Add new phone data fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089946 (https://phabricator.wikimedia.org/T376686) [05:49:27] (03PS3) 10Eileen: Populate new phone fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089969 (https://phabricator.wikimedia.org/T376686) [06:17:44] (03CR) 10CI reject: [V:04-1] Populate new phone fields [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089969 (https://phabricator.wikimedia.org/T376686) (owner: 10Eileen) [06:23:03] (03CR) 10CI reject: [V:04-1] Localisation updates from https://translatewiki.net. [extensions/DonationInterface] (REL1_43) - 10https://gerrit.wikimedia.org/r/1090096 (owner: 10L10n-bot) [07:55:48] 03Fundraising Sprint: void(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Make cancellation reason available to Acoustic - https://phabricator.wikimedia.org/T379206#10311209 (10MSuijkerbuijk_WMF) @Eileenmcnaughton Jumping in here - I would suggest option 1 as we will exclude some of the options,... [11:58:39] 10fundraising-tech-ops, 06Data-Platform-SRE: [Hive] Investigate packaging, install, security monitoring. - https://phabricator.wikimedia.org/T377635#10311880 (10BTullis) >>! In T377635#10305578, @Jgreen wrote: > We aren't running docker etc so far, but it was pretty straightforward to automate the fetch/verify... [12:37:54] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 3 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10311981 (10cmooney) @Jgreen @Dwisehaupt I think we have broadly two options for how to proceed today: **O... [12:41:00] 03Fundraising Sprint: void(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Make cancellation reason available to Acoustic - https://phabricator.wikimedia.org/T379206#10311984 (10MSuijkerbuijk_WMF) Hi @Eileenmcnaughton I'm looking at my notes (for the call we missed yesterday --open to reschedule a... [14:28:14] PROBLEM - check_mysql on frdb1004 is CRITICAL: SLOW_SLAVE CRITICAL: Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 11390 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [14:31:56] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 3 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10312547 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=fd1b13c3-25ae-42de-a138-bb1a39... [14:33:16] RECOVERY - check_mysql on frdb1004 is OK: Uptime: 429263 Threads: 4 Questions: 33581918 Slow queries: 214 Opens: 2557 Open tables: 1127 Queries per second avg: 78.231 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [14:49:18] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 07Transaction-missing-in-CiviCRM: Transactions not found in Civi for the month of August'24 - https://phabricator.wikimedia.org/T378634#10312631 (10Damilare) Some of this transactions are Fundraiseups, but not all. I checked our logs and noticed we were... [14:54:17] 10fundraising-tech-ops, 06Data-Platform-SRE: [Hive] Investigate packaging, install, security monitoring. - https://phabricator.wikimedia.org/T377635#10312645 (10Jgreen) >>! In T377635#10311880, @BTullis wrote: >>>! In T377635#10305578, @Jgreen wrote: >> We aren't running docker etc so far, but it was pretty st... [15:04:33] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 3 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10312686 (10ops-monitoring-bot) Icinga downtime and Alertmanager silence (ID=6d3e8237-b81b-47ec-a63c-afd9f7... [15:20:44] PROBLEM - Host payments1008 is DOWN: PING CRITICAL - Packet loss = 100% [15:20:58] PROBLEM - Host frdata1002 is DOWN: PING CRITICAL - Packet loss = 100% [15:20:58] PROBLEM - Host fran1001 is DOWN: PING CRITICAL - Packet loss = 100% [15:21:08] PROBLEM - Host frmon1002 is DOWN: PING CRITICAL - Packet loss = 100% [15:21:08] PROBLEM - Host frav1003 is DOWN: PING CRITICAL - Packet loss = 100% [15:21:10] PROBLEM - Host frauth1002 is DOWN: PING CRITICAL - Packet loss = 100% [15:21:10] PROBLEM - Host frban1001 is DOWN: PING CRITICAL - Packet loss = 100% [15:21:10] PROBLEM - Host frqueue1004 is DOWN: PING CRITICAL - Packet loss = 100% [15:21:10] PROBLEM - Host frdb1004 is DOWN: PING CRITICAL - Packet loss = 100% [15:21:10] PROBLEM - Host frdb1003 is DOWN: PING CRITICAL - Packet loss = 100% [15:21:10] PROBLEM - Host frnetmon1001 is DOWN: PING CRITICAL - Packet loss = 100% [15:21:11] PROBLEM - Host frdb1005 is DOWN: PING CRITICAL - Packet loss = 100% [15:21:11] PROBLEM - Host frbast1002 is DOWN: PING CRITICAL - Packet loss = 100% [15:21:12] PROBLEM - Host frpig1002 is DOWN: PING CRITICAL - Packet loss = 100% [15:21:12] PROBLEM - Host frdev1002 is DOWN: PING CRITICAL - Packet loss = 100% [15:21:13] PROBLEM - Host frmx1001 is DOWN: PING CRITICAL - Packet loss = 100% [15:21:13] PROBLEM - Host frpm1002 is DOWN: PING CRITICAL - Packet loss = 100% [15:21:14] PROBLEM - Host frlog1002 is DOWN: PING CRITICAL - Packet loss = 100% [15:21:14] PROBLEM - Host frdb1006 is DOWN: PING CRITICAL - Packet loss = 100% [15:22:28] RECOVERY - Host frqueue1004 is UP: PING OK - Packet loss = 0%, RTA = 0.41 ms [15:22:28] RECOVERY - Host frdb1006 is UP: PING OK - Packet loss = 0%, RTA = 0.45 ms [15:22:30] RECOVERY - Host frdb1005 is UP: PING OK - Packet loss = 0%, RTA = 0.41 ms [15:22:40] RECOVERY - Host frmon1002 is UP: PING OK - Packet loss = 0%, RTA = 0.49 ms [15:22:40] RECOVERY - Host frauth1002 is UP: PING OK - Packet loss = 0%, RTA = 0.40 ms [15:22:40] RECOVERY - Host frpig1002 is UP: PING OK - Packet loss = 0%, RTA = 0.39 ms [15:22:40] RECOVERY - Host frdev1002 is UP: PING OK - Packet loss = 0%, RTA = 0.45 ms [15:22:41] RECOVERY - Host fran1001 is UP: PING OK - Packet loss = 0%, RTA = 0.41 ms [15:22:41] RECOVERY - Host frmx1001 is UP: PING OK - Packet loss = 0%, RTA = 0.41 ms [15:23:12] RECOVERY - Host frnetmon1001 is UP: PING OK - Packet loss = 0%, RTA = 0.45 ms [15:23:14] RECOVERY - Host frdb1004 is UP: PING OK - Packet loss = 0%, RTA = 0.43 ms [15:23:16] RECOVERY - Host frpm1002 is UP: PING OK - Packet loss = 0%, RTA = 0.47 ms [15:23:16] RECOVERY - Host frlog1002 is UP: PING OK - Packet loss = 0%, RTA = 0.55 ms [15:24:33] ^^^ fyi we're experiencing an expected network outage, work in progress [15:25:10] RECOVERY - Host frdata1002 is UP: PING OK - Packet loss = 0%, RTA = 0.52 ms [15:25:12] RECOVERY - Host payments1008 is UP: PING OK - Packet loss = 0%, RTA = 0.44 ms [15:25:18] PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [15:26:08] PROBLEM - check_mysql on frdb2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [15:26:12] PROBLEM - check_mysql on frdb2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [15:26:12] RECOVERY - Host frdb1003 is UP: PING OK - Packet loss = 0%, RTA = 228.10 ms [15:26:12] RECOVERY - Host frbast1002 is UP: PING OK - Packet loss = 0%, RTA = 0.44 ms [15:27:10] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [15:27:14] PROBLEM - check_mysql on frdata2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_mysql [15:27:16] PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [15:27:18] PROBLEM - check_redis on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 424 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 20 hours 57 minutes - replication_delay is 424, memory use is 258.57M (peak 263.50M, 3.29% of max, fragmentation 1.04%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [15:27:18] PROBLEM - check_redis_donor_prefs on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 428 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 20 hours 57 minutes - replication_delay is 428, memory use is 1.09M (peak 1.09M, 0.09% of max, fragmentation 6.67%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [15:27:20] PROBLEM - check_redis on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 426 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 21 hours 6 minutes - replication_delay is 426, memory use is 258.57M (peak 263.52M, 3.29% of max, fragmentation 1.04%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [15:27:20] PROBLEM - check_redis_donor_prefs on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 430 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 21 hours 6 minutes - replication_delay is 430, memory use is 1.09M (peak 1.09M, 0.09% of max, fragmentation 6.73%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [15:30:14] PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [15:30:18] PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [15:31:10] PROBLEM - check_mysql on frdb2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [15:31:12] PROBLEM - check_mysql on frdb2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [15:32:10] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [15:32:14] PROBLEM - check_mysql on frdata2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_mysql [15:32:16] PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [15:32:18] PROBLEM - check_redis on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 724 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 21 hours 2 minutes - replication_delay is 724, memory use is 258.57M (peak 263.50M, 3.29% of max, fragmentation 1.04%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [15:32:18] PROBLEM - check_redis_donor_prefs on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 728 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 21 hours 2 minutes - replication_delay is 728, memory use is 1.09M (peak 1.09M, 0.09% of max, fragmentation 6.67%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [15:32:20] PROBLEM - check_redis on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 726 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 21 hours 11 minutes - replication_delay is 726, memory use is 258.57M (peak 263.52M, 3.29% of max, fragmentation 1.04%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [15:32:22] PROBLEM - check_redis_donor_prefs on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 730 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 21 hours 11 minutes - replication_delay is 730, memory use is 1.09M (peak 1.09M, 0.09% of max, fragmentation 6.73%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [15:33:48] RECOVERY - Host frav1003 is UP: PING OK - Packet loss = 0%, RTA = 0.47 ms [15:35:14] PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [15:35:18] PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [15:36:08] PROBLEM - check_mysql on frdb2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [15:36:14] PROBLEM - check_mysql on frdb2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [15:37:10] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [15:37:16] PROBLEM - check_mysql on frdata2001 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_mysql [15:37:16] PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [15:37:18] PROBLEM - check_redis on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 1024 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 21 hours 7 minutes - replication_delay is 1024, memory use is 258.57M (peak 263.50M, 3.29% of max, fragmentation 1.04%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [15:37:18] PROBLEM - check_redis_donor_prefs on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 1028 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 21 hours 7 minutes - replication_delay is 1028, memory use is 1.09M (peak 1.09M, 0.09% of max, fragmentation 6.67%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [15:37:20] PROBLEM - check_redis on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 1026 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 21 hours 16 minutes - replication_delay is 1026, memory use is 258.57M (peak 263.52M, 3.29% of max, fragmentation 1.04%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [15:37:20] PROBLEM - check_redis_donor_prefs on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 1030 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 21 hours 16 minutes - replication_delay is 1030, memory use is 1.09M (peak 1.09M, 0.09% of max, fragmentation 6.73%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [15:40:14] PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [15:40:18] PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [15:41:08] PROBLEM - check_mysql on frdb2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [15:41:12] PROBLEM - check_mysql on frdb2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [15:42:10] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [15:42:14] RECOVERY - check_mysql on frdata2001 is OK: Uptime: 76197 Threads: 3 Questions: 32306 Slow queries: 0 Opens: 94 Open tables: 88 Queries per second avg: 0.423 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2001&service=check_mysql [15:42:16] PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [15:42:18] RECOVERY - check_redis on frqueue2002 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 21 hours 12 minutes - replication_delay is 1, memory use is 258.59M (peak 263.50M, 3.29% of max, fragmentation 1.04%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [15:42:18] RECOVERY - check_redis_donor_prefs on frqueue2002 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 21 hours 12 minutes - replication_delay is 5, memory use is 1.09M (peak 1.09M, 0.09% of max, fragmentation 6.66%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [15:42:20] RECOVERY - check_redis on frqueue2003 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 13 keys, up 21 hours 21 minutes - replication_delay is 3, memory use is 258.57M (peak 263.52M, 3.29% of max, fragmentation 1.04%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [15:42:20] RECOVERY - check_redis_donor_prefs on frqueue2003 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 21 hours 21 minutes - replication_delay is 7, memory use is 1.09M (peak 1.09M, 0.09% of max, fragmentation 6.95%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [15:45:14] RECOVERY - check_mysql on payments2006 is OK: Uptime: 57948 Threads: 6 Questions: 299933 Slow queries: 0 Opens: 144 Open tables: 138 Queries per second avg: 5.175 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [15:45:18] RECOVERY - check_mysql on payments2005 is OK: Uptime: 58242 Threads: 5 Questions: 767627 Slow queries: 0 Opens: 142 Open tables: 136 Queries per second avg: 13.179 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [15:46:10] RECOVERY - check_mysql on frdb2004 is OK: Uptime: 75660 Threads: 4 Questions: 2190081 Slow queries: 41 Opens: 2528 Open tables: 1138 Queries per second avg: 28.946 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [15:46:14] RECOVERY - check_mysql on frdb2005 is OK: Uptime: 75283 Threads: 4 Questions: 2182969 Slow queries: 41 Opens: 2517 Open tables: 1122 Queries per second avg: 28.996 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [15:47:10] RECOVERY - check_mysql on frdb2003 is OK: Uptime: 434224 Threads: 7 Questions: 312708952 Slow queries: 79 Opens: 7079 Open tables: 1802 Queries per second avg: 720.155 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [15:47:16] RECOVERY - check_mysql on payments2004 is OK: Uptime: 58362 Threads: 6 Questions: 302103 Slow queries: 0 Opens: 144 Open tables: 138 Queries per second avg: 5.176 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [15:57:23] (03CR) 10Abijeet Patro: "Looks good, but can't merge due to permissions" [extensions/DonationInterface] (REL1_43) - 10https://gerrit.wikimedia.org/r/1090096 (owner: 10L10n-bot) [16:07:15] RECOVERY - Host frban1001 is UP: PING OK - Packet loss = 0%, RTA = 9.56 ms [16:23:58] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 3 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10313129 (10cmooney) Migration work is now complete, bastion and all hosts are reachable again following th... [16:24:15] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: "Zendesk TIckets" tab not showing related tickets on for some donors - https://phabricator.wikimedia.org/T374154#10313147 (10jgleeson) a:03jgleeson [18:11:36] 06Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Tech-Analytics: swap analytics db origin from frdb1003 to frdb2003 - https://phabricator.wikimedia.org/T378859#10313873 (10Dwisehaupt) [18:30:26] 06Fundraising-Backlog, 10fundraising-tech-ops, 10FR-Tech-Analytics: swap analytics db origin from frdb1003 to frdb2003 - https://phabricator.wikimedia.org/T378859#10314012 (10Dwisehaupt) [18:45:12] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-donorservices: Possible issue with the civi snooze - https://phabricator.wikimedia.org/T376959#10314069 (10SHust) @bsisolak This is what I see on our end for CID 64598559: {F57696438} [19:07:13] PROBLEM - check_mysql on frdb2003 is CRITICAL: Cant connect to local server through socket /var/run/mysqld/mysqld.sock (2) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [19:08:15] PROBLEM - check_mysql on frdb1004 is CRITICAL: Cant connect to local server through socket /var/run/mysqld/mysqld.sock (2) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [19:12:13] PROBLEM - check_mysql on frdb2003 is CRITICAL: Cant connect to local server through socket /var/run/mysqld/mysqld.sock (2) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [19:13:13] PROBLEM - check_mysql on frdb1004 is CRITICAL: Cant connect to local server through socket /var/run/mysqld/mysqld.sock (2) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [19:17:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Cant connect to local server through socket /var/run/mysqld/mysqld.sock (2) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [19:18:15] PROBLEM - check_mysql on frdb1004 is CRITICAL: Cant connect to local server through socket /var/run/mysqld/mysqld.sock (2) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [19:22:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Cant connect to local server through socket /var/run/mysqld/mysqld.sock (2) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [19:23:15] PROBLEM - check_mysql on frdb1004 is CRITICAL: Cant connect to local server through socket /var/run/mysqld/mysqld.sock (2) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1004&service=check_mysql [19:23:45] sorry about those. i thought i had a longer downtime set for those two. [20:12:15] ejegg: did you see the link [20:32:23] alert spam! :) [20:42:04] 06Fundraising-Backlog, 10FR-donorservices: Donation history in a single email feature request - https://phabricator.wikimedia.org/T368668#10314360 (10Eileenmcnaughton) I think this was resolved in https://phabricator.wikimedia.org/T299096 [20:50:30] 03Fundraising Sprint: void(), 06Fundraising-Backlog, 13Patch-For-Review: Acoustic SMS stage 5 - update queue consumer support for SMS RML - https://phabricator.wikimedia.org/T376686#10314371 (10Eileenmcnaughton) After discussion with @bsisolak I think we need to create a new table to store the consent data s... [20:53:57] 06Fundraising-Backlog: Acoustic SMS: Domain needed for short links - https://phabricator.wikimedia.org/T379318#10314393 (10AKanji-WMF) @Ejegg to confirm why current link shortener doesn't allow parameters; @greg to initiate process of securing new domain with Production SRE. [20:58:14] 03Fundraising Sprint: void(), 06Fundraising-Backlog: Update donation interface such that recipient_id reaches donationn queue - https://phabricator.wikimedia.org/T379680 (10Eileenmcnaughton) 03NEW [20:58:18] 06Fundraising-Backlog: Test recipient_ID in URL - https://phabricator.wikimedia.org/T379310#10314428 (10AKanji-WMF) 05Open→03Declined killed in favour of {T376686} [21:10:42] 06Fundraising-Backlog, 10MediaWiki-extensions-WikimediaMaintenance, 10WikimediaMessages, 13Patch-For-Review: Donate sidebar link consistency (sitesupport-url) - https://phabricator.wikimedia.org/T379205#10314513 (10AKanji-WMF) cc: @Pcoombe [21:18:51] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Individual Matching Import Issue - Recurring - https://phabricator.wikimedia.org/T379171#10314536 (10AKanji-WMF) [21:22:26] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Temporary accounts: [Temporary Accounts] Update CentralNotice extension to support Temporary Accounts - https://phabricator.wikimedia.org/T374437#10314557 (10AKanji-WMF) @AKanji-WMF to check in with central notice users and @Sheilakaruku [21:22:52] 06Fundraising-Backlog: Auto Rescue the initial payment for ach and sepa - https://phabricator.wikimedia.org/T379129#10314570 (10AKanji-WMF) Revisit this in Q3 [21:27:36] 06Fundraising-Backlog, 10FR-donorservices: Civi merge is not saving the state when a mailing address is present. - https://phabricator.wikimedia.org/T379684 (10SHust) 03NEW [21:31:53] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-donorservices: Civi merge is not saving the state when a mailing address is present. - https://phabricator.wikimedia.org/T379684#10314624 (10AKanji-WMF) [22:07:38] aha, I think I found the secret config file that controls which DonationInterface messages get translated [22:09:06] cstone: I think it's here: https://phabricator.wikimedia.org/diffusion/GTWN/browse/master/groups/MediaWiki/mediawiki-extensions.txt$1133 [22:19:02] Oooohhh [22:29:14] 03Fundraising Sprint: void(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Make cancellation reason available to Acoustic - https://phabricator.wikimedia.org/T379206#10314767 (10Eileenmcnaughton) Note when exporting - lcase + underscore the options [22:38:38] I'll try submitting a patch to add the email prefs folders [22:47:41] 03Fundraising Sprint: void(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Make cancellation reason available to Acoustic - https://phabricator.wikimedia.org/T379206#10314815 (10Eileenmcnaughton) OK - so @NNgu-WMF has created a field most_recent_cancel_reason with the options other_and_unspecifi... [23:08:33] 03Fundraising Sprint: void(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 07FR-Imports: Engage import_Matching - Ind contact import *updated - https://phabricator.wikimedia.org/T377605#10314924 (10MDemosWMF) @Eileenmcnaughton I was able to take a look at these and see there was an issue - eithe... [23:18:17] 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog: Individual Matching Import Issue - Recurring - https://phabricator.wikimedia.org/T379171#10314981 (10MDemosWMF) I think the issue in T377605 is related to this - since that one was solved this one should be too. [23:36:09] PROBLEM - check_mysql on frdb2004 is CRITICAL: Cant connect to local server through socket /var/run/mysqld/mysqld.sock (2) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [23:43:48] 06Fundraising-Backlog: Populating both_funds_latest_donation_source field - https://phabricator.wikimedia.org/T379700#10315061 (10Wargo)