[05:09:11] FIRING: [2x] PfwCoreBGPDown: Fundraising Firewall core BGP session down between pfw1-codfw and (null) (10.195.0.248) - group VPN - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DPfwCoreBGPDown [05:12:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [05:12:11] PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql [05:12:15] PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [05:12:19] PROBLEM - check_redis on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 342 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 19 keys, up 58 days 3 hours - replication_delay is 342, memory use is 258.71M (peak 627.14M, 3.38% of max, fragmentation 1.07%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [05:12:19] PROBLEM - check_redis on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 344 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 19 keys, up 58 days 3 hours - replication_delay is 344, memory use is 258.71M (peak 627.14M, 3.38% of max, fragmentation 1.07%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [05:12:19] PROBLEM - check_redis_donor_prefs on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 348 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 58 days 3 hours - replication_delay is 348, memory use is 2.20M (peak 2.24M, 0.11% of max, fragmentation 3.99%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [05:12:21] PROBLEM - check_redis_donor_prefs on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 350 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 58 days 3 hours - replication_delay is 350, memory use is 2.18M (peak 2.24M, 0.11% of max, fragmentation 4.03%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [05:13:13] PROBLEM - check_mysql on frdb1007 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql [05:15:13] PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [05:15:17] PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [05:16:09] PROBLEM - check_mysql on frdb2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [05:16:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [05:17:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [05:17:11] PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql [05:17:13] PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [05:17:17] PROBLEM - check_redis on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 642 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 19 keys, up 58 days 3 hours - replication_delay is 642, memory use is 258.71M (peak 627.14M, 3.38% of max, fragmentation 1.07%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [05:17:19] PROBLEM - check_redis_donor_prefs on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 648 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 58 days 3 hours - replication_delay is 648, memory use is 2.20M (peak 2.24M, 0.11% of max, fragmentation 3.99%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [05:17:19] PROBLEM - check_redis on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 644 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 19 keys, up 58 days 3 hours - replication_delay is 644, memory use is 258.71M (peak 627.14M, 3.38% of max, fragmentation 1.07%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [05:17:21] PROBLEM - check_redis_donor_prefs on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 650 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 58 days 3 hours - replication_delay is 650, memory use is 2.18M (peak 2.24M, 0.11% of max, fragmentation 4.03%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [05:18:11] PROBLEM - check_mysql on frdb1007 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql [05:20:13] PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [05:20:17] PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [05:21:09] PROBLEM - check_mysql on frdb2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [05:21:13] PROBLEM - check_mysql on frdb2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [05:22:09] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [05:22:11] PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql [05:22:13] PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [05:22:17] PROBLEM - check_redis on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 942 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 19 keys, up 58 days 3 hours - replication_delay is 942, memory use is 258.71M (peak 627.14M, 3.38% of max, fragmentation 1.07%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [05:22:19] PROBLEM - check_redis_donor_prefs on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 948 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 58 days 3 hours - replication_delay is 948, memory use is 2.20M (peak 2.24M, 0.11% of max, fragmentation 3.99%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [05:22:19] PROBLEM - check_redis on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 945 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 19 keys, up 58 days 3 hours - replication_delay is 945, memory use is 258.71M (peak 627.14M, 3.38% of max, fragmentation 1.07%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [05:22:21] PROBLEM - check_redis_donor_prefs on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 951 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 58 days 3 hours - replication_delay is 951, memory use is 2.18M (peak 2.24M, 0.11% of max, fragmentation 4.03%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [05:23:11] PROBLEM - check_mysql on frdb1007 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql [05:25:13] PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [05:25:17] PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [05:26:09] PROBLEM - check_mysql on frdb2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [05:26:15] PROBLEM - check_mysql on frdb2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [05:27:11] PROBLEM - check_mysql on frdb2003 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [05:27:11] PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql [05:27:15] PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [05:27:17] PROBLEM - check_redis on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 1242 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 19 keys, up 58 days 3 hours - replication_delay is 1242, memory use is 258.71M (peak 627.14M, 3.38% of max, fragmentation 1.07%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [05:27:17] PROBLEM - check_redis_donor_prefs on frqueue2002 is CRITICAL: CRITICAL: replication_delay is 1248 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 58 days 3 hours - replication_delay is 1248, memory use is 2.20M (peak 2.24M, 0.11% of max, fragmentation 3.99%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [05:27:19] PROBLEM - check_redis on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 1244 300 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 19 keys, up 58 days 3 hours - replication_delay is 1244, memory use is 258.71M (peak 627.14M, 3.38% of max, fragmentation 1.07%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [05:27:21] PROBLEM - check_redis_donor_prefs on frqueue2003 is CRITICAL: CRITICAL: replication_delay is 1250 300 - REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 58 days 3 hours - replication_delay is 1250, memory use is 2.18M (peak 2.24M, 0.11% of max, fragmentation 4.03%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [05:28:13] PROBLEM - check_mysql on frdb1007 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql [05:29:11] FIRING: [2x] PfwCoreBGPDown: Fundraising Firewall core BGP session down between pfw1-codfw and (null) (10.195.0.248) - group VPN - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DPfwCoreBGPDown [05:30:13] RECOVERY - check_mysql on payments2006 is OK: Uptime: 5021869 Threads: 6 Questions: 22645877 Slow queries: 0 Opens: 149 Open tables: 143 Queries per second avg: 4.509 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql [05:30:17] RECOVERY - check_mysql on payments2005 is OK: Uptime: 5021883 Threads: 6 Questions: 22609349 Slow queries: 0 Opens: 150 Open tables: 144 Queries per second avg: 4.502 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql [05:31:09] RECOVERY - check_mysql on frdb2004 is OK: Uptime: 5050012 Threads: 4 Questions: 932293410 Slow queries: 2836 Opens: 838540 Open tables: 1623 Queries per second avg: 184.612 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2004&service=check_mysql [05:31:13] RECOVERY - check_mysql on frdb2005 is OK: Uptime: 5050173 Threads: 4 Questions: 932277056 Slow queries: 2874 Opens: 838475 Open tables: 1571 Queries per second avg: 184.602 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2005&service=check_mysql [05:32:11] RECOVERY - check_mysql on frdb2003 is OK: Uptime: 4422405 Threads: 137 Questions: 1642501326 Slow queries: 23544 Opens: 842180 Open tables: 2917 Queries per second avg: 371.404 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 1441 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb2003&service=check_mysql [05:32:11] RECOVERY - check_mysql on frdata2002 is OK: Uptime: 5042398 Threads: 3 Questions: 2309192 Slow queries: 0 Opens: 135 Open tables: 129 Queries per second avg: 0.457 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql [05:32:15] RECOVERY - check_mysql on payments2004 is OK: Uptime: 5023348 Threads: 6 Questions: 22622198 Slow queries: 0 Opens: 148 Open tables: 142 Queries per second avg: 4.503 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql [05:32:17] RECOVERY - check_redis on frqueue2002 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 16 keys, up 58 days 3 hours - replication_delay is 1, memory use is 258.73M (peak 627.14M, 3.38% of max, fragmentation 1.07%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis [05:32:19] RECOVERY - check_redis_donor_prefs on frqueue2002 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 58 days 3 hours - replication_delay is 2, memory use is 2.18M (peak 2.24M, 0.10% of max, fragmentation 4.01%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis_donor_prefs [05:32:19] RECOVERY - check_redis on frqueue2003 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 16 keys, up 58 days 3 hours - replication_delay is 0, memory use is 258.76M (peak 627.14M, 3.38% of max, fragmentation 1.07%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis [05:32:21] RECOVERY - check_redis_donor_prefs on frqueue2003 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6380 has 0 databases (), up 58 days 3 hours - replication_delay is 5, memory use is 2.18M (peak 2.24M, 0.10% of max, fragmentation 4.01%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2003&service=check_redis_donor_prefs [05:33:13] RECOVERY - check_mysql on frdb1007 is OK: Uptime: 4517571 Threads: 4 Questions: 8575823177 Slow queries: 975 Opens: 840470 Open tables: 2297 Queries per second avg: 1898.326 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 5873 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdb1007&service=check_mysql [05:34:16] RESOLVED: [2x] PfwCoreBGPDown: Fundraising Firewall core BGP session down between pfw1-codfw and (null) (10.195.0.248) - group VPN - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://alerts.wikimedia.org/?q=alertname%3DPfwCoreBGPDown [19:46:11] 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 10FR-WMF-Audit: Channel sometimes empty for extra refunds / adjustments / chargebacks - https://phabricator.wikimedia.org/T414286 (10Eileenmcnaughton) 03NEW [21:18:30] (03PS5) 10Eileen: First pass on Paypal audit process [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1224180 (https://phabricator.wikimedia.org/T409362) [21:18:30] (03PS1) 10Eileen: Add STL support [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1225167 [21:18:31] (03PS1) 10Eileen: Fix save settlement record [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1225168 [21:18:50] (03PS1) 10Eileen: Add Other Online as a default channel [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1225169 (https://phabricator.wikimedia.org/T414286)