[00:00:12] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[00:00:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[00:00:36] <icinga-wm>	 PROBLEM - Host frqueue1003 is DOWN: PING CRITICAL - Packet loss = 100%
[00:00:48] <icinga-wm>	 PROBLEM - Host frban1002 is DOWN: PING CRITICAL - Packet loss = 100%
[00:01:06] <icinga-wm>	 PROBLEM - Host frnetmon1002 is DOWN: PING CRITICAL - Packet loss = 100%
[00:01:06] <icinga-wm>	 PROBLEM - Host fran1002 is DOWN: PING CRITICAL - Packet loss = 100%
[00:01:24] <icinga-wm>	 PROBLEM - Host frpig1002 is DOWN: PING CRITICAL - Packet loss = 100%
[00:02:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[00:02:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[00:04:28] <icinga-wm>	 PROBLEM - Host payments1006 is DOWN: PING CRITICAL - Packet loss = 100%
[00:04:44] <icinga-wm>	 PROBLEM - Host frqueue1004 is DOWN: PING CRITICAL - Packet loss = 100%
[00:04:54] <icinga-wm>	 PROBLEM - Host frav1003 is DOWN: PING CRITICAL - Packet loss = 100%
[00:05:10] <icinga-wm>	 PROBLEM - Host pay-lb1002 is DOWN: PING CRITICAL - Packet loss = 100%
[00:05:10] <icinga-wm>	 PROBLEM - Host payments1005 is DOWN: PING CRITICAL - Packet loss = 100%
[00:05:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[00:05:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[00:07:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[00:07:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[00:08:38] <jinxer-wm>	 FIRING: [2x] GnmiTargetDown: fasw2-e16a-eqiad is unreachable through gNMI - https://wikitech.wikimedia.org/wiki/Network_telemetry#Troubleshooting - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic - https://alerts.wikimedia.org/?q=alertname%3DGnmiTargetDown
[00:10:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[00:10:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[00:12:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[00:12:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[00:15:12] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[00:15:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[00:17:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[00:17:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[00:20:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[00:20:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[00:22:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[00:22:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[00:25:12] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[00:25:16] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[00:27:10] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[00:27:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[00:30:12] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[00:30:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[00:32:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[00:32:16] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[00:35:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[00:35:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[00:37:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[00:37:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[00:40:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[00:40:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[00:42:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[00:42:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[00:45:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[00:45:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[00:47:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[00:47:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[00:50:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[00:50:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[00:52:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[00:52:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[00:55:12] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[00:55:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[00:57:10] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[00:57:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[01:00:12] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[01:00:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[01:02:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[01:02:16] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[01:05:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[01:05:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[01:07:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[01:07:16] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[01:10:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[01:10:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[01:12:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[01:12:12] <icinga-wm>	 RECOVERY - Host frdev1002 is UP: PING OK - Packet loss = 0%, RTA = 0.56 ms
[01:12:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[01:15:12] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[01:15:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[01:17:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[01:17:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[01:20:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[01:20:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[01:22:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[01:22:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[01:23:52] <icinga-wm>	 RECOVERY - Host payments1008 is UP: PING OK - Packet loss = 0%, RTA = 0.40 ms
[01:25:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[01:25:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[01:27:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[01:27:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[01:30:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[01:30:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[01:32:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[01:32:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[01:35:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[01:35:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[01:36:17] <icinga-wm>	 RECOVERY - Host frauth1002 is UP: PING OK - Packet loss = 0%, RTA = 0.57 ms
[01:37:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[01:37:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[01:40:12] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[01:40:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[01:42:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[01:42:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[01:45:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[01:45:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[01:47:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[01:47:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[01:50:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[01:50:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[01:52:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[01:52:16] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[01:55:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[01:55:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[01:57:12] <icinga-wm>	 PROBLEM - check_mysql on frdata2002 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[01:57:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[02:00:12] <icinga-wm>	 RECOVERY - Host frdata1002 is UP: PING OK - Packet loss = 0%, RTA = 0.42 ms
[02:00:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[02:00:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[02:02:10] <icinga-wm>	 RECOVERY - check_mysql on frdata2002 is OK: Uptime: 1905077 Threads: 6 Questions: 870525 Slow queries: 0 Opens: 149 Open tables: 143 Queries per second avg: 0.456 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frdata2002&service=check_mysql
[02:02:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[02:05:12] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[02:05:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[02:07:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[02:10:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[02:10:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[02:12:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[02:13:14] <icinga-wm>	 RECOVERY - Host frqueue1003 is UP: PING OK - Packet loss = 0%, RTA = 0.61 ms
[02:15:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[02:15:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[02:17:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[02:19:40] <icinga-wm>	 RECOVERY - Host frqueue1004 is UP: PING OK - Packet loss = 0%, RTA = 0.47 ms
[02:20:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[02:20:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[02:22:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[02:25:12] <icinga-wm>	 PROBLEM - check_mysql on payments1008 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_mysql
[02:25:12] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[02:25:16] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[02:27:16] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[02:30:12] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[02:30:14] <icinga-wm>	 PROBLEM - check_mysql on payments1008 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_mysql
[02:30:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[02:32:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[02:35:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[02:35:14] <icinga-wm>	 PROBLEM - check_mysql on payments1008 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_mysql
[02:35:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[02:37:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[02:40:00] <wikibugs>	 (03PS4) 10Eileen: Add settled fields spec provider [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1236871 (https://phabricator.wikimedia.org/T412172)
[02:40:00] <wikibugs>	 (03PS1) 10Eileen: Add updated Accounting Searches [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237370
[02:40:12] <icinga-wm>	 PROBLEM - check_mysql on payments1008 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_mysql
[02:40:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[02:40:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[02:42:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[02:44:01] <wikibugs>	 (03PS1) 10Wfan: no longer have drush use cv instead [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1237371
[02:44:33] <wikibugs>	 (03CR) 10Eileen: Add settled fields spec provider (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1236871 (https://phabricator.wikimedia.org/T412172) (owner: 10Eileen)
[02:45:12] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[02:45:14] <icinga-wm>	 PROBLEM - check_mysql on payments1008 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_mysql
[02:45:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[02:47:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[02:48:26] <wikibugs>	 (03PS1) 10Wfan: add script to consume recurring locally [wikimedia/fundraising/dev] - 10https://gerrit.wikimedia.org/r/1237372
[02:50:14] <icinga-wm>	 PROBLEM - check_mysql on payments1008 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_mysql
[02:50:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[02:50:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[02:52:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[02:55:12] <icinga-wm>	 PROBLEM - check_mysql on payments1008 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_mysql
[02:55:12] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[02:55:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[02:57:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[03:00:12] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[03:00:14] <icinga-wm>	 PROBLEM - check_mysql on payments1008 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_mysql
[03:00:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[03:02:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[03:05:12] <icinga-wm>	 PROBLEM - check_mysql on payments1008 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_mysql
[03:05:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[03:05:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[03:07:12] <icinga-wm>	 RECOVERY - Host frmon1002 is UP: PING OK - Packet loss = 0%, RTA = 0.35 ms
[03:07:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[03:10:12] <icinga-wm>	 PROBLEM - check_mysql on payments1008 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_mysql
[03:10:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[03:10:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[03:12:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[03:12:16] <icinga-wm>	 PROBLEM - check_redis on frqueue2002 is CRITICAL: CRITICAL: connected_slaves is 0 1 - REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 26 keys, up 22 days 4 hours - memory use is 265.66M (peak 706.03M, 3.47% of max, fragmentation 1.07%), donations is 291, jobs is 0, jobs-adyen is 33, jobs-dlocal is 0, jobs-gravy is 733, jobs-paypal is 400, payments-antifraud is 1243, payments-init is 557, pending is 1038, recurring is 273, refun
[03:12:16] <icinga-wm>	 nsubscribe is 2 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis
[03:15:12] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[03:15:16] <icinga-wm>	 PROBLEM - check_mysql on payments1008 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_mysql
[03:15:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[03:17:08] <icinga-wm>	 RECOVERY - Host frav1003 is UP: PING WARNING - Packet loss = 66%, RTA = 0.47 ms
[03:17:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[03:17:18] <icinga-wm>	 RECOVERY - check_redis on frqueue2002 is OK: OK: REDIS 7.0.15 on 127.0.0.1:6379 has 1 databases (db0) with 26 keys, up 22 days 4 hours - replication_delay is 2, memory use is 265.66M (peak 706.03M, 3.47% of max, fragmentation 1.07%) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frqueue2002&service=check_redis
[03:20:12] <icinga-wm>	 PROBLEM - check_mysql on payments1008 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_mysql
[03:20:14] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[03:20:18] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[03:22:14] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[03:23:25] <icinga-wm>	 ACKNOWLEDGEMENT - Host pay-lb1001 is DOWN: PING CRITICAL - Packet loss = 100% Dwisehaupt known - host relocation
[03:23:26] <icinga-wm>	 ACKNOWLEDGEMENT - Host pay-lb1002 is DOWN: PING CRITICAL - Packet loss = 100% Dwisehaupt known - host relocation
[03:23:27] <icinga-wm>	 ACKNOWLEDGEMENT - Host payments1005 is DOWN: PING CRITICAL - Packet loss = 100% Dwisehaupt known - host relocation
[03:23:28] <icinga-wm>	 ACKNOWLEDGEMENT - Host payments1006 is DOWN: PING CRITICAL - Packet loss = 100% Dwisehaupt known - host relocation
[03:23:29] <icinga-wm>	 ACKNOWLEDGEMENT - Host payments1007 is DOWN: PING CRITICAL - Packet loss = 100% Dwisehaupt known - host relocation
[03:25:11] <icinga-wm>	 PROBLEM - check_mysql on payments1008 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_mysql
[03:25:13] <icinga-wm>	 PROBLEM - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[03:25:17] <icinga-wm>	 PROBLEM - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[03:26:11] <icinga-wm>	 RECOVERY - Host frpig1002 is UP: PING OK - Packet loss = 0%, RTA = 0.50 ms
[03:27:15] <icinga-wm>	 PROBLEM - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[03:28:14] <icinga-wm>	 ACKNOWLEDGEMENT - check_mysql on payments1008 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) Dwisehaupt known. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_mysql
[03:28:14] <icinga-wm>	 ACKNOWLEDGEMENT - check_mysql on payments2004 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) Dwisehaupt known. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[03:28:14] <icinga-wm>	 ACKNOWLEDGEMENT - check_mysql on payments2005 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) Dwisehaupt known. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[03:28:14] <icinga-wm>	 ACKNOWLEDGEMENT - check_mysql on payments2006 is CRITICAL: Slave IO: Connecting Slave SQL: Yes Seconds Behind Master: (null) Dwisehaupt known. https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[03:29:18] <wikibugs>	 (03CR) 10Wfan: [C:03+2] "Looks pretty cool to have a link there! https://wmf.localhost:32353/civicrm/contribution/settled#?contribution_settlement.settlement_batch" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237370 (owner: 10Eileen)
[03:29:55] <eileen>	 thanks wfan - the one before it is needed to make it merge
[03:30:10] <wikibugs>	 (03CR) 10Wfan: [C:03+2] Add settled fields spec provider [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1236871 (https://phabricator.wikimedia.org/T412172) (owner: 10Eileen)
[03:30:20] <eileen>	 double thanks wfan 
[03:30:43] <wfan>	 just checking since the civi link shared on the first one not working due to maintenance, thanks for this fantastic ui update haha
[03:31:55] <eileen>	 wfan: yeah - it will help me figure out why some batches aren't matching - I've found it's pretty quick to sort the csv by amount & put a running column on the csv & then use that to play with filters on the search until I find the missing or extra transaction but that total was messing with me
[03:32:28] <wfan>	 Looks super cool 🆒
[03:52:20] <wikibugs>	 (03Merged) 10jenkins-bot: Add settled fields spec provider [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1236871 (https://phabricator.wikimedia.org/T412172) (owner: 10Eileen)
[03:52:41] <wikibugs>	 (03Merged) 10jenkins-bot: Add updated Accounting Searches [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237370 (owner: 10Eileen)
[04:08:38] <jinxer-wm>	 FIRING: [2x] GnmiTargetDown: fasw2-e16a-eqiad is unreachable through gNMI - https://wikitech.wikimedia.org/wiki/Network_telemetry#Troubleshooting - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic - https://alerts.wikimedia.org/?q=alertname%3DGnmiTargetDown
[04:11:05] <icinga-wm>	 RECOVERY - Host frbast1002 is UP: PING OK - Packet loss = 0%, RTA = 0.36 ms
[04:25:09] <icinga-wm>	 RECOVERY - Host frnetmon1002 is UP: PING OK - Packet loss = 0%, RTA = 0.42 ms
[04:29:19] <icinga-wm>	 RECOVERY - Host frmx1001 is UP: PING OK - Packet loss = 0%, RTA = 0.39 ms
[04:41:01] <icinga-wm>	 RECOVERY - Host frdb1004 is UP: PING OK - Packet loss = 0%, RTA = 0.45 ms
[04:45:11] <icinga-wm>	 RECOVERY - Host payments1007 is UP: PING OK - Packet loss = 0%, RTA = 0.52 ms
[04:49:56] <wikibugs>	 (03PS1) 10Eileen: Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1237374
[04:54:13] <icinga-wm>	 ACKNOWLEDGEMENT - Host fran1002 is DOWN: PING CRITICAL - Packet loss = 100% Dwisehaupt known - machine moves.
[04:54:13] <icinga-wm>	 ACKNOWLEDGEMENT - Host frban1002 is DOWN: PING CRITICAL - Packet loss = 100% Dwisehaupt known - machine moves.
[04:54:14] <icinga-wm>	 ACKNOWLEDGEMENT - Host frdb1006 is DOWN: PING CRITICAL - Packet loss = 100% Dwisehaupt known - machine moves.
[04:54:15] <icinga-wm>	 ACKNOWLEDGEMENT - Host frdb1007 is DOWN: PING CRITICAL - Packet loss = 100% Dwisehaupt known - machine moves.
[06:21:47] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for jmando - https://phabricator.wikimedia.org/T415402#11590303 (10Dwisehaupt) certificate renewed and sent via email. password sent via sms.
[06:23:18] <wikibugs>	 (03CR) 10Eileen: [C:03+2] Merge branch 'master' of https://gerrit.wikimedia.org/r/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1237374 (owner: 10Eileen)
[06:24:50] <eileen>	 !log  civicrm upgraded from b63d7146 to 06842fbf
[06:24:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL
[06:25:11] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for ccogdill - https://phabricator.wikimedia.org/T415401#11590305 (10Dwisehaupt) certificate renewed and sent via email. password sent via sms.
[08:08:38] <jinxer-wm>	 FIRING: [2x] GnmiTargetDown: fasw2-e16a-eqiad is unreachable through gNMI - https://wikitech.wikimedia.org/wiki/Network_telemetry#Troubleshooting - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic - https://alerts.wikimedia.org/?q=alertname%3DGnmiTargetDown
[12:08:38] <jinxer-wm>	 FIRING: [2x] GnmiTargetDown: fasw2-e16a-eqiad is unreachable through gNMI - https://wikitech.wikimedia.org/wiki/Network_telemetry#Troubleshooting - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic - https://alerts.wikimedia.org/?q=alertname%3DGnmiTargetDown
[12:31:30] <wikibugs>	 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad: Fr-tech expansion - https://phabricator.wikimedia.org/T403035#11591130 (10cmooney) @VRiley-WMF great work on these moves yesterday, I don't think it could have gone much smoother tbh :)  In terms of the cable label...
[12:41:08] <icinga-wm>	 RECOVERY - Host payments1006 is UP: PING OK - Packet loss = 0%, RTA = 0.53 ms
[12:53:23] <jinxer-wm>	 RESOLVED: [2x] GnmiTargetDown: fasw2-e16a-eqiad is unreachable through gNMI - https://wikitech.wikimedia.org/wiki/Network_telemetry#Troubleshooting - https://grafana.wikimedia.org/d/eab73c60-a402-4f9b-a4a7-ea489b374458/gnmic - https://alerts.wikimedia.org/?q=alertname%3DGnmiTargetDown
[12:58:40] <wikibugs>	 (03PS2) 10Damilare Adedoyin: Add recently cancelled contributions to DonorSummary [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237290 (https://phabricator.wikimedia.org/T410409)
[12:59:01] <wikibugs>	 (03PS3) 10Damilare Adedoyin: Add recently cancelled contributions to DonorSummary [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237290 (https://phabricator.wikimedia.org/T410409)
[13:00:15] <wikibugs>	 (03PS2) 10Damilare Adedoyin: WIP: Add contribution status to donation history [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237316 (https://phabricator.wikimedia.org/T416370)
[14:01:53] <wikibugs>	 (03CR) 10Jgleeson: "Thanks Dami. I left a couple of questions inline." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237290 (https://phabricator.wikimedia.org/T410409) (owner: 10Damilare Adedoyin)
[14:07:06] <wikibugs>	 (03CR) 10Jgleeson: "Ha, so I just asked Claude and he said:" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237290 (https://phabricator.wikimedia.org/T410409) (owner: 10Damilare Adedoyin)
[14:24:15] <wikibugs>	 (03CR) 10Damilare Adedoyin: "Thanks for the CR jgleeson, I responded inline. Yh I found the filter in SK, its quite handy. You can test locally on the API." [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237290 (https://phabricator.wikimedia.org/T410409) (owner: 10Damilare Adedoyin)
[14:28:10] <wikibugs>	 (03PS2) 10Jgleeson: Add orchestrator|backend_processor layer to Gravy API timing logs [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1236844 (https://phabricator.wikimedia.org/T416525)
[14:28:33] <wikibugs>	 (03CR) 10Jgleeson: "When testing this locally with a Gravy Adyen payment, I get" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1236844 (https://phabricator.wikimedia.org/T416525) (owner: 10Jgleeson)
[14:28:34] <damilare>	 That's quite impressive that claude has all that context
[14:28:59] <jgleeson>	 indeed!
[14:29:04] <jgleeson>	 as long as it's right ha!
[14:31:36] <damilare>	 well the class and method reference sound right, I wonder if it did an internet search before responding
[14:32:58] <damilare>	 though composeClause isn't a static function
[14:39:24] <wikibugs>	 (03PS1) 10Damilare Adedoyin: Add refund and inactive recurring status to donations history list. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1237505 (https://phabricator.wikimedia.org/T416370)
[14:39:42] <wikibugs>	 (03PS1) 10Jgleeson: Add empty orchestrator placeholder to non-Gravy timing log tags [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1237503 (https://phabricator.wikimedia.org/T416525)
[14:40:09] <wikibugs>	 (03CR) 10Jgleeson: "When testing this locally, I now get:" [wikimedia/fundraising/SmashPig] - 10https://gerrit.wikimedia.org/r/1237503 (https://phabricator.wikimedia.org/T416525) (owner: 10Jgleeson)
[14:41:47] <wikibugs>	 (03PS3) 10Damilare Adedoyin: Add contribution status to donation history [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237316 (https://phabricator.wikimedia.org/T416370)
[14:41:52] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Add refund and inactive recurring status to donations history list. [extensions/DonationInterface] - 10https://gerrit.wikimedia.org/r/1237505 (https://phabricator.wikimedia.org/T416370) (owner: 10Damilare Adedoyin)
[14:43:53] <wikibugs>	 03Fundraising Sprint: Ball pit, 06Fundraising-Backlog, 13Patch-For-Review: Add Gravy [Orchestrator|Processor] structured logs layer - https://phabricator.wikimedia.org/T416525#11591695 (10jgleeson) a:03jgleeson
[14:48:53] <ejegg>	 hmm, donations queue is failing all the runs?
[14:49:32] <jgleeson>	 oh wow ejegg did that just start 30 mins ago?
[14:49:33] <jgleeson>	 I didn't see it
[14:49:45] <ejegg>	 ah deadlocks
[14:50:29] <jgleeson>	  247   │ 2026-02-06 14:47:16,896 ERROR     [Civi\Core\Exception\DBQueryException]  
[14:50:29] <jgleeson>	  248   │ 2026-02-06 14:47:16,896 ERROR     DB Error: deadlock         
[14:50:30] <jgleeson>	 yep
[14:50:32] <ejegg>	 huh, nothing long-running in innotop
[14:50:55] <jgleeson>	 ejegg: I see eileenrunning this
[14:51:10] <jgleeson>	 eileen   pts/1    frbast2002.frack 06:30    1:29m  1.32s   ?    /bin/sh /usr/local/bin/wmf-cv api4 -vv WMFAudit.parse gateway=paypal logSearchPastDays=0 settleMode=queue fileLimit=30
[14:51:10] <jgleeson>	 eileen   pts/2    frbast2002.frack 13:20    1:29m  1:22m   ?    sudo -u www-data /srv/cv/bin/cv --cwd=/srv/org.wikimedia.civicrm/ --user=root api4 -vv WMFAudit.parse gateway=paypal logSearchPastDays=0 settle
[14:51:52] <jgleeson>	 I don't think paypal volume would hurt
[14:51:55] <jgleeson>	 seems unrelated
[14:52:14] <ejegg>	 i guess maybe the audit searches could lock some tables?
[14:54:27] <wikibugs>	 (03CR) 10Jgleeson: [C:03+2] "LGTM! thanks for confirming" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237290 (https://phabricator.wikimedia.org/T410409) (owner: 10Damilare Adedoyin)
[14:56:28] <wikibugs>	 (03PS2) 10Lars SG: Only look for double opt in activities with the donor's email address [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237350 (https://phabricator.wikimedia.org/T416198)
[14:56:38] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Only look for double opt in activities with the donor's email address [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237350 (https://phabricator.wikimedia.org/T416198) (owner: 10Lars SG)
[14:58:24] <wikibugs>	 (03CR) 10Ejegg: "OK, I could do that. I guess we also need to make sure that opting out an email revokes the Double-Opt-In activity, right?" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237334 (https://phabricator.wikimedia.org/T407424) (owner: 10Ejegg)
[15:00:33] <jgleeson>	 ejegg: I was trying to confirm that SEPA patch doesnt change anything but I'm struggling to capture the transaction to trigger the webhook. Do you remember how we triggered that from sandbox? https://sandbox.wikimedia.gr4vy.app/merchants/default/transactions/affadb56-bd19-4399-a3de-1aa62f2f055e/overview
[15:01:11] <wikibugs>	 06Fundraising-Backlog, 07fundraising-supporter-opt-in, 10Wikimedia-Fundraising-CiviCRM, 10FR-Email, 10FR-email-preferences: Cancel Double Opt-In activities if donor later opts out. - https://phabricator.wikimedia.org/T416702 (10Ejegg) 03NEW
[15:01:33] <wikibugs>	 06Fundraising-Backlog, 07fundraising-supporter-opt-in, 10Wikimedia-Fundraising-CiviCRM, 10FR-Email, 10FR-email-preferences: Cancel Double Opt-In activities if donor later opts out. - https://phabricator.wikimedia.org/T416702#11591743 (10Ejegg)
[15:01:36] <wikibugs>	 06Fundraising-Backlog, 06Fundraising-Tech-Roadmap, 10Wikimedia-Fundraising-CiviCRM, 07Epic, 13Patch-For-Review: Epic: Double opt-in - https://phabricator.wikimedia.org/T407017#11591744 (10Ejegg)
[15:02:23] <jgleeson>	 I'll leave my ssh tunnelling active whilst I pick the kids up and hopefully the webhook will be there when I get back! 
[15:02:35] <ejegg>	 yeah jgleeson|skrun I think it's just really delayed
[15:13:14] <wikibugs>	 (03PS1) 10Ejegg: Revert "403 = scream" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237519
[15:13:17] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Revert "403 = scream" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237519 (owner: 10Ejegg)
[15:13:40] <wikibugs>	 (03PS1) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1237520
[15:13:44] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1237520 (owner: 10Ejegg)
[15:15:50] <wikibugs>	 (03Merged) 10jenkins-bot: Add recently cancelled contributions to DonorSummary [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237290 (https://phabricator.wikimedia.org/T410409) (owner: 10Damilare Adedoyin)
[15:20:37] <wikibugs>	 (03Merged) 10jenkins-bot: Only look for double opt in activities with the donor's email address [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237350 (https://phabricator.wikimedia.org/T416198) (owner: 10Lars SG)
[15:24:33] <wikibugs>	 (03PS2) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1237520
[15:24:34] <wikibugs>	 (03PS1) 10Ejegg: Revert "403 = scream" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1237523
[15:24:36] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1237520 (owner: 10Ejegg)
[15:31:13] <ejegg>	 ugh, that didn't quite line up right
[15:32:26] <wikibugs>	 (03PS3) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1237520
[15:32:28] <wikibugs>	 (03CR) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1237520 (owner: 10Ejegg)
[15:32:43] <wikibugs>	 (03Abandoned) 10Ejegg: Revert "403 = scream" [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1237523 (owner: 10Ejegg)
[15:32:44] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "403 = scream" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237519 (owner: 10Ejegg)
[15:38:02] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Donation-Processing: USD Benevity import Duplicate Error - https://phabricator.wikimedia.org/T416233#11591961 (10Elbar53) @Lars I'm in the middle of the Benevity import. I'm still getting a lot of errors. I was able to resolve quite a few. Here is...
[15:38:54] <wikibugs>	 (03PS4) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1237520
[15:38:56] <wikibugs>	 (03CR) 10Ejegg: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1237520 (owner: 10Ejegg)
[15:40:01] <wikibugs>	 (03Merged) 10jenkins-bot: Merge branch 'master' into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1237520 (owner: 10Ejegg)
[15:41:16] <ejegg>	 !log fundraising civicrm upgraded from 06842fbf to 40189afa
[15:41:17] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Fundraising/SAL
[15:41:54] <wikibugs>	 10fundraising-tech-ops: Fundraising access request for Segun Oworu - https://phabricator.wikimedia.org/T416711 (10IAckerman-WMF) 03NEW
[15:55:54] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Donation-Processing: USD Benevity import Duplicate Error - https://phabricator.wikimedia.org/T416233#11592051 (10Lars) Thanks @Elbar53, having those error messages makes it clear what the issue is. So we have a bunch of "Multiple contact matches wi...
[15:58:13] <wikibugs>	 (03PS4) 10Damilare Adedoyin: Add contribution status to donation history [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237316 (https://phabricator.wikimedia.org/T416370)
[15:59:21] <wikibugs>	 (03PS3) 10Ejegg: Show double opt-in on deduper screen [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237334 (https://phabricator.wikimedia.org/T407424)
[15:59:41] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Donation-Processing: USD Benevity import Duplicate Error - https://phabricator.wikimedia.org/T416233#11592057 (10Elbar53) Sometimes we don't have the Organization in Civi. I'll have to add it.
[16:03:11] <wikibugs>	 06Fundraising-Backlog, 07fundraising-supporter-opt-in, 10Wikimedia-Fundraising-CiviCRM, 10FR-Email, 10FR-email-preferences: Cancel Double Opt-In activities if donor later opts out. - https://phabricator.wikimedia.org/T416702#11592071 (10Ejegg) I wonder if it would be best to do this in a contact save hoo...
[16:05:09] <wikibugs>	 (03CR) 10Ejegg: "OK Lars, PS3 should act as suggested. I filed this task to cancel double opt-ins on opt-out: https://phabricator.wikimedia.org/T416702" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237334 (https://phabricator.wikimedia.org/T407424) (owner: 10Ejegg)
[16:07:06] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for shust - https://phabricator.wikimedia.org/T416426#11592085 (10Dwisehaupt) certificate renewed and sent via email. password sent via sms.
[16:12:28] <wikibugs>	 06Fundraising-Backlog, 10fundraising-tech-ops: Issue new SSL Client Certificate for llugo - https://phabricator.wikimedia.org/T415404#11592097 (10Dwisehaupt) certificate renewed and sent via email. password sent via sms.
[16:23:44] <jgleeson>	 still no webhook for that sepa trxn :S
[16:24:02] <jgleeson>	 although there are some real interesting ones
[16:25:00] <jgleeson>	 I'll put a pin in this for now. I've asked Gravy for help here https://wikimedia.slack.com/archives/C070F1DVBRN/p1770394996626629?thread_ts=1770118535.986829&cid=C070F1DVBRN and I'll revert back to the donor portal task I wanna progress on
[16:26:10] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Donation-Processing: USD Benevity import Duplicate Error - https://phabricator.wikimedia.org/T416233#11592131 (10Lars) The "Did not find exactly one Organization with the details" problem should be fixed by {T415295} which is pending review and we...
[16:36:18] <jgleeson>	 fr-tech, completely unexpected side bonus to our new logging. We get logging for events triggered by gravys webhooks!
[16:36:32] <jgleeson>	 I'm not sure if that will be useful, but it was unexpected ha
[16:36:46] <damilare>	 nice!
[16:39:37] <wikibugs>	 (03PS1) 10Wfan: Add more location type for more email type [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237533 (https://phabricator.wikimedia.org/T414538)
[16:39:38] <wikibugs>	 (03PS1) 10Wfan: Replacing email to phone for venmo check [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237534 (https://phabricator.wikimedia.org/T416052)
[16:40:47] <jgleeson>	 huh fr-tech
[16:40:53] <jgleeson>	 is something testing braintree 2026-02-06T16:40:16+00:00 f22bc454a88e braintree_gateway: 800:800.1 Donor Request URL: https://paymentstest1.wmcloud.org/index.php/Special:BraintreeGateway
[16:41:02] <jgleeson>	 and using me as the test environment :)
[16:41:16] <jgleeson>	 I mean, that's fine, but I imagine it's not intended
[16:47:22] <damilare>	 not me lol
[16:49:01] <ejegg>	 huh, no post hook fired after editing a contact's custom fields in the UI
[16:55:10] <wikibugs>	 (03CR) 10Ejegg: [C:03+2] Only count double opt in if activity is for correct email [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1237353 (https://phabricator.wikimedia.org/T416018) (owner: 10Lars SG)
[16:56:35] <wikibugs>	 (03Merged) 10jenkins-bot: Only count double opt in if activity is for correct email [wikimedia/fundraising/tools] - 10https://gerrit.wikimedia.org/r/1237353 (https://phabricator.wikimedia.org/T416018) (owner: 10Lars SG)
[16:56:50] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Replacing email to phone for venmo check [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237534 (https://phabricator.wikimedia.org/T416052) (owner: 10Wfan)
[16:59:11] <wikibugs>	 (03CR) 10Wfan: "recheck" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237534 (https://phabricator.wikimedia.org/T416052) (owner: 10Wfan)
[17:00:35] <wfan>	 also not me haha
[17:46:34] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM, 06Wikipedia-iOS-App-Backlog, 06Wikipedia-Android-App-Backlog (Android Release - FY2025-26): Apps Big EN 2025 Banner Name and Impression data /campaign_id mismatch - ... - https://phabricator.wikimedia.org/T415692#11592360
[17:48:13] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add missing B2526 to source/banner in CiviCRM for app donations - https://phabricator.wikimedia.org/T416722 (10Lars) 03NEW
[18:21:50] <icinga-wm>	 RECOVERY - Host frdb1007 is UP: PING OK - Packet loss = 0%, RTA = 0.52 ms
[18:30:12] <icinga-wm>	 RECOVERY - check_mysql on payments1008 is OK: Uptime: 61664 Threads: 4 Questions: 28554 Slow queries: 0 Opens: 135 Open tables: 129 Queries per second avg: 0.463 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments1008&service=check_mysql
[18:30:14] <icinga-wm>	 RECOVERY - check_mysql on payments2006 is OK: Uptime: 1962843 Threads: 8 Questions: 9101378 Slow queries: 0 Opens: 154 Open tables: 148 Queries per second avg: 4.636 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2006&service=check_mysql
[18:30:18] <icinga-wm>	 RECOVERY - check_mysql on payments2005 is OK: Uptime: 1964258 Threads: 8 Questions: 9162862 Slow queries: 0 Opens: 155 Open tables: 149 Queries per second avg: 4.664 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2005&service=check_mysql
[18:31:52] <icinga-wm>	 RECOVERY - Host payments1005 is UP: PING OK - Packet loss = 0%, RTA = 0.37 ms
[18:32:14] <icinga-wm>	 RECOVERY - check_mysql on payments2004 is OK: Uptime: 1965191 Threads: 10 Questions: 9047075 Slow queries: 0 Opens: 152 Open tables: 146 Queries per second avg: 4.603 Slave IO: Yes Slave SQL: Yes Seconds Behind Master: 0 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=payments2004&service=check_mysql
[18:44:41] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Donation-Processing: USD Benevity import Duplicate Error - https://phabricator.wikimedia.org/T416233#11592573 (10Lars) "Multiple contact matches with employer connection" means we have more than one contact who matches on first name, last name, ema...
[18:47:08] <icinga-wm>	 RECOVERY - Host fran1002 is UP: PING OK - Packet loss = 0%, RTA = 0.42 ms
[18:47:58] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Donation-Processing: USD Benevity import Duplicate Error - https://phabricator.wikimedia.org/T416233#11592598 (10Elbar53)  I think it makes sense to match to the zip code when it's the same name and company. @MDemosWMF what do you think?
[18:55:08] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Donation-Processing: USD Benevity import Duplicate Error - https://phabricator.wikimedia.org/T416233#11592622 (10MDemosWMF) Agreed! I think matching the zip would help as it seems that is more commonly included in the donor info. @Lars
[18:58:10] <icinga-wm>	 RECOVERY - Host frban1002 is UP: PING OK - Packet loss = 0%, RTA = 0.57 ms
[19:01:18] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Donation-Processing: USD Benevity import Duplicate Error - https://phabricator.wikimedia.org/T416233#11592642 (10Elbar53) The import is complete. There are 47 errors. Let me know when you want me to try importing them again.
[19:24:55] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Donation-Processing: USD Benevity import Duplicate Error - https://phabricator.wikimedia.org/T416233#11592710 (10Lars) Sounds good, zip code it is. This will take at least a few days to change on our end, but I'll let you know. If we can't wait for...
[19:28:33] <wikibugs>	 (03PS1) 10Ejegg: Disable geotargeting selectors when not editable [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1237562 (https://phabricator.wikimedia.org/T393834)
[19:29:06] <wikibugs>	 (03PS2) 10Ejegg: Disable geotargeting selectors when not editable [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1237562 (https://phabricator.wikimedia.org/T393834)
[19:29:56] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Donation-Processing: USD Benevity import Duplicate Error - https://phabricator.wikimedia.org/T416233#11592731 (10Elbar53) @MDemosWMF are we ok to wait until next week? It will take a long time to merge each one.
[19:33:04] <wikibugs>	 (03CR) 10CI reject: [V:04-1] Disable geotargeting selectors when not editable [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1237562 (https://phabricator.wikimedia.org/T393834) (owner: 10Ejegg)
[19:36:08] <jgleeson>	 When I see y'all talking about SK (SearchKit), it reminds me of my online-gaming-addicted late teen years, spending most days and nights sitting on Counter-Strike. I was seriously hooked, played for numerous teams and clans, and used to go to LAN parties, had midweek practice, and sit on Ventrilo (voice chat) till like 3-4am talking all night with my teammates. Good times, albeit a slightly unhealthy, vampire-like sleeping 
[19:36:08] <jgleeson>	 pattern. Anyway, the best team in the world for most of that period, circa 2002-2005, was a Swedish/German outfit called SK (Schroet Kommando). They were immense, the first e-sports paid gamers I knew of, legends!  they beat everyone, and we all admired them https://en.wikipedia.org/wiki/SK_Gaming I used to love their Counter-Strike match videos, some of the compilations and music were great - ah, the nostalgia!!!!  Here's a 
[19:36:08] <jgleeson>	 sample :D https://www.youtube.com/watch?v=I6IBTNG8DWk I guess it's like new era retro gaming :( 
[19:39:17] <wikibugs>	 (03PS1) 10Ejegg: Stop caching Special:HideBanners [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1237569 (https://phabricator.wikimedia.org/T285210)
[19:39:37] <jgleeson>	 "In 2003, SK Gaming became the first electronic sports organization to contract players, beginning with the SK Sweden Counter-Strike squad.[3]" - https://en.wikipedia.org/wiki/SK_Gaming#Counter-Strike
[19:39:48] <jgleeson>	 wow they really were the first
[19:43:28] <wikibugs>	 (03PS3) 10Ejegg: Disable geotargeting selectors when not editable [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1237562 (https://phabricator.wikimedia.org/T393834)
[19:44:07] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10FR-Donation-Processing: USD Benevity import Duplicate Error - https://phabricator.wikimedia.org/T416233#11592765 (10MDemosWMF) @Elbar53 you can relay the information to Caitlin/Evelyn and see what they say as far as urgency.
[19:49:34] <jgleeson>	 have a good weekend all
[19:49:56] <wikibugs>	 (03PS2) 10Ejegg: Stop emitting cache headers on Special:HideBanners [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1237569 (https://phabricator.wikimedia.org/T285210)
[19:50:22] <ejegg>	 fr-tech ^^^ should clear up a lot of warning noise in production
[19:51:00] <ejegg>	 I just confirmed over in -operations that the current code is not resulting in anything being cached anyway, as the CDN refuses to cache anything with a set-cookie header
[19:51:04] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Add missing B2526 to source/banner in CiviCRM for app donations - https://phabricator.wikimedia.org/T416722#11592793 (10Lars) a:03Lars
[20:01:55] <wikibugs>	 (03CR) 10Ejegg: "Oh dang, sorry my introduction of the sequenceNumber left things in an inconsistent state. Maybe we should just remove the sequenceNumber " [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1236393 (https://phabricator.wikimedia.org/T365488) (owner: 10Cstone)
[20:03:13] <wikibugs>	 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Improve handling of large number of rows for SearchKit download - https://phabricator.wikimedia.org/T416740 (10Lars) 03NEW
[20:09:01] <wikibugs>	 (03PS1) 10Ejegg: Remove sequenceNumber, find activity by workflow [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237582 (https://phabricator.wikimedia.org/T365488)
[20:09:39] <wikibugs>	 (03CR) 10Ejegg: [C:04-1] "How about this alternative that gets rid of sequenceNumber altogether? I641a45938f4bd2a36eab96cedfa9535315fd0d19" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1236393 (https://phabricator.wikimedia.org/T365488) (owner: 10Cstone)
[20:12:25] <wikibugs>	 (03CR) 10Ejegg: Show double opt-in on deduper screen (031 comment) [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1237334 (https://phabricator.wikimedia.org/T407424) (owner: 10Ejegg)
[20:30:06] <wikibugs>	 (03PS1) 10Ejegg: Disable mixin inputs when not editable [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1237584 (https://phabricator.wikimedia.org/T393834)
[20:40:36] <ejegg>	 hmm, I can't connect to the db on frdev
[20:40:53] <ejegg>	 did I miss an announcement about some change?
[20:41:01] * ejegg goes back to catch up on email
[20:41:42] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 13Patch-For-Review, 07Sustainability (Incident Followup): High frequency warning logged in production: Cookies set on {url} with Cache-Control "{cache-control}" - https://phabricator.wikimedia.org/T285210#11592920...
[20:46:42] <eileen>	 ejegg: it's not yet back up post yesterday works
[20:47:03] <eileen>	 - nothing in email AFAIK but in irc backscroll I asked Dallas last night
[20:47:10] <ejegg>	 thanks eileen 
[20:51:04] <eileen>	 Julie asked me yesterday about Paypal status for Jan - as of last night 535/537 paypal batches had been pushed to Intacct for Jan (31 *17) - the 2 with issues relate to Paypal grants. One of them is probably a data tidy up. The other one is awkward - ie we get 1 grant ($1387) that represents 60 individual donations, which we have entered as such. It's awkward because I validate the number of donations in the batch
[20:51:12] <eileen>	 + it's just awkward
[20:51:36] <eileen>	 I'm still parsing some older paypal batches - I think there is at least one lingering edge case 
[20:54:42] <ejegg>	 yep, i've been seeing the emails come in. impressive to see the number of batches that DO resolve
[20:55:24] <ejegg>	 in cstone's absence, do you think you could review https://gerrit.wikimedia.org/r/1236393 ?
[20:56:00] <ejegg>	 basically, I added a 'sequenceNumber' property earlier to allow for sending two different (workflow, activity) combinations
[20:56:21] <ejegg>	 but I didn't account for the fact that there were other places that set the workflow directly
[20:56:44] <ejegg>	 so that patch simplifies things by removing the sequenceNumber
[20:56:56] <ejegg>	 and just making activity depend on workflow
[20:57:10] <icinga-wm>	 PROBLEM - check_memory on fransw2001 is CRITICAL: CRIT Memory 97% used. Largest process: trino-server-co (476332) = 96.4% https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_memory
[21:02:10] <icinga-wm>	 RECOVERY - check_memory on fransw2001 is OK: OK Memory 49% used https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fransw2001&service=check_memory
[21:07:59] <wikibugs>	 (03PS1) 10Ejegg: Disable project / language selectors when not editable [extensions/CentralNotice] - 10https://gerrit.wikimedia.org/r/1237590 (https://phabricator.wikimedia.org/T393834)
[21:08:48] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 13Patch-For-Review: CentralNotice campaign page lets people without permissions visually modify stuff - https://phabricator.wikimedia.org/T393834#11593002 (10Ejegg)
[21:10:15] <icinga-wm>	 ACKNOWLEDGEMENT - Host pay-lb1001 is DOWN: PING CRITICAL - Packet loss = 100% Jeff_Green rack move
[21:10:16] <icinga-wm>	 ACKNOWLEDGEMENT - Host pay-lb1002 is DOWN: PING CRITICAL - Packet loss = 100% Jeff_Green rack move
[21:16:08] <icinga-wm>	 RECOVERY - Host pay-lb1001 is UP: PING OK - Packet loss = 0%, RTA = 0.31 ms
[21:17:10] <icinga-wm>	 RECOVERY - Host pay-lb1002 is UP: PING OK - Packet loss = 0%, RTA = 0.44 ms
[21:23:42] <wikibugs>	 14Fundraising Sprint: meow(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Test out, maybe implement new scheduled communication functionality - https://phabricator.wikimedia.org/T369109#11593026 (10Ejegg) @Eileenmcnaughton was this for reminding people about grant deadlines?
[21:26:09] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations, 05MW-1.46-notes (1.46.0-wmf.15; 2026-02-10): Apple Pay donors seeing first & lastname error messages - https://phabricator.wikimedia.org/T407033#11593031 (10Ejegg) OK @MBeat33 / @AMJohnson we figured out how to display the error w...
[21:29:35] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations, 05MW-1.46-notes (1.46.0-wmf.15; 2026-02-10): Apple Pay donors seeing first & lastname error messages - https://phabricator.wikimedia.org/T407033#11593055 (10AMJohnson) Awesome. Thank you, @Ejegg! We'll shout if we see any confusio...
[21:30:30] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Donations failed with "expired card", but in Gr4vy the expiration date appears to be valid? - https://phabricator.wikimedia.org/T413665#11593056 (10Ejegg) Right @AMJohnson, we don't have any better information available to us on the...
[21:31:08] <wikibugs>	 06Fundraising Tech - Chaos Crew, 06Fundraising-Backlog, 06FR-donorrelations: Donations failed with "expired card", but in Gr4vy the expiration date appears to be valid? - https://phabricator.wikimedia.org/T413665#11593057 (10AMJohnson) Gotcha. Thank you for looking into it, @Ejegg!
[21:35:16] <wikibugs>	 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad: Fr-tech expansion - https://phabricator.wikimedia.org/T403035#11593081 (10wiki_willy) a:03VRiley-WMF
[21:35:38] <icinga-wm>	 RECOVERY - Host frdb1006 is UP: PING OK - Packet loss = 0%, RTA = 0.44 ms
[21:37:35] <wikibugs>	 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10ops-eqiad, 06SRE: Eqiad: Fr-tech expansion - https://phabricator.wikimedia.org/T403035#11593091 (10VRiley-WMF) Thanks, I will take a look at this. However, checking everything else, the entire move has been completed. I will work on some o...