[09:42:44] 06Fundraising-Backlog, 10MediaWiki-extensions-WikimediaMaintenance, 10WikimediaMessages, 13Patch-For-Review: Donate sidebar link consistency (sitesupport-url) - https://phabricator.wikimedia.org/T379205#10308647 (10Tgr) > Delete all the [[MediaWiki:Sitesupport-url]] pages. Cleaning up those pages and then... [12:14:02] 10fundraising-tech-ops, 06DC-Ops, 06Infrastructure-Foundations, 10netops, and 3 others: Frack eqiad network upgrade: design, installation and configuration - https://phabricator.wikimedia.org/T377381#10309416 (10Jclark-ctr) @cmooney thanks for the list I have populated both new switches up to port 27 wit... [12:46:16] howdy damilare hows it goin [12:46:24] wb jgleeson [12:46:29] I see it's a holiday in the US [12:46:34] yep [12:46:34] another lonely day for us :) [12:46:48] or peaceful [12:46:50] :D [12:46:52] well Christine and Dallas might be around [12:46:55] but yep [12:47:11] hopefully it stays peaceful [12:47:31] was just trying to sign into the paypal account [12:47:51] my inbox was full of asana emails today [12:48:10] and a few 100 failmails [12:49:48] yh paypal was a bit tricky last week with the webhooks going to the current integration [12:50:07] gravy webhooks going to the current integration listener** [12:50:25] what account are we using in prod, is it the tony le one or is there another one [12:50:31] we used to use tle@ [12:50:33] I think [12:50:41] yh that's what I'm trying to confirm [12:50:57] tbf I'm not sure which is tle [12:51:04] we need to fix that really [12:51:09] I think he left a while back [12:51:50] yh it is the tle [12:51:54] ah I'm still on chaos I see [12:52:07] UK-based chaos [12:52:14] yep only 1 week left [12:52:49] have you been getting all those new asana project emails from the wider fr-online folks ? [12:53:14] yea there has to be away to silence those [12:53:32] at least the ones that do not concern us [12:53:41] exactly [14:46:56] 06Fundraising-Backlog, 10MediaWiki-extensions-CentralNotice, 10Temporary accounts: [Temporary Accounts] Update CentralNotice extension to support Temporary Accounts - https://phabricator.wikimedia.org/T374437#10309823 (10jgleeson) Hi @kostajh, this one feels more like feature work, so I'll move it out of our... [14:46:58] 06Fundraising-Backlog, 06MediaWiki-Platform-Team, 10MediaWiki-Vendor, 10Quibble, and 4 others: mediawiki/core and mediawiki/vendor both skip composer.lock checks - https://phabricator.wikimedia.org/T370380#10309836 (10Krinkle) 05Open→03Resolved [16:35:46] welcome back jgleeson and hi damilare ! yeah I filtered those asanas out of the inbox [17:01:37] make sense cstone [17:27:08] howdy cstone! ty [18:20:16] PROBLEM - Host frqueue2003 is DOWN: PING CRITICAL - Packet loss = 100% [18:22:15] ^^ expected for kernel updates. [18:22:16] RECOVERY - Host frqueue2003 is UP: PING OK - Packet loss = 0%, RTA = 30.53 ms [18:22:23] and there's the recovery. [18:27:54] PROBLEM - Host frban2002 is DOWN: PING CRITICAL - Packet loss = 100% [18:29:05] same here. [18:29:12] PROBLEM - Host frqueue2002 is DOWN: PING CRITICAL - Packet loss = 100% [18:30:12] RECOVERY - Host frban2002 is UP: PING OK - Packet loss = 0%, RTA = 30.36 ms [18:30:14] PROBLEM - check_ipsec on frban2002 is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: fran2001_v4 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_ipsec [18:30:16] PROBLEM - Host pay-lb2002 is DOWN: PING CRITICAL - Packet loss = 100% [18:30:30] PROBLEM - check_ipsec on fran2001 is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: frban2002_v4 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran2001&service=check_ipsec [18:32:10] RECOVERY - Host pay-lb2002 is UP: PING OK - Packet loss = 0%, RTA = 30.44 ms [18:32:14] PROBLEM - Host frdata2001 is DOWN: PING CRITICAL - Packet loss = 100% [18:32:14] RECOVERY - Host frqueue2002 is UP: PING OK - Packet loss = 0%, RTA = 30.57 ms [18:32:18] PROBLEM - Host frpig2002 is DOWN: PING CRITICAL - Packet loss = 100% [18:33:16] RECOVERY - Host frpig2002 is UP: PING OK - Packet loss = 0%, RTA = 30.38 ms [18:35:14] PROBLEM - check_ipsec on frban2002 is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: fran2001_v4 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_ipsec [18:35:32] PROBLEM - check_ipsec on fran2001 is CRITICAL: Strongswan CRITICAL - ok: 0 not-conn: frban2002_v4 https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran2001&service=check_ipsec [18:37:12] RECOVERY - Host frdata2001 is UP: PING OK - Packet loss = 0%, RTA = 30.44 ms [18:38:09] ipsec bits should recover now. [18:40:08] RECOVERY - check_ipsec on fran2001 is OK: Strongswan OK - 1 ESP OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=fran2001&service=check_ipsec [18:40:16] RECOVERY - check_ipsec on frban2002 is OK: Strongswan OK - 1 ESP OK https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=frban2002&service=check_ipsec [19:07:28] PROBLEM - Host frpm2002 is DOWN: PING CRITICAL - Packet loss = 100% [19:12:10] RECOVERY - Host frpm2002 is UP: PING OK - Packet loss = 0%, RTA = 30.51 ms [19:48:52] (03PS1) 10Jgleeson: Revert "Temporarily comment out new field" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089848 [19:57:00] (03CR) 10Eileen: [C:03+2] Revert "Temporarily comment out new field" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089848 (owner: 10Jgleeson) [20:07:41] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1089851 [20:07:57] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1089851 (owner: 10Eileen) [20:19:37] (03Merged) 10jenkins-bot: Revert "Temporarily comment out new field" [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1089848 (owner: 10Jgleeson) [20:19:37] (03Merged) 10jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1089851 (owner: 10Eileen) [20:21:55] !log civicrm upgraded from 65a8de90 to 4330588d [20:21:56] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [20:33:50] (03CR) 10Cstone: [C:03+2] Add riverlea theme [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1087617 (https://phabricator.wikimedia.org/T371543) (owner: 10Eileen) [20:57:01] (03Merged) 10jenkins-bot: Add riverlea theme [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1087617 (https://phabricator.wikimedia.org/T371543) (owner: 10Eileen) [21:37:12] (03Abandoned) 10Eileen: wip [wikimedia/fundraising/crm] - 10https://gerrit.wikimedia.org/r/1082550 (owner: 10Eileen) [22:05:11] (03PS1) 10Eileen: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1089865 [22:05:24] (03CR) 10Eileen: [C:03+2] Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1089865 (owner: 10Eileen) [22:06:12] (03Merged) 10jenkins-bot: Merge branch 'master' of ssh://gerrit.wikimedia.org:29418/wikimedia/fundraising/crm into deployment [wikimedia/fundraising/crm] (deployment) - 10https://gerrit.wikimedia.org/r/1089865 (owner: 10Eileen) [22:13:30] !log civicrm upgraded from 4330588d to bcd072a1 [22:13:32] Logged the message at https://wikitech.wikimedia.org/wiki/Server_Admin_Log [22:20:10] PROBLEM - check_coworker on civi1002 is CRITICAL: PROCS CRITICAL: 1 process with args php /srv/coworker/bin/coworker https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_coworker [22:20:44] ^^ acking that while we are in maint mode. [22:21:18] ACKNOWLEDGEMENT - check_coworker on civi1002 is CRITICAL: PROCS CRITICAL: 1 process with args php /srv/coworker/bin/coworker Dwisehaupt Maint mode side effect https://icinga.wikimedia.org/cgi-bin/icinga/extinfo.cgi?type=2&host=civi1002&service=check_coworker [23:34:18] PROBLEM - Host payments2004 is DOWN: PING CRITICAL - Packet loss = 100% [23:37:12] RECOVERY - Host payments2004 is UP: PING OK - Packet loss = 0%, RTA = 30.47 ms [23:39:46] 03Fundraising Sprint: void(), 06Fundraising-Backlog, 10Wikimedia-Fundraising-CiviCRM: Make cancellation reason available to Acoustic - https://phabricator.wikimedia.org/T379206#10310852 (10Eileenmcnaughton) @NNgu-WMF how do you want this to be visible in Acoustic? I think the options are 1) new field 'most... [23:47:13] 03Fundraising Sprint: void(), 06Fundraising-Backlog: Acoustic SMS stage 5 - update queue consumer support for SMS RML - https://phabricator.wikimedia.org/T376686#10310858 (10Eileenmcnaughton)