[13:32:32] <andrewbogott>	 Amir1: can I get a few minutes of your time to look at labtestwikitech?  I migrated it to a new db server (as per your request on T310795) but am still getting the same 'has gone away' failure that I started with :)
[13:32:32] <stashbot>	 T310795: Revive Labtestwikitech  (formerly: Abolish labtestwikitech) - https://phabricator.wikimedia.org/T310795
[13:33:40] <andrewbogott>	 app server is cloudweb2002-dev.wikimedia.org, db server is clouddb2002-dev.codfw.wmnet
[13:33:47] <Amir1>	 andrewbogott: does it have the grants with the new password?
[13:34:04] <andrewbogott>	 yes, we double- and triple-checked connectivity and permissions.
[13:34:20] <andrewbogott>	 Also the logs aren't showing a grant failure, it's something more like an unexpected disconnect.
[13:34:46] <andrewbogott>	 "Aborted connection 2666 to db: 'unconnected' user: 'unauthenticated' host: 'cloudweb2002-dev.wikimedia.org' (This connection closed normally without authentication)"
[13:34:58] <andrewbogott>	 so seems like mediawiki misbehaving
[13:35:50] <Amir1>	 hmm, trying to figure it out
[13:35:54] <andrewbogott>	 thank you!
[13:36:05] <Amir1>	 it can be ferm needing update
[13:37:50] <andrewbogott>	 pretty sure it's not a connectivity issue because cli mysql works fine on the app server
[13:37:59] <andrewbogott>	 but please double-check my work :)
[13:42:01] <andrewbogott>	 also feel free to scap pull or alter config or whatever on cloudweb
[14:23:28] <RhinosF1>	 Amir1: s2 went again
[14:23:30] <RhinosF1>	 PROBLEM - MariaDB Replica SQL: s2 on dbstore1007 is CRITICAL: CRITICAL slave_sql_state Slave_SQL_Running: No, Errno: 1054, Errmsg: Error Unknown column tl_namespace in field list on query. Default database: itwiki. [Query snipped] 
[14:24:41] <Amir1>	 I check it
[14:24:41] <RhinosF1>	 s4 also has ~13 hour lag
[14:24:51] <Amir1>	 That's intentional 
[14:24:59] <Amir1>	 one thing at a time
[14:27:28] <RhinosF1>	 ok, i see why for s4
[15:22:10] <Emperor>	 sigh, it would be nice if any given swift cluster could have a full complement of working drives :(
[15:38:38] <Amir1>	 or they boot at first try
[15:38:40] <Amir1>	 :D
[15:50:12] <Emperor>	 I'm not that optimistic ;-)
[16:25:01] <Emperor>	 Amir1: what's your gerrit username, please? I want to tag you in a CR...
[16:25:57] <Reedy>	 ladsgroup
[16:26:28] <Emperor>	 that looks to be a gmail address not a wmf one?
[16:27:01] <Reedy>	 Yeah, it's right though
[16:27:04] <Reedy>	 Lots of people do that
[16:27:35] <Emperor>	 OK
[17:28:15] <andrewbogott>	 Amir1: no luck?
[18:48:27] <Amir1>	 ok, back now, I check Empero.r and Andrew's issues
[18:54:03] <Amir1>	 Emperor: you're now an owner, do you wanna try again?