[08:10:09] <slyngs>	 Rechecking the cookbooks for tomorrow and 00-downtime-db-readonly-checks findes 26 masters, as oppose to the expect 24
[08:15:59] <marostegui>	 slyngs: can you list them?
[08:17:04] <slyngs>	 Maybe, I'll need to check how spicerack gets them. Hang on
[08:17:24] <marostegui>	 Yeah just to see which ones are missing
[08:17:28] <marostegui>	 or rather, added
[08:19:35] <slyngs>	 So it might just be sudo cumin A:db-role-master and divided by two, minus something
[08:19:45] <Emperor>	 buy one off-by-one error, get one free?
[08:21:23] <slyngs>	 Okay, so spicerack finds: db[2157,2159,2161,2179,2191,2203-2205,2229,2241,2248].codfw.wmnet,db[1160,1162,1173,1184,1189,1193,1210,1220,1236,1258,1263].eqiad.wmnet,es[2037,2039].codfw.wmnet,es[1035,1037].eqiad.wmnet
[08:21:33] <marostegui>	 slyngs: let me check
[08:21:43] <slyngs>	 The query is sudo cumin "A:db-core and A:db-role-master"
[08:24:22] <marostegui>	 slyngs: that list looks good
[08:24:41] <marostegui>	 maybe it is x4 which is new (1 master in eqiad and 1 master in codfw)
[08:26:07] <slyngs>	 The 24 is CORE_SECTIONS * CORE_DATACENTERS
[08:26:29] <slyngs>	 So "s6",
[08:26:29] <slyngs>	     "s5",
[08:26:29] <slyngs>	     "s2",
[08:26:29] <slyngs>	     "s7",
[08:26:29] <slyngs>	     "s3",
[08:26:30] <slyngs>	     "s8",
[08:26:30] <slyngs>	     "s4",
[08:26:31] <slyngs>	     "x4",
[08:26:31] <slyngs>	     "s1",
[08:26:32] <slyngs>	     "x1",
[08:26:32] <slyngs>	     "x3",
[08:26:33] <slyngs>	     "es6",
[08:26:33] <slyngs>	     "es7",
[08:26:59] <slyngs>	 Time two
[08:27:08] <marostegui>	 slyngs: that is correct
[08:27:10] <slyngs>	 13 * 2 = 26
[08:27:12] <slyngs>	 What
[08:27:21] <slyngs>	 Where is spicerack getting 24
[08:28:02] <slyngs>	 This is not a DB issue, I shall escalate to volans :-)
[08:28:13] <marostegui>	 slyngs: ok let me know if i can help
[08:28:55] <slyngs>	 https://gerrit.wikimedia.org/r/c/operations/software/spicerack/+/1337864 <- We probably need a new release
[08:30:45] <slyngs>	 Thanks, I'll check on the spicerack release and let you know
[08:30:51] <marostegui>	 ok!
[09:03:15] <slyngs>	 marostegui: elukey is doing a spicerack release for us :-)
[09:03:47] <marostegui>	 oh nice
[09:03:50] <marostegui>	 elukey: thank you :*
[09:06:40] <elukey>	 <3
[09:29:51] <marostegui>	 federico3 cezmunsta lets coordinate here the es7 issue
[09:30:03] <federico3>	 ok
[09:30:04] <cezmunsta>	 +1
[09:30:15] <marostegui>	 any error on the cookbook?
[09:30:21] <federico3>	 marostegui: yes:
[09:30:41] <federico3>	 Failed to run cookbooks.sre.switchdc.databases.prepare.PrepareSection.enable_circular_replication: MASTER_FROM es1035.eqiad.wmnet wrong SLAVE STATUS Read_Master_Log_Pos=1034005804, expected 1033818972 instead
[09:31:02] <federico3>	 I suspect the cb is caching an older value and not reading the current one  when "retry" is issued perhaps
[09:31:15] <cezmunsta>	 ^ eqiad showed as running and codfw showed as stopped at the point that it complained
[09:32:03] <cezmunsta>	 federico3: did we see the issue with the Preparing state for es7 or was that just an earlier one?
[09:32:08] <marostegui>	 ok let's disable writes in es7 for now
[09:32:12] <federico3>	 there's a little race when the cb starts repl and then checks for it: """wrong SLAVE STATUS Slave_IO_Running=Preparing, expected Yes instead""" 
[09:32:52] <federico3>	 and this glitch then cascades into the cb having to go through the manual "retry" step and getting confused 
[09:32:54] <marostegui>	 ok es7 depooled
[09:36:16] <marostegui>	 i've disabled circular replication
[09:36:20] <marostegui>	 we need to check why codfw is lagging behind
[09:37:28] <marostegui>	 so heartbeat isn't running on codfw
[09:39:09] <marostegui>	 ok things are back again
[09:39:15] <marostegui>	 let's try another run of the cookbook
[09:39:24] <marostegui>	 writes are disabled in es7
[09:39:30] <marostegui>	 so let's try another run of the cookbook?
[09:39:33] <marostegui>	 cezmunsta federico3 ^
[09:39:40] <cezmunsta>	 ack
[09:40:06] <federico3>	 https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1343935  a quick mitigation
[09:41:12] <marostegui>	 There were some Sep 22 09:34:02 es1035 mysqld[4517]: 2026-09-22  9:34:02 154900019 [ERROR] Error reading packet from server: Lost connection to server during query (server_errno=2013) on the log
[09:41:20] <marostegui>	 I wonder if there was some network issues?
[09:41:43] <federico3>	 yet it failed also during retries
[09:42:13] <marostegui>	 i can access fine both severs from each other too...
[09:42:15] <federico3>	 dryrunning the change
[09:42:18] <marostegui>	 federico3: how's the cookbook doing now?
[09:42:19] <marostegui>	 right
[09:42:33] <federico3>	 can I run https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1343935 ?
[09:42:50] <marostegui>	 sure
[09:42:54] <federico3>	 ok
[09:43:21] <marostegui>	 remember writes are disabled in es7
[09:43:37] <cezmunsta>	 cb is running again
[09:43:42] <marostegui>	 ok
[09:44:18] <federico3>	 @marostegui BTW you can connect to the tmux session on cumin1004 if you want
[09:44:37] <cezmunsta>	 orc looks better now
[09:44:37] <marostegui>	 things look good now in orchestrator
[09:44:44] <marostegui>	 logs too
[09:44:50] <federico3>	 and it just ran ok
[09:45:14] <cezmunsta>	 There are uncommitted dbctl changes in the alerts
[09:45:19] <cezmunsta>	 .. from before
[09:45:28] <marostegui>	 checking
[09:45:41] <federico3>	 https://grafana.wikimedia.org/d/fcdr7bv/mariadb-aggregated-replication-lag?from=now-15m&to=now&timezone=utc&refresh=30s is back to normal
[09:45:42] <marostegui>	 what? so the depool script didn't commit the changes?
[09:46:18] <marostegui>	 https://phabricator.wikimedia.org/P96490
[09:46:26] <marostegui>	 ok so es7 never had writes disabled...
[09:46:29] <marostegui>	 let me clean that dbctl
[09:47:04] <federico3>	 yes I'm seeing the depooling uncommitted
[09:47:25] <marostegui>	 i've cleaned dbctl
[09:47:35] <marostegui>	 but check the above paste (later) because looks like it didn't commit things
[09:47:44] <cezmunsta>	 logged
[09:48:05] <federico3>	 I suggest we use https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1343935 for the rest of the changes 
[09:48:09] <marostegui>	 +1
[09:48:29] <marostegui>	 I will create a task for the cookbook dbctl depooling thing
[09:48:29] <federico3>	 (hoping 5 s is enough)
[09:48:48] <marostegui>	 federico3: I don't think it was related to the cookbook, but it doesn't hurt
[09:49:19] <federico3>	 no but the glitch in the cb creates a red herring
[09:49:38] <federico3>	 maybe also a task for doing the sleep properly (with retries, timeouts)
[09:49:57] <cezmunsta>	 Creating one
[09:50:47] <federico3>	 can we move on to s6 ?
[09:50:48] <marostegui>	 https://phabricator.wikimedia.org/T438831
[09:50:55] <marostegui>	 federico3: yes, keep going
[09:51:03] <elukey>	 new spicerack deployed on both cumin1004 and 2003
[09:51:59] <cezmunsta>	 https://phabricator.wikimedia.org/T438833
[09:54:46] <federico3>	 cezmunsta: we can continue discussing in query unless we want to stay here
[09:55:04] <cezmunsta>	 yep, return to DM is fine 
[09:55:13] <federico3>	 s6 ran ok without the "Preparing" glitch
[09:56:05] <marostegui>	 :)
[10:13:49] <cezmunsta>	 The rest went through without issue
[10:14:41] <marostegui>	 nice
[10:14:57] <marostegui>	 I added this https://phabricator.wikimedia.org/T438831#12349386 to see if it helps on fiding the root cause
[10:32:06] <slyngs>	 marostegui: The dc switchover cookbook now findes all the db masters, all good. 
[10:32:18] <marostegui>	 great thank you
[12:59:03] <cezmunsta>	 marostegui: so, the issue with es7 earlier seems to be due to using "retry" when execution hit the Preparing state during PrepareSection.enable_circular_replication. The same issue hit x3, but that was during PrepareSection.master_to_restart_replication, so that retried starting pt-heartbeat rather than retrying CHANGE MASTER after replication had started
[13:10:43] <marostegui>	 But what triggered that?
[13:10:55] <marostegui>	 Like why it's never happened before
[13:11:12] <cezmunsta>	 Luck
[13:11:21] <cezmunsta>	 Connecting would also have caused it
[13:39:20] <marostegui>	 cezmunsta: just read the summary on the task, very clear there, thanks for digging into that
[15:18:37] <Emperor>	 Ah, "good", my rclone bug reproduces with s3 and the latest rclone
[15:19:48] <Emperor>	 https://phabricator.wikimedia.org/P96495