[07:37:45] <jynus>	 as it was predicted: "Last snapshot for x1 at eqiad (db1216) taken on 2026-08-04 21:20:11 is 331 GiB, but the previous one was 365 GiB, a change of -9.4 %"
[07:49:54] <marostegui>	 dhinus: I am going to start depooling old clouddb* https://phabricator.wikimedia.org/T434048
[08:03:06] <marostegui>	 dhinus https://phabricator.wikimedia.org/T434048#12186164
[08:10:55] <dhinus>	 marostegui: thanks, btw I'm off this week, back next Tue
[08:40:37] <marostegui>	 Sorry! Enjoy!
[09:05:24] <federico3>	 i'm resetting test-s4 replication chain to remove some stale stuff
[09:15:29] <Amir1>	 T433990
[09:15:30] <stashbot>	 T433990: Optimize echo tables in x1 - https://phabricator.wikimedia.org/T433990
[09:23:01] <jinxer-wm>	 FIRING: SystemdUnitFailed: pt-heartbeat-wikimedia.service on db1267:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[09:23:12] <marostegui>	 ^ expected
[09:25:19] <marostegui>	 federico3: I am trying to add a host in zarcillo UI and I am getting 403 token expired
[09:25:52] <marostegui>	 I just logged in btw
[09:26:43] <federico3>	 it's the same token used for every API route, maybe it was clicked before getting the fresh token?
[09:29:15] <federico3>	 if you can reproduce it can you look at the browser network logging to see if it's sending the usual cookie?
[09:35:19] <marostegui>	 federico3: does it work for you? can you try adding db1267 to ms1, group core and port 3306?
[09:37:37] <federico3>	 oh that's really odd, I got the 403 error in the js popup but not in the logs. The browser log shows 200
[09:37:50] <marostegui>	 yeah
[09:38:09] <federico3>	 and I did not change the js popup thingy recently 
[09:48:01] <jinxer-wm>	 RESOLVED: SystemdUnitFailed: pt-heartbeat-wikimedia.service on db1267:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed
[10:02:54] <federico3>	 ah the netbox token expired
[10:03:39] <federico3>	 it's a pynetbox error that bubbled up
[10:27:02] <federico3>	 @marostegui I extended the token lifetime on netbox and added db1267 and the popup was green
[10:37:39] <marostegui>	 federico3: sweet, did you add the host or should I?
[10:39:11] <federico3>	 i just wrote I added it ;D
[10:40:28] <marostegui>	 thanks!
[16:51:46] <swfrench-wmf>	 federico3: are you still around? I'm about to perform some etcd maintenance in eqiad, and I see some connections that should not be there.
[16:53:15] <swfrench-wmf>	 I suspect this might be zarcillo?
[17:00:51] <swfrench-wmf>	 yes, that's the only thing that makes sense ... and it appears to be configured incorrectly - i.e., the network policies only allow it to connect to eqiad etcd hosts
[17:03:48] <swfrench-wmf>	 ... and it seems I cannot fix the network policy, since it's not in a safe state to deploy (i.e., there are latent diffs that look unsafe to apply)
[17:08:33] <swfrench-wmf>	 if anyone is around who knows enough about zarcillo to help, that would be greatly appreciated. this has been subtly broken since yesterday when the etcd switchover happened and will be broken for "real" when we start reimaging etcd hosts in eqiad shortly.
[17:49:17] <swfrench-wmf>	 opened https://phabricator.wikimedia.org/T434126 for this. federico3 - if you're still around, it would be great to have your help fixing this. if that's not possible, and I'd like an active ACK from DBA that it's acceptable for zarcillo to be degraded / broken for a time.
[18:25:22] <federico3>	 swfrench-wmf: thanks for the detailed bug report, the dbconfig scraper can stay in a degraded state for a while, as it's not being actively used for failovers or similar
[18:26:23] <swfrench-wmf>	 federico3: ah, thanks - that's good to know! okay, great - I'll go ahead and merge my patch that updates the network policy, but otherwise won't touch zarcillo :)
[18:30:16] <federico3>	 swfrench-wmf: if you have the conf change handy I can apply it now
[18:31:44] <swfrench-wmf>	 federico3: oh, that's great! it's https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1321611 and it's merging now. should be live on the deploy host in ~ 1m.
[18:32:54] <swfrench-wmf>	 federico3: it's live now - you should be good to apply, as long as you have whatever it is that provides the additional values I see in the diff :)
[18:33:13] <federico3>	 thanks
[18:37:32] <federico3>	 @swfrench-wmf  It's just a deployment helper that injects credentials - I just ran it so it should be updated now - unless something in the charts needs version bumping
[18:39:01] <swfrench-wmf>	 amazing, thank you! I no longer see dbconfig-instance polling since ~ 18:35 :)
[18:39:34] <swfrench-wmf>	 so yeah, I think we're good. thank you so much for your help <3
[18:39:47] <swfrench-wmf>	 and apologies for the fire drill - we had no idea this was here
[18:39:57] <swfrench-wmf>	 ... or how / whether it was critical
[18:43:45] <federico3>	 it's not critical and I didn't know dbctl polling had to be documented somewhere
[18:44:29] <swfrench-wmf>	 got it, yeah I remember us talking at one point months(?) ago about introducing a direct etcd client somewhere here, but completely lost track of whether that actually happened
[18:45:18] <federico3>	 (more than a year :D )
[18:45:22] <swfrench-wmf>	 (!)
[18:46:46] <swfrench-wmf>	 in any case, I'll work this into our depool and switchover runbooks. at your convenience (I see you're going to be out for a bit), it would be swell if you could add it to https://wikitech.wikimedia.org/wiki/Etcd/Main_cluster#Usage_in_production
[18:54:58] <federico3>	 ok, I added it