[08:42:08] <cezmunsta>	 About to do s1 codfw switchover
[08:42:21] <marostegui>	 good luck!
[08:46:51] <marostegui>	 cezmunsta: ok to merge your dbctl change?
[08:46:55] <marostegui>	 db2212 with 0 weight?
[08:47:12] <marostegui>	 ah you merged it already with my change
[08:47:48] <cezmunsta>	 sry, yes hit enter a little too quick 
[08:47:54] <marostegui>	 no problem
[08:47:56] <marostegui>	 it was safe to merge :)
[09:00:16] <cezmunsta>	 All done
[09:58:48] <federico3>	 cezmunsta: I added some details to https://phabricator.wikimedia.org/T429455
[10:13:11] <cezmunsta>	 federico3: why are those steps on a review ticket?
[10:16:15] <federico3>	 if you prefer we can move them elsewhere
[10:19:47] <marostegui>	 federico3: Yeah, I think longer discussions on what's needed etc. Let's wait for the review and then we can sit down and talk about its future
[10:20:23] <federico3>	 we already discussed some of the improvements in meetings
[10:21:35] <marostegui>	 federico3: Yes, but it is better to wait for this review/audit to happen so the three of us can have an idea of its status and where we want for its future
[10:22:00] <marostegui>	 We have all those on docs
[10:22:07] <federico3>	 ok
[10:33:44] <cezmunsta>	 I have moved the list to a comment 
[10:39:26] <moritzm>	 federico3: have you been able to CI-build wmf-admin for trixie? 
[10:42:04] <federico3>	 moritzm: I think so: https://gitlab.wikimedia.org/repos/sre/wmfdb/-/jobs/862096 ran on trixie and generated https://gitlab.wikimedia.org/repos/sre/wmfdb/-/jobs/862096/artifacts/browse/WMF_BUILD_DIR/
[10:42:55] <moritzm>	 ok, if these are working fine, then please import it into "main" for apt.wikimedia.org / trixie-wikimedia
[11:20:05] <jinxer-wm>	 FIRING: MySQLReplicaNotUsingGTID: MySQL replica db1182:9104 not using GTID - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting - https://grafana.wikimedia.org/d/0fec1d02-1b0b-44c0-84b0-64894f3ba682/mariadb-gtid - https://alerts.wikimedia.org/?q=alertname%3DMySQLReplicaNotUsingGTID
[11:20:13] <marostegui>	 ^ me testing
[11:23:37] <federico3>	 moritzm: when you have a minute I'm not sure how to make reprepro understand which CI run to pick the packages from
[11:28:18] <moritzm>	 see https://wikitech.wikimedia.org/wiki/APT_Staging_repository
[11:40:58] <federico3>	 uhm this looks sus https://phabricator.wikimedia.org/P94244
[11:44:50] <moritzm>	 check with the sre collab folks, they run this setup
[11:45:56] <federico3>	 thanks
[12:10:21] <cezmunsta>	 What is this server? labtestservices2001.codfw.wmnet
[12:12:26] <marostegui>	 cezmunsta: We do not own that server, but I think it was decommissioned long ago
[12:12:52] <marostegui>	 https://gerrit.wikimedia.org/r/c/operations/puppet/+/505812 yep, decommissioned 7 years ago 
[12:13:03] <marostegui>	 https://phabricator.wikimedia.org/T218022
[12:19:40] <cezmunsta>	 Hmm it comes out of SELECT name, port, section FROM zarcillo.instances INNER JOIN zarcillo.section_instances ON instances.name = section_instances.instance ORDER BY instances.name DESC
[12:19:58] <marostegui>	 we can defeinitely delete it from zarcillo
[12:20:01] <marostegui>	 want me to clean it up?
[12:20:32] <cezmunsta>	 there could be more, one sec...
[12:21:32] <cezmunsta>	 labtestservices2001, labservices1002, labservices1001 with their sections labtestservices and labservices
[12:21:56] <marostegui>	 we don't own any of those
[12:22:01] <marostegui>	 so we can probably delete them 
[12:22:40] <cezmunsta>	 es1045, db1262, db1224 can't be reached, I think that is all expected, but then... what about the grant that needs to be applied?
[12:23:02] <marostegui>	 yeah those are being worked on
[12:23:04] <marostegui>	 es1045 reimage
[12:23:13] <marostegui>	 db1262 down https://phabricator.wikimedia.org/T428832
[12:23:23] <marostegui>	 db1224 down https://phabricator.wikimedia.org/T427535
[12:23:55] <marostegui>	 es1045 is now up
[12:29:15] <cezmunsta>	 ack
[12:55:05] <jinxer-wm>	 RESOLVED: MySQLReplicaNotUsingGTID: MySQL replica db1182:9104 not using GTID - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting - https://grafana.wikimedia.org/d/0fec1d02-1b0b-44c0-84b0-64894f3ba682/mariadb-gtid - https://alerts.wikimedia.org/?q=alertname%3DMySQLReplicaNotUsingGTID
[15:56:08] <dhinus>	 FYI, today I deployed a new version of the "maintain-views" script to all clouddb hosts. it is backward-compatible, but ping me if you see any issues.
[15:56:23] <dhinus>	 I will add some docs to wikitech soon documenting the new features
[16:27:17] <cezmunsta>	 federico3: only pc2024 shows in Zarcillo for pc4, yet orchestrator shows the real state of 2, with pc1024 present. Any ideas why?
[16:28:54] <federico3>	 select * from instances  where name like 'pc1024%'; -> nothing in the instances table on zarcillo
[16:29:10] <cezmunsta>	 It is in the masters table though
[16:29:22] <cezmunsta>	 Oh, no that is a different one
[16:29:37] <federico3>	 but it should also be in the instances table I think
[16:29:47] <cezmunsta>	 pc1016 shows in orc as pc6
[16:29:58] <federico3>	 there's also instances_view 
[16:30:34] <federico3>	 pc6?
[16:31:48] <cezmunsta>	 select * from masters where section = "pc4" 
[16:32:02] <cezmunsta>	 shows a host from https://zarcillo.wikimedia.org/ui/sections#pc6
[16:42:12] <federico3>	 cezmunsta: 
[16:42:14] <federico3>	 https://www.irccloud.com/pastebin/GY2v29z0/
[16:43:01] <federico3>	 other than pc4 having only one instance, I'm seeing pcABCD where D == section number
[16:45:09] <cezmunsta>	 Yes, I can see that. What I am asking is why that does not represent reality
[16:46:19] <cezmunsta>	 Also, is the masters table used?
[17:01:02] <federico3>	 I don't know why it went missing. BTW Are we doing any primary switch tomorrow morning?
[17:07:16] <cezmunsta>	 I was going to do s1, but that has been done so not from me
[19:26:18] <marostegui>	 cezmunsta: deploying pc hosts isnty very automated so probably I forgot some zarcillo steps a few weeks ago when refreshing all that hw. I'll review and fix tomorrow 
[19:26:24] <marostegui>	 Thanks for the heads up
[19:27:00] <marostegui>	 cezmunsta: I don't think we use the masters table for anything else apart from zarcillo internal stuff
[19:28:15] <jynus>	 it is used for monitoring classification- the master is marked as such for prometheus tags
[19:28:43] <jynus>	 but yeah, nothing substantial
[19:30:28] <cezmunsta>	 marostegui: jynus: thanks for confirming +1 
[19:31:31] <jynus>	 cezmunsta: the minimal docs: https://wikitech.wikimedia.org/wiki/MariaDB/Monitoring#Prometheus the details can be seen on puppet