[10:25:46] fixed pc entries on zarcillo.masters [10:37:24] *\o/* [11:04:39] FIRING: SystemdUnitFailed: wmf_auto_restart_prometheus-mysqld-exporter.service on db1262:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [11:07:23] downtime expired ^ giving it more [11:17:49] marostegui: why do some es instances replicate? e.g. es2037 [11:18:42] So we always have 2 sections es being RW (now es6 and es7) once those fill up, we spawn 2 new and close (set RO) the previous ones [11:19:14] Once we have RO sections, we do not replicate as they are closed and only receive read traffic. So disconnect replication to avoid errors and issues with them [11:19:33] so they are standalone for safety purposes, as there is no real reason to have replication between them [11:19:48] so at the moment the writable es sections are es6 and es7 [11:20:00] We usually close sections and spawn new ones every 4-5 years [11:20:09] essentially whenever they reach 80% disk [11:20:28] finally I have the proper packages for wmfdb imported in the apt staging repo and I documented the process in https://wikitech.wikimedia.org/wiki/Debian_packaging/Package_your_software_as_deb#Method_2:_One_commit_for_all_Debian_versions [11:21:05] cezmunsta: https://wikitech.wikimedia.org/wiki/External_storage [11:23:09] marostegui: yep, saw that just didn't spot the desired info when scanning it [11:24:16] cezmunsta: I think we discussed also the fact that those hosts being standaloine are a bit of a pain, as they do not show in orch [11:24:28] They can show there, but they report as host.eqiad.wmnet or whatrever, which isn't very useful [11:24:39] We should explore how to set aliases for standalone hosts [11:24:59] I think we tried years ago and it wasn't very easy or we weren't able to set it up, worth a try again inthe future [11:29:09] @moritzm the packages for wmfdb* are imported on apt-staging for all Debian releases [11:31:24] ok, please let me know when you've imported them into the "main" component of apt.wikimedia.org [11:36:32] moritzm: sudo -i reprepro --noskipold checkupdate bookworm-wikimedia is not showing them - can you do it if you are more familiar with it? [11:45:24] have you followed the documentation I sent you yesterday? https://wikitech.wikimedia.org/wiki/APT_Staging_repository#Promote_packages_to_the_main_repository shows how to you need to add an import config first [11:51:43] not yet, I've been fixing the process for the package build (see above) [11:55:57] I can do the puppet merge in a while [13:00:12] @marostegui @cezmunsta anybody working on m* codfw masters? I need to do a switchover for reboots [13:00:57] federico3: you don't need to switch things for misc [13:00:57] Not me [13:01:57] marostegui: right, I'll do the dbproxy "flip" and update the docs [13:02:35] federico3: what do you mean flip? [13:03:22] letting dbproxy move traffic away from the master and later on restarting haproxy to reset it [13:03:42] federico3: nothing used codfw misc [13:03:56] You can simply go ahead and reboot the hosts [13:04:00] And then reload the proxy [13:04:42] https://wikitech.wikimedia.org/wiki/MariaDB/Upgrading_a_section#Misc_masters <-- look ok like this? [13:05:41] federico3: you need to downtime the hosts and its slaves [13:46:52] marostegui: is there a reason that these helper mysql.
scripts do not accept args?? [13:49:48] https://grafana.wikimedia.org/d/fc48lf4/dbproxy?orgId=1&from=now-30m&to=now&timezone=utc&refresh=30s <-- reboot done [14:08:06] PROBLEM - MariaDB sustained replica lag on m3 on db2234 is CRITICAL: 48 ge 10 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2234&var-port=9104 [14:09:06] RECOVERY - MariaDB sustained replica lag on m3 on db2234 is OK: (C)10 ge (W)5 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/Troubleshooting%23Incident_Response https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2234&var-port=9104 [14:24:36] that was the reboot, all good [15:28:14] Is there a reason that we have both mariabackup AND xtrabackup on a host .. and that their versions differ? [15:32:40] mariadb-backup and mariabackup are both from the same package, but are built against different versions [15:38:08] cezmunsta: j.ynus would be the person to know [15:43:39] Emperor: yep, the backups seem to use mariadb-backup. This appears to be a MariaDB "thing" as they do it in the container too [15:43:50] Anyone fancy a little light code review? /tmp/ on trixie is a tmpfs, so you can't rename() from there to /var - https://gitlab.wikimedia.org/repos/data_persistence/swift-ring/-/merge_requests/21 [15:45:00] It depends upon how you install :D /dev/mapper/os-tmp on /tmp type xfs [15:45:30] sure, but I'm not mucking with the default setup we're using for trixie :) [15:58:38] We seem to have it default to 50% RAM [15:58:59] I wonder if it is worth changing that [16:01:20] thanks - I'll merge it in the morning, rather than at 5pm :) [18:48:47] cezmunsta: I use the version from the mariadb package, but I cannot prevent other people from installing other versions [18:49:28] xtrabackup should be on path the version from the wmf-mariadb1011 package [18:49:39] and the one to use for production [18:49:57] if you use mariabackup, you may have broken the recovery due to a version mismatch [18:50:35] xtrabackup is the one I setup on package for update-alternatives [18:51:02] but mariadb renamed a lot of binaries on version 10.5 or something [18:51:34] we kept the xtrabackup name for consistency, in case in the future we migrate to another vendor [19:02:29] in any case, if you are using mariadb-backup or mariabackup, you are not using the documented procedure