[06:49:07] federico3: can you give this a look with relatively high priority? https://phabricator.wikimedia.org/T429748 [07:14:54] yes I saw the email minutes ago - when did it start failing tho? [07:15:57] I tried today [07:16:04] No idea since when [07:33:31] @marostegui do you remember on which cumin host? I'm looking for the log file [07:34:10] cumin1003 shows in PS1 on the ticket [07:34:29] Yep, cumin1003 [07:55:34] @marostegui do you have any opportunity to test it with a real cloning? https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1304730 [07:55:43] yep [07:55:45] let me check [07:56:04] running it now [07:56:06] I will report back [07:57:50] federico3: seems to be working, I will report once it finishes and/or if stops somewhere else, but it looks like it went beyond the original crash already [07:57:57] thanks for taking a look so quickly [08:00:03] is it this one? [08:00:09] what? [08:00:12] https://usercontent.irccloud-cdn.com/file/BloUA0Qc/image.png [08:00:25] yes, that one will be used to clone db1203 [08:00:30] so that is correct [08:01:07] good, first time I see the cloning dashboard with an event! [10:14:01] can I start reboots on the pc sections? [10:14:33] not yet, I am upgrading some [10:20:57] ok [13:02:01] urandom: happy fun team meeting time? [13:02:49] A/V issues… [13:02:56] Working on it [14:28:30] Hi folks, could I get a couple of reviews from things I've found working on the trixie upgrade in Pontoon, please? First https://gerrit.wikimedia.org/r/c/operations/puppet/+/1304818 to install the rewrite middleware in the correct location for the distribution python version. And second https://gerrit.wikimedia.org/r/c/operations/puppet/+/1304822 to look for puppetserver::pontoon (rather than puppetserver) in non-prod environments. [14:28:31] Both have now on-disk changes in production. [14:29:22] sorry _no_ on-disk changes in production [14:30:48] FIRING: MysqlReplicationLag: MySQL instance db1262:9104@s4 has too large replication lag (5d 1h 10m 41s). Its replication source is db1244.eqiad.wmnet. - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1262&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLag [14:30:48] FIRING: MysqlReplicationLagPtHeartbeat: MySQL instance db1262:9104 has too large replication lag (5d 1h 10m 41s) - https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting#Depooling_a_replica - https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&refresh=1m&var-job=All&var-server=db1262&var-port=9104 - https://alerts.wikimedia.org/?q=alertname%3DMysqlReplicationLagPtHeartbeat [14:37:16] marostegui: I see that you started db1262 again hence ^ ? [14:37:25] yep, it is catching up [14:37:48] I am going to downtime that for ab it [14:38:00] ack [15:24:40] FIRING: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:40:03] RESOLVED: SystemdUnitFailed: swift_rclone_sync.service on ms-be1069:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:49:20] hello! Apologies for the short notice, codfw racks A6 and A7 this Wednesday and Thursday. There are a few data persistence nodes. I can take care of db2155 and db2156, but a few hosts have no info on if a depool is needed or not - https://phabricator.wikimedia.org/T429812 for A6 and soon one for A7 [15:50:15] XioNoX: db wise, those db2155 and db2156 are depoolable yes [15:51:07] I already know thanks to federico3's API :) [15:51:29] the ones I don't know are the others: apus-fe2004, aqs2001, aqs2002, aqs2003, aqs2004, ms-backup2003, thanos-fe2005 [15:52:26] and I'll soon follow up for A7 [15:54:32] excellent thanks! [16:08:10] and here is A7 with a few for your team too https://phabricator.wikimedia.org/T429817 [16:25:09] I commented there too [17:16:34] dhinus: there's been a big spike on clouddb1014:s2, which made monitoring a bit unreliable, so far everything looks fine again https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&from=now-3h&to=now&timezone=utc&var-server=clouddb1014&var-datasource=000000026&var-cluster=mysql&refresh=5m&viewPanel=panel-38 [17:16:35] https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&from=now-3h&to=now&timezone=utc&var-serve [17:16:35] r=clouddb1014&var-datasource=000000026&var-cluster=mysql&refresh=5m&viewPanel=panel-31 [17:16:38] great [17:16:43] https://grafana.wikimedia.org/goto/ffpx9bmimnd34c?orgId=1 [17:16:55] https://grafana.wikimedia.org/goto/afpx9c9jeyj9ce?orgId=1 [17:24:34] @marostegui @cezmunsta I'm planning the switchover of s6 in eqiad for tomorrow morning [17:36:22] marostegui: thanks I'm afk but I'll check later [19:40:24] looks like there were 3 big spikes, starting 17:05 UTC, 17:45 UTC and 18:20 UTC. nothing after that, looks like user activity that has now ended. did the spikes cause any issues? did they trigger alerts? [19:45:04] if they reoccur tomorrow I can try to take snapshots of the processlist during the spikes. maybe we could consider enabling performance_schema to get more insights on which user is causing this. [21:21:18] dhinus: is performance_schema disabled on that host?