[07:31:28] Hm, tumbleweed overnight [08:45:43] Hey godog, hnowlan regarding the swift/tegola issue: the new container has more tiles than the fallback. Can we send traffic to the new container while we keep copying files ? [08:54:21] nemo-yiannis: yeah sure, I think so [08:54:29] 👍 [08:55:45] nemo-yiannis: just to be safe, make sure to check the prefix for the new container being eqiad-v0.0.1 (and/or add me to the review, that's fine too) [10:49:17] I found the issue with reimaging to bullseye backup hosts- the network card doesn't detect link with a 5.10.X kernel- probably a driver issue [10:50:44] I have managed to install a 4.19.X kernel that makes the card work- hopefully that will help me debugging [13:11:35] marostegui: can i get a review of T307101 and the associated CR, pls? [13:11:36] T307101: Reboot pc1013 - https://phabricator.wikimedia.org/T307101 [13:31:39] kormat: yes, doing it. I was in a meeting [13:31:49] you have my condolences [13:33:57] it looks good, I made some small corrections [13:34:05] marostegui: saw, thanks! [14:20:36] godog: we started using the new container for tegola and things look fairly stable. I think we can even stop copying files over from the old container [14:22:59] nemo-yiannis: oh ok! glad to know things are stable, sure I'm fine to stop copying [14:24:07] i will keep an eye for now, lets check-in tomorrow [14:24:23] SGTM [14:24:29] 👍 [14:26:13] meanwhile i will try to export the object filenames from the fallback container to have a good starting point to bootstrap new environments from a tileset [14:27:02] (not the actual data just the tiles z/x/y coordinates [14:27:06] (not the actual data just the tiles z/x/y coordinates) [14:42:02] just fyi: i have mass bullseye reimages running in s1/codfw right now [15:13:22] kormat: lucky you [15:13:42] godog: did you manage to delete the over-full tiles container OK in the end? [15:16:09] Emperor: haven't deleted it yet, still copying the tiles over, we'll check-in tomorrow with nemo-yiannis on what to do tho [15:16:59] gotta go, ttyl [15:35:39] PROBLEM - MariaDB sustained replica lag on s1 on db2146 is CRITICAL: 3116 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2146&var-port=9104 [15:37:29] PROBLEM - MariaDB sustained replica lag on s1 on db2145 is CRITICAL: 3048 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2145&var-port=9104 [15:44:13] RECOVERY - MariaDB sustained replica lag on s1 on db2146 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2146&var-port=9104 [15:46:11] RECOVERY - MariaDB sustained replica lag on s1 on db2145 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2145&var-port=9104