[10:11:01] marostegui: since you're decomissioning broken s6 replica, can I continue with my schema change there? [10:11:18] it's been blocked for a while [10:11:32] yeah, I've asked arnaudb to provision a new host asap to replace db2117 so we are not down on one host for next week. So better to coordinate with him [10:12:26] Amir1: I'll go over it right now so you can restart your schema change! [10:13:03] thanks! [10:13:21] If you can clone from a host that already has the schema change, that'd make my life even easier [10:13:35] sure! do you have one in mind? [10:13:37] (it goes through them sorted by dbname, so anything earlier should be fine) [10:13:44] sure! [10:14:33] now looking, on codfw it's only db2114 (candidate master SBR) + db2097 (multiinstance) [10:14:46] so don't worry, pick another one, I'll do the schema change [10:15:12] but I can run it from 2097 with the new cookbook iteration? [10:15:32] if it works, then sure :D [10:15:35] double check with jaime that there are no backups running at the moment on that host [10:15:46] it'll swap around stuff, it's a good test, if it fails I'll pick one from the unchanged pool [10:15:55] jynus: ^ can we steal your host for a bit? [10:16:23] or simply provision the new host from the last existing backup [10:18:32] it might not have the schema change though [11:15:11] please don't clone the backup sources, use the recovery method [11:35:21] We can create a new backup if that helps [11:38:02] let me run a new backup now of s6 on codfw [11:39:02] [11:38:51]: INFO - Running XtraBackup at db2097.codfw.wmnet:3316 and sending it to dbprov2004.codfw.wmnet [12:19:13] oh I wasn't aware, thanks [12:42:32] mar*stegui could you change the "No more DB maintenance" & "DB Maintenance can start" invites from avaibility "busy" to "free"? [12:43:05] As I don't think it would be a blocker for meetings (?) [12:46:09] ah sure [13:36:27] Emperor: In preparation of the forthcoming update of the apt servers I noticed that the ceph-reef repo you recently added 404s, could you have a look adn update accordingly? [13:40:52] moritzm: I think that's a temporary screw-up by the ceph authors cf https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/message/LZM3FV7UPQP3CX36YZNYKWKXYEMTB2VU/ [13:44:51] ugh, ok [13:47:59] I am runing some long-runnig queries on db1150 (backup source) to identify a logical corruption issue on production affecting backups [13:49:14] Emperor can I recruit you for recovering 2 files potentially lost (should be an easy task this time) [13:49:36] I will file a ticket for context [13:58:08] lost from swift? [14:04:36] it is likely to be on swift, but the metadata is corrupted [14:04:57] one second, I will file the task and will be clearer there [14:09:27] These 2 files https://phabricator.wikimedia.org/T359176 [14:09:59] I think I can get you a calculated path and hopefulyl they are still there [14:11:54] and/or we can permantently delete them [14:12:43] This seems like very recent corruption: November 2023 [14:14:16] yeah, the other was moved on january of this year [14:16:32] it's not clear to me from the ticket which of those paths is the path that (is? should?) exist in swift [14:16:53] yeah, I am calculating it for you, one sec 0:-) [14:18:11] of course, as the upload or move seems to have failed, it will be a guess, but at least we will know if to fix or discard [14:21:44] this is where the files should be: https://phabricator.wikimedia.org/T359176#9601120 [14:25:55] this is a new mw behaviour, and was breaking my backups, so I will try to give a heads up, but otherwise I would like to just cleanup the db and move on [14:28:50] found one, now looking at the second [14:29:00] ah, great news! [14:29:02] there has been a bunch of changes to the uploader recently [14:29:11] :-( [14:29:45] I don't think we should handle those, we can fix this if it is 2 files only, and then report it [14:29:47] jynus: e.g. https://phabricator.wikimedia.org/T350917 (Bawolff has done a bunch of work here) [14:29:58] I will ping him [14:30:22] in general this is more of the same issue as ever- we shouldn't ever move a file after upload, ever [14:33:50] They both seem to be there, so I think I will propose to fix the metadata by hand (ugh), if the dbas and someone from mw agrees [14:34:00] I've located the second [14:34:17] "someone from mw" it is vague, I know :-D [14:34:48] although let me download those first to make sure the hash matches [14:35:11] thank you a lot, Emperor, you helped a lot [14:35:28] the rest is on mine and other hands :-) [14:35:46] you're welcome :) [14:37:00] we could have a periodic check of SELECT count(*) FROM oldimage WHERE NOT (oi_archive_name = CONVERT(oi_archive_name USING utf8mb4)) [14:37:40] as long as you don't email the output to me ;p [14:37:45] ha ha [14:39:01] This smells to me like a byte to character limit bug [14:39:15] as both paths are cut almost at the end of the string [14:40:52] yeah, they are cut at 255 bytes [14:41:20] this could rebound potentially back to the DBAs sorry [14:41:49] although I wonder how it never bite us before [14:42:04] *bit [14:42:13] * Emperor doesn't know how pathnames end up encoded all over the place [14:42:30] (but yes, that second string comes out as 273 bytes if my python-runes are correct) [14:42:44] well, long story they are UTF-8 strings but inside a binary field [14:42:56] so the length is measured in bytes [14:43:00] not characters [14:43:07] right, so len(bytes(x,"utf8")) which is 273 for the second of these [14:43:29] and I belive in this case mysql silently truncates the string in the middle of a multybyte character [14:43:37] making it invalid UTF8 [14:44:05] so it may be more common, just most files won't be affected because they use latin characters [14:44:26] what's the data type for the relevant column? [14:44:43] varbinary(255) [14:45:02] this is scary [14:45:22] I found 2 obvious examples, but every time a file gets archived we will be creating more [14:45:36] most not yet detected [14:45:53] and this happens because the name is limited to 255 bytes [14:46:11] but the archive path adds /archive/1/11/date! before [14:46:31] how exciting [14:46:38] a bomb that must be hidden for years possibly? [14:47:12] and only reavealed when a 250-ish titled file with multibyte characters gets gut incorrectly [14:47:22] *cut [14:47:54] you could presumably go looking for other cases where the oi_archive_name is 255 chars long? [14:48:26] luckyly, our ancient file metadata storage is both terrible but also now very optimized and very redundany [14:48:35] *not [14:48:56] (and similarly >= 200ish for others where this will go wrong if archived) [14:49:03] Emperor: yeah [14:49:53] I think any title 240 charactes and higher [14:50:03] 14 for the date + admiration point [14:50:09] the path is not fully stored [14:51:01] s/characters/bytes/ [14:52:08] the fact that this is biased towards december or january could mean the logic is new (?), or something triggered it recently [14:52:30] SELECT count(*) from oldimage where length(oi_name) >= 240; ~~~> 44176 [14:52:50] oddly similar to the number of backup failures :-D [14:52:50] "yay" [14:53:31] I think this needs a task to track (possibly of High priority? I'm not sure how recoverable the lossage is) [14:53:42] yeah [14:54:15] it is not UBN, because archived are not that important compared to latest files [14:54:24] but we should make sure it doesn't keep happening [14:55:08] and as a former DBA my first instinct is to just make the field longer [14:56:07] I dunno how intensive a schema change that would be (the alternative would be to reduce the maximum object length); might also be worth having the archiving code check the resulting name for length? [14:56:19] yeah, both [14:56:34] that is why I said I was bias towards the db change solution [14:58:02] This looks much better: SELECT count(*) from oldimage where length(oi_name) >= 240 AND RIGHT(oi_archive_name, LENGTH(oi_name)) <> oi_name; ~~~> 22 [14:58:56] the archived table is tiny, so shouldn't be an issue [14:59:08] I'd have to scratch my head a bit to work out exactly what that query is doing / telling us :) [14:59:52] finding archived images whose path doesn't end with the name [15:00:04] so most likely truncated [15:00:19] ah, OK [15:01:43] it is not perfect because there are edge cases with false positives (renames after archiving) [15:01:51] but good enough [15:04:38] https://phabricator.wikimedia.org/P58482 [15:29:51] fyi I had a network error upon commit on dbctl: https://phabricator.wikimedia.org/P58487 it seems to have retried properly because a GET confirms that I have my data persisted afaict [15:31:35] arnaudb: https://gerrit.wikimedia.org/r/c/operations/puppet/+/1008881 [15:32:07] is it because of the trailing space? [15:32:23] That host shows up twice on that list [15:32:31] -_- indeed [15:32:42] arnaudb: can you merge it yourself? [15:32:42] I'll merge it [15:32:44] yes [15:32:51] thanks <3 [15:33:06] thank you, good catch! [15:34:07] arnaudb: also be aware https://gerrit.wikimedia.org/r/c/operations/puppet/+/1008085/comments/b37ad4c9_543edc13 [15:34:31] ack! [15:34:43] are they all in zarcillo too yet? [15:34:48] nope [15:34:55] should they? [15:35:15] Nah, you can add them once they have the data [15:35:20] good! [15:35:33] it can be done anytime really, just make sure they are there before going to production, otherwiese they won't have graphs [15:36:17] it's included in the tasks of the hackish playbook that lets me gain time to code its replacement :D [15:36:38] https://usercontent.irccloud-cdn.com/file/dinnKW9S/image.png [15:36:54] great :) [15:37:28] that looks surprisingly like ansible output :) [15:37:52] * arnaudb whistles [15:38:54] I've a clandestine ansible project that helps me delegate some stuff to avoid mistakes performing while cloning hosts, its basically a cookbook wrapper with a mustache [15:39:30] its supposed to go away as soon as I manage to integrate new iterations on spicerack and cookbooks :) [15:42:47] * volans is watching... :D [15:43:34] is the page expected? [15:44:05] there has been a page? [15:44:28] ** PROBLEM alert - db2096 #page/MariaDB Replica SQL: x1 #page is CRITICAL ** [15:44:43] ** PROBLEM alert - db2096 #page/MariaDB Replica IO: x1 #page is CRITICAL ** [15:44:51] I don't think it's expected [15:44:53] email to alerts@wm.org [15:44:53] let me check [15:44:53] urgh I missed it, nope it must have been wrongfully downtime [15:45:01] there has been, just ack'd the splunk on-call alert [15:45:03] two of them [15:45:04] Amir1: it's cloning [15:45:24] cool [15:45:28] let me check if it's depooled [15:45:29] (SystemdUnitFailed) firing: prometheus-mysqld-exporter.service on db2199:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:45:37] it is supposed to anyway :/ [15:45:45] the clone should downtime it [15:45:58] weirdly I didn't see a post on IRC from icinga [15:46:04] I just saw the page on vops [15:46:09] Same here [15:46:23] it is on icinga though [15:46:33] it's depooled [15:46:33] indeed [15:46:40] 🤔 [15:46:52] but the cookbook automatically downtimes both [15:47:04] if icinga command execution is lagging a lot it's possible the downtime didn't go through [15:47:25] ah I see what happened [15:47:32] what was it? [15:47:32] there was an icinga failover not long ago AFAIUI [15:47:35] there was a network issue during the downtime call [15:47:51] network issue? [15:48:02] arnaudb: let's ACK it on icinga too [15:48:04] on the app level I mean https://usercontent.irccloud-cdn.com/file/r151TxF2/image.png [15:48:19] I'll force downtime via cookbook first marostegui lets see what it gives us [15:48:29] arnaudb: that's a network error posting messages to IRC, not talking to icinga [15:48:30] sounds good [15:48:38] arnaudb: ok yeah thanks [15:48:57] https://grafana.wikimedia.org/d/rsCfQfuZz/icinga?orgId=1&var-datasource=eqiad%20prometheus%2Fops&from=now-7d&to=now&viewPanel=4 [15:49:00] this doesn't look healthy [15:49:27] ah active icinga is alert2001 [15:49:30] https://grafana.wikimedia.org/d/rsCfQfuZz/icinga?orgId=1&var-datasource=codfw%20prometheus%2Fops&from=now-7d&to=now&viewPanel=4 [15:49:31] well, that doesn't look healthy either [15:50:25] (SystemdUnitFailed) firing: (4) prometheus-mysqld-exporter.service on db2198:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_systemd_state - https://grafana.wikimedia.org/d/g-AaZRFWk/systemd-status - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitFailed [15:50:42] will mute those [15:51:06] yep, mute all the new hosts [15:55:33] you have snapshot.s6.2024-03-05--11-38-50 on dbprov2004.codfw.wmnet if needed [15:55:56] thanks jynus! [16:00:17] arnaudb: how are we looking in terms of codfw rack b8? [16:00:40] everything ready, sorry I forgot to be more specific with all those events [16:00:43] go for it! [16:02:39] no probs thanks! [16:13:21] anyone for a DML review? https://phabricator.wikimedia.org/P58491 [16:13:38] arnaudb: all done in rack b8, thanks! [16:15:44] amazing topranks thanks! [16:16:14] jynus: maybe use hex() since the it's invalid utf-8 [16:16:31] e.g. it can break db-mysql [16:16:49] (but nitpick) [16:18:06] ? [16:18:41] I think if our db cannot handle binary, we would have worse problems [16:25:17] I mean "db-mysql" not the db itslef [16:25:33] It has crashed for me with invalid utf-8 [16:25:38] yeah, I wouldn't trust that either [16:25:57] so I don't use it, I showed a few cases where it produces invalid commands [16:27:03] but I wonder if your problem is with the pager, not the client wrapper [16:54:25] volans: looks like dcops create ipv6 records again for es[2035-2040] even though we created a task for them to remember not to, could you do your magic to remove them? (this is not urgent, we are not installing them this week or next week anyway) [16:54:45] if you want me to create a task, I can do so [20:20:44] marostegui: no, not again :/ I can use the existing one for provisioning [20:27:18] volans: thanks, I appreciate it [20:29:25] no prob sigh [20:58:12] marostegui: FYI it's all done [20:58:54] volans: oh wow thanks!! [21:59:24] https://wikis.world/@nixCraft@mastodon.social/112021323302102128