[04:53:29] 10Blocked-on-schema-change, 10DBA: Extend iwlinks.iwl_prefix to VARBINARY(32) - https://phabricator.wikimedia.org/T277123 (10Marostegui) [05:33:25] 10Blocked-on-schema-change, 10DBA: Extend iwlinks.iwl_prefix to VARBINARY(32) - https://phabricator.wikimedia.org/T277123 (10Marostegui) [05:35:50] 10Blocked-on-schema-change, 10DBA: Extend iwlinks.iwl_prefix to VARBINARY(32) - https://phabricator.wikimedia.org/T277123 (10Marostegui) [06:25:41] 10Blocked-on-schema-change, 10DBA: Extend iwlinks.iwl_prefix to VARBINARY(32) - https://phabricator.wikimedia.org/T277123 (10Marostegui) s6 is fully done - only pending the master which will be done once eqiad is passive. [06:26:10] 10Blocked-on-schema-change, 10DBA: Extend iwlinks.iwl_prefix to VARBINARY(32) - https://phabricator.wikimedia.org/T277123 (10Marostegui) [07:08:47] 10Blocked-on-schema-change, 10DBA: Extend iwlinks.iwl_prefix to VARBINARY(32) - https://phabricator.wikimedia.org/T277123 (10Marostegui) [07:25:06] 10Blocked-on-schema-change, 10DBA: Extend iwlinks.iwl_prefix to VARBINARY(32) - https://phabricator.wikimedia.org/T277123 (10Marostegui) [09:26:28] 10DBA, 10netbox: Grants not working with DB hosts with to ipv6 - https://phabricator.wikimedia.org/T270101 (10ayounsi) If we do: ` wikiadmin 10.64.% wikiuser 10.64.% wikiadmin 10.192.% wikiuser 10.192.% ` Can we "just" add the following? ` wikiadmin 2620:0:861:% wikiuser 2620:0:861:% wikiadmin 2620:0:860:% wi... [10:59:24] 10Blocked-on-schema-change, 10DBA: Extend iwlinks.iwl_prefix to VARBINARY(32) - https://phabricator.wikimedia.org/T277123 (10Marostegui) [11:00:02] 10Blocked-on-schema-change, 10DBA: Extend iwlinks.iwl_prefix to VARBINARY(32) - https://phabricator.wikimedia.org/T277123 (10Marostegui) codfw is entirely done - waiting for the DC switch to finish everything in eqiad. [11:09:26] 10DBA: DB maintenance work to do while eqiad is passive - https://phabricator.wikimedia.org/T285139 (10Marostegui) [11:09:35] sobanski: I have created ^ and I am going to start adding tasks as a subtasks [11:09:51] 10DBA: DB maintenance work to do while eqiad is passive - https://phabricator.wikimedia.org/T285139 (10Marostegui) p:05Triage→03Medium [11:12:37] 10DBA, 10wikitech.wikimedia.org: Move database for wikitech (labswiki) to a main cluster section - https://phabricator.wikimedia.org/T167973 (10Marostegui) [11:12:43] 10DBA: DB maintenance work to do while eqiad is passive - https://phabricator.wikimedia.org/T285139 (10Marostegui) [11:14:12] 10DBA: DB maintenance work to do while eqiad is passive - https://phabricator.wikimedia.org/T285139 (10Marostegui) [11:14:14] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) [11:14:16] 10Blocked-on-schema-change, 10DBA: Rename name_title index on page to page_name_title - https://phabricator.wikimedia.org/T284375 (10Marostegui) [11:14:18] 10Blocked-on-schema-change, 10DBA: Schema change for making cuc_id in cu_changes unsigned - https://phabricator.wikimedia.org/T283093 (10Marostegui) [11:14:20] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) [11:14:22] 10Blocked-on-schema-change, 10DBA: Schema change to make rc_id unsigned and rc_timestamp BINARY - https://phabricator.wikimedia.org/T276150 (10Marostegui) [11:14:24] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) [11:14:26] 10Blocked-on-schema-change, 10DBA: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Marostegui) [11:14:28] 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) [11:15:27] 10DBA: fa_deleted_timestamp and fa_timestamp are binary(14) in code but varbinary(14) in production - https://phabricator.wikimedia.org/T277116 (10Marostegui) [11:15:29] 10DBA: DB maintenance work to do while eqiad is passive - https://phabricator.wikimedia.org/T285139 (10Marostegui) [11:15:31] 10DBA: job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007 - https://phabricator.wikimedia.org/T278621 (10Marostegui) [11:15:33] 10DBA: iw_url in interwiki is varbinary(127) in production but blob in code - https://phabricator.wikimedia.org/T277118 (10Marostegui) [11:17:54] 10DBA: DB maintenance work to do while eqiad is passive - https://phabricator.wikimedia.org/T285139 (10Marostegui) [11:17:56] 10Blocked-on-schema-change, 10DBA: Extend iwlinks.iwl_prefix to VARBINARY(32) - https://phabricator.wikimedia.org/T277123 (10Marostegui) [11:18:22] 10DBA: DB maintenance work to do while eqiad is passive - https://phabricator.wikimedia.org/T285139 (10Marostegui) [11:18:24] 10DBA: iw_url in interwiki is varbinary(127) in production but blob in code - https://phabricator.wikimedia.org/T277118 (10Marostegui) [11:18:49] 10DBA: DB maintenance work to do while eqiad is passive - https://phabricator.wikimedia.org/T285139 (10Marostegui) [11:18:51] 10DBA: job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007 - https://phabricator.wikimedia.org/T278621 (10Marostegui) [11:20:02] 10DBA: DB maintenance work to do while eqiad is passive - https://phabricator.wikimedia.org/T285139 (10Marostegui) [11:20:05] 10Blocked-on-schema-change, 10DBA: Schema change for renaming page_timestamp index on revision table to rev_page_timestamp - https://phabricator.wikimedia.org/T283499 (10Marostegui) [11:20:45] Amir1: I appreciate your love token, maybe I should rename the task to: work generated by Amir1 to be done while eqiad is passive [11:21:32] marostegui: https://tenor.com/view/adam-levine-the-voice-hide-gif-4810901 :D [11:21:39] hahahahahaha [11:22:05] 10DBA: DB maintenance work to do while eqiad is passive - https://phabricator.wikimedia.org/T285139 (10Marostegui) [11:23:16] marostegui: I really really don't want to push anything on your already full plate but this is rather large T270620 and would be nice to get it done during the switch over instead of depool-repool dance [11:23:17] T270620: Schema change for renaming several indexes in logging table - https://phabricator.wikimedia.org/T270620 [11:23:41] yeah, it is on my radar [11:23:48] clearly I leave it on your discretion [11:24:34] Amir1: I tend to leave the index ones for later as those can be done on the master without much issues (unless they are UNIQUE indexes) [11:24:53] oh okay [11:24:55] But yeah, I will try to rename a few on eqiad hosts next week and if there are no issues with queries, I will push it to be done in eqiad while codfw is active [11:24:59] you know better [11:25:01] and once codfw is back as standby, do it there [11:25:22] yeah, that's a good idea. It can be done when eqiad is passive [11:25:27] unlike the rest [11:25:44] Actually, I am going to do 2 hosts in eqiad s6 now [11:25:50] So we can see during the weekend if something happened [11:26:08] And if we want to do the revision one too, the rev_page_id (I think it was that one?) we should probably to use the dc switch as well [11:26:34] makes sense [11:26:34] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in logging table - https://phabricator.wikimedia.org/T270620 (10Marostegui) a:03Marostegui [11:26:52] I hope revision gets slightly smaller [11:27:09] if you are ready for it, can you create a normal schema change for that one? (doesn't have to be today) [11:27:14] btw, the patch for fixing one image bug got merged, I'm looking at the next one [11:27:22] oh sweet [11:27:48] Sure [11:28:00] Are we sure we want to drop it? [11:28:07] let me check something quickly [11:28:11] the revision one? [11:28:35] I haven't seen anything different on the queries for those two hosts, I suppose we can do another one and let it run for a few more days [11:29:04] it's just I got PTSD from the previous experience [11:29:10] I know :( [11:29:21] we won't start with enwiki though :) [11:29:49] so I checked and currently we have rev_pag,rev_timestamp beside rev_page,rev_id [11:30:07] meaning at worst case, the queries can't get too slower [11:30:21] yeah, and I haven't seen any query getting slower for any of those hosts [11:30:53] unless on pages with a lot of history, so it's not as radical as tmp1 [11:31:04] ignoring everything about the data [11:31:06] yeah [11:31:12] wow, logging table on ruwiki is 20GB... [11:31:30] did we shrink it? [11:31:37] maybe autopatrol stuff [11:31:46] maybe we didn't optimize it [11:31:58] how many rows it has? [11:32:26] maybe I forgot to run the script :D [11:33:53] one, sec, can you double check if this looks good? https://phabricator.wikimedia.org/P16632 [11:33:56] going to check the rows now [11:34:27] sure [11:35:01] 74331700 [11:35:03] rows ^ [11:36:46] the schema looks good to me [11:36:59] sweet [11:37:00] that's rather big [11:37:31] not super big though. I remember wikidata went from 660M rows to 6M [11:37:36] yeah [11:37:43] I was surprised to see ruwiki with that size [11:37:44] that gives a scale of what it was [11:38:16] going to deploy the logging change on db1096:3316 [11:39:31] 10M rows was 2GB in wikidata, so this looks roughly correct but why 74M rows [11:39:59] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in logging table - https://phabricator.wikimedia.org/T270620 (10Marostegui) Deployed this change on db1180 and db1096:3316, if nothing goes wrong on the query front, we can deploy it everywhere on codfw (or on eqiad if eqiad is passi... [11:40:18] Amir1: did the script run? [11:40:43] I run it on "general mode" years ago [11:41:01] but things might have happened, restore, mishap, etc. [11:41:17] I double check it later [11:41:57] sure, not a big deal [11:44:31] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in logging table - https://phabricator.wikimedia.org/T270620 (10Marostegui) [12:18:07] 10DBA: job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007 - https://phabricator.wikimedia.org/T278621 (10Marostegui) [12:18:09] 10DBA: DB maintenance work to do while eqiad is passive - https://phabricator.wikimedia.org/T285139 (10Marostegui) [12:18:45] 10DBA: job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007 - https://phabricator.wikimedia.org/T278621 (10Marostegui) [12:18:47] 10DBA: DB maintenance work to do while eqiad is passive - https://phabricator.wikimedia.org/T285139 (10Marostegui) [13:26:11] 10DBA, 10netbox: Grants not working with DB hosts with to ipv6 - https://phabricator.wikimedia.org/T270101 (10Marostegui) Yes, but deploying that across all our databases will need to be done carefully as I would rather not do it with replication enabled. I would love to audit all the grants first, especially... [15:44:51] 10Blocked-on-schema-change, 10DBA: Schema change for dropping rev_page_id index - https://phabricator.wikimedia.org/T285149 (10Ladsgroup) [15:45:43] 10DBA, 10Patch-For-Review, 10Platform Team Initiatives (Revision Storage Schema Improvements), 10Schema-change, 10User-Ladsgroup: Drop index rev_page_id (rev_page, rev_id) - https://phabricator.wikimedia.org/T163532 (10Ladsgroup) a:03Ladsgroup Filed {T285149} for DBA work. I'll create patch here. [15:55:53] marostegui: you know what other good thing you can get from abstract schema? Automate check for useless indexes (the ones that are already covered by another index) [15:55:56] found two already [15:57:55] ugh ignore the ones I found, it's the same name :D [16:47:42] marostegui: the ruwiki mess is because of flaggedrevs... [16:48:02] https://www.irccloud.com/pastebin/Pi3nPHBp/ [21:11:38] 10DBA: DB maintenance work to do while eqiad is passive (June 2021) - https://phabricator.wikimedia.org/T285139 (10Krinkle)