[04:44:34] 10DBA, 10SRE, 10ops-codfw: Degraded RAID on pc2012 - https://phabricator.wikimedia.org/T284845 (10Marostegui) 05Open→03Invalid This was the raid being built after the install: ` root@pc2012:~# megacli -LDInfo -Lall -aALL Adapter 0 -- Virtual Drive Information: Virtual Drive: 0 (Target Id: 0) Name... [04:52:10] 10DBA, 10SRE, 10ops-codfw: Degraded RAID on db2148 - https://phabricator.wikimedia.org/T284852 (10Marostegui) p:05Triage→03Medium a:03Papaul This is not correct, the RAID is ok and so is the BBU: Something strange happened, as the RAID was degraded but then recovered: ` [8023918.895348] megaraid_sas... [04:54:36] 10DBA, 10SRE, 10ops-codfw: Degraded RAID on pc2014 - https://phabricator.wikimedia.org/T284849 (10Marostegui) 05Open→03Invalid This was the raid being built after the install: ` root@pc2014:~# megacli -LDInfo -Lall -aALL Adapter 0 -- Virtual Drive Information: Virtual Drive: 0 (Target Id: 0) Name... [05:00:08] 10DBA: Productionize pc2011-pc2014 and pc1011-pc1014 - https://phabricator.wikimedia.org/T284825 (10Marostegui) codfw hosts are now ready to be productionized as the racking and installing task in codfw is done (T282482) Reminder: set this hosts into Active mode in netbox. [05:00:17] 10DBA: Productionize pc2011-pc2014 and pc1011-pc1014 - https://phabricator.wikimedia.org/T284825 (10Marostegui) 05Stalled→03Open [05:02:53] 10DBA, 10Patch-For-Review: Productionize pc2011-pc2014 and pc1011-pc1014 - https://phabricator.wikimedia.org/T284825 (10Marostegui) p:05Triage→03Medium [05:14:24] 10DBA, 10Epic, 10Tracking-Neverending: Database tables to be dropped on Wikimedia wikis and other WMF databases (tracking) - https://phabricator.wikimedia.org/T54921 (10Marostegui) [05:14:27] 10DBA, 10Data-Services, 10Projects-Cleanup: Drop DB tables for now-deleted fixcopyrightwiki from production - https://phabricator.wikimedia.org/T246055 (10Marostegui) 05Open→03Resolved This is all done [05:16:53] 10DBA, 10Patch-For-Review, 10Platform Team Initiatives (Revision Storage Schema Improvements), 10Schema-change: Drop index rev_page_id (rev_page, rev_id) - https://phabricator.wikimedia.org/T163532 (10Marostegui) Dropped also on db1099:3311 so we have a host serving on rc,watchlist,logpager, contributions.... [05:43:51] 10DBA: Productionize pc2011-pc2014 and pc1011-pc1014 - https://phabricator.wikimedia.org/T284825 (10Marostegui) Just ran ` lvextend -L+1100G /dev/mapper/tank-data && sudo xfs_growfs /srv` on all the codfw hosts. ` ===== NODE GROUP ===== (4) pc[2011-2014].codfw.wmnet ----- OUTPUT of 'df -hT /srv;' ----- Filesyst... [06:13:41] Amir1: you will like https://phabricator.wikimedia.org/T284888 [06:13:42] XD [06:53:38] 10DBA, 10DiscussionTools, 10Performance-Team, 10Editing-team (Tracking): Post-deployment: evaluate impact on site performance - https://phabricator.wikimedia.org/T280606 (10Peter) Let us add at three talk pages to our synthetic testing (we used three in the past to just rule out problems on one page) at th... [07:37:21] 10DBA, 10SRE, 10ops-codfw: Degraded RAID on db2148 - https://phabricator.wikimedia.org/T284852 (10Marostegui) I have rebooted the host and everything came as normal, all disks online, raid optimal... Leaving this open until @Papaul confirms he wasn't touching these disks while on-site. [08:10:03] 10DBA: Pre DC switchover DB work - https://phabricator.wikimedia.org/T284897 (10Marostegui) [08:10:20] 10DBA: Pre DC switchover eqiad -> eqiad DB work - https://phabricator.wikimedia.org/T284897 (10Marostegui) p:05Triage→03Medium [09:41:13] 10Blocked-on-schema-change, 10DBA: Rename name_title index on page to page_name_title - https://phabricator.wikimedia.org/T284375 (10Marostegui) [09:41:25] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) [09:55:05] 10Blocked-on-schema-change, 10DBA: Rename name_title index on page to page_name_title - https://phabricator.wikimedia.org/T284375 (10Marostegui) [09:55:07] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) [10:13:30] 10DBA: Pre DC switchover eqiad -> eqiad DB work - https://phabricator.wikimedia.org/T284897 (10Marostegui) [10:44:39] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) This change is very very complex, so I am going to deploy it on codfw only and once we have done the switchover (the 28th) will deploy it on eqiad. [10:45:30] 10Blocked-on-schema-change, 10DBA: Rename name_title index on page to page_name_title - https://phabricator.wikimedia.org/T284375 (10Marostegui) [10:45:37] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) [10:47:57] 10Blocked-on-schema-change, 10DBA: Rename name_title index on page to page_name_title - https://phabricator.wikimedia.org/T284375 (10Marostegui) [10:48:02] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) [10:56:56] 10Blocked-on-schema-change, 10DBA: Rename name_title index on page to page_name_title - https://phabricator.wikimedia.org/T284375 (10Marostegui) [10:57:00] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) [11:43:11] 10Blocked-on-schema-change, 10DBA: Rename name_title index on page to page_name_title - https://phabricator.wikimedia.org/T284375 (10Marostegui) [11:43:18] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) [12:03:15] marostegui: :((((( [12:03:30] :_( [12:51:32] 10DBA, 10SRE, 10ops-codfw: Degraded RAID on db2148 - https://phabricator.wikimedia.org/T284852 (10Papaul) @maqrostegui I haven't been on rack B8 for the pass 2 weeks so no I was not touching these disks while on-site [12:53:41] 10DBA, 10SRE, 10ops-codfw: Degraded RAID on db2148 - https://phabricator.wikimedia.org/T284852 (10Marostegui) 05Open→03Resolved Thanks - closing this. It might have been a glitch. [16:27:05] A question from #wikimedia-tech that maybe folks here know how to answer: "[16:16] < MC8> friend wondering how big the article text of all of wikimedia is. i'm sure we've got stats for that somewhere but I can't find it?" [17:43:53] 10DBA, 10Commons, 10Patch-For-Review: Avoid capacity issues from the image table holding the text of pdf/djvu files as part of their metadata - https://phabricator.wikimedia.org/T275268 (10Mainframe98) It appears this (rMWb4849e03b700813aca63cdc4d7b92a40cfa26f3f) breaks viewing files from foreign repos (See... [17:54:09] 10DBA, 10DiscussionTools, 10Performance-Team, 10Editing-team (Tracking): Post-deployment: evaluate impact on site performance - https://phabricator.wikimedia.org/T280606 (10ppelberg) Thank you for the ping, @peter. Before offering suggestions for which three talk pages y'all might use in synthetic testing,... [21:00:01] 10DBA, 10DiscussionTools, 10Performance-Team, 10Editing-team (Tracking): Post-deployment: evaluate impact on site performance - https://phabricator.wikimedia.org/T280606 (10matmarex) My proposals: - https://en.wikipedia.org/wiki/Talk:COVID-19_pandemic - https://en.wikipedia.org/wiki/User_talk:Jimbo_Wal...