[03:12:43] Tendril didn't like all the insults we made to it yesterday :( [04:45:54] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) [04:45:56] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) [04:46:02] 10Blocked-on-schema-change, 10DBA: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Marostegui) [04:46:27] 10DBA, 10Patch-For-Review: Upgrade s5 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283235 (10Marostegui) 05Open→03Stalled Waiting for the switchover to happen to be able to continue the rest of steps [04:48:37] 10DBA: Disable/remove unused features on Tendril - https://phabricator.wikimedia.org/T231185 (10Marostegui) [04:48:39] 10DBA: Create a cronjob or an event to truncate/delete rows from tendril.general_log_sampled table - https://phabricator.wikimedia.org/T281486 (10Marostegui) 05Open→03Declined Closing this for now - don't think we are doing this anytime soon [05:10:24] 10DBA, 10Patch-For-Review, 10Platform Team Initiatives (Revision Storage Schema Improvements), 10Schema-change: Drop index rev_page_id (rev_page, rev_id) - https://phabricator.wikimedia.org/T163532 (10Marostegui) So far, I haven't seen any different query plans between db1135 and the rest of enwiki hosts.... [05:55:44] 10DBA, 10Data-Services, 10Projects-Cleanup: Drop DB tables for now-deleted fixcopyrightwiki from production - https://phabricator.wikimedia.org/T246055 (10Marostegui) a:03Marostegui I am going to slowly start truncating tables. [06:48:36] 10DBA, 10Data-Services, 10Projects-Cleanup: Drop DB tables for now-deleted fixcopyrightwiki from production - https://phabricator.wikimedia.org/T246055 (10Marostegui) This needs to be done on a per-host basis as the tables were already dropped from sanitarium masters (T246055#5941020). If not, replication wi... [07:24:57] 10DBA, 10Data-Services, 10Projects-Cleanup: Drop DB tables for now-deleted fixcopyrightwiki from production - https://phabricator.wikimedia.org/T246055 (10Marostegui) I have truncated all the tables in codfw. Before doing so I took a quick mysqldump and left it at: ` root@cumin1001:/home/marostegui/fixcopyr... [08:09:33] 10DBA, 10Data-Services, 10Projects-Cleanup: Drop DB tables for now-deleted fixcopyrightwiki from production - https://phabricator.wikimedia.org/T246055 (10Marostegui) eqiad truncate progress [x] dbstore1007 [x] dbstore1004 [] db1179 [] db1175 [] db1171 [] db1166 [] db1157 [] db1123 [] db1102 [10:21:26] marostegui: you around? thinking of merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/699213/ now, with puppet disabled [10:21:35] o/ [10:21:48] I would do it on monday to be honest [10:21:49] XD [10:21:56] oh, yeah. good point :) [10:28:26] 10DBA, 10MediaWiki-Parser, 10Performance-Team, 10Parsoid (Tracking), 10Patch-For-Review: purgeParserCache.php should not take over 24 hours for its daily run - https://phabricator.wikimedia.org/T282761 (10Kormat) Optimize of pc1009 (and replica) finished. Disk space usage went from 3.9TB to 2TB. {F3449... [10:28:42] marostegui: i'll revert pc3 to it's usual state on monday too [10:28:55] ah, but i can set it up to replicate from pc1010 until then [10:29:24] sounds good yeah [10:31:40] done [10:42:15] oops. i broke orchestrator? https://orchestrator.wikimedia.org/web/cluster/alias/pc3 [10:42:39] I guess the heartbeat entry for that server_id is missing [10:42:41] that's my guess [10:43:23] looking [10:43:37] ahh [10:43:47] heartbeat for pc1010 is claiming to be in pc1. naughty. [10:44:10] yeah, you probably need to delete all the entries on the heartbeat table and insert the pc3 one [10:44:27] once we switch dcs, we will have to delete the eqiad one, to avoid showing lag in orch [10:44:55] found the problem [10:45:00] there's _2_ heartbeats running on pc1010 [10:45:04] "great" [10:45:18] one per pX? [10:45:19] one for pc1, one for pc3 [10:45:19] pcX? [10:45:21] yep [10:45:21] right [10:45:33] one from feb16 [10:45:43] riiight. when i was rolling out the systemd pt-heartbeat service, [10:45:45] i touched 'masters' [10:45:59] forgetting that all pc instances are counted as masters [10:46:35] yeah [10:48:28] found another stray one on pc2010, killed it too [10:48:53] pc3 looks good now [10:49:46] pc1009/pc2009 are still downtimed for 2 more days, which should give them plenty of time to get warmed up again [10:51:06] good! [13:01:28] 10DBA: Deploy wmfmariadbpy 0.7.1 - https://phabricator.wikimedia.org/T284819 (10Kormat) [13:01:49] 10DBA: Deploy wmfmariadbpy 0.7.1 - https://phabricator.wikimedia.org/T284819 (10Kormat) [13:02:14] ^ leaving this for next week, too. [13:03:54] ah sweet [13:11:37] 10DBA: Deploy wmfmariadbpy 0.7.1 - https://phabricator.wikimedia.org/T284819 (10LSobanski) p:05Triage→03Medium [14:01:39] 10DBA, 10DiscussionTools, 10Performance-Team, 10Editing-team (FY2020-21 Kanban Board), and 2 others: Reduce parser cache retention temporarily for DiscussionTools - https://phabricator.wikimedia.org/T280605 (10Krinkle) [14:04:43] 10DBA, 10DiscussionTools, 10Performance-Team, 10Editing-team (FY2020-21 Kanban Board), and 2 others: Reduce parser cache retention temporarily for DiscussionTools - https://phabricator.wikimedia.org/T280605 (10Krinkle) >>! In T282761#7148210, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikim... [14:06:38] 10DBA, 10DiscussionTools, 10Performance-Team, 10Editing-team (FY2020-21 Kanban Board), and 2 others: Reduce parser cache retention temporarily for DiscussionTools - https://phabricator.wikimedia.org/T280605 (10Kormat) >>! In T280605#7151500, @Krinkle wrote: > Done from our side @ppelberg. We'll monitor ove... [14:14:46] 10DBA, 10DiscussionTools, 10Performance-Team, 10Editing-team (FY2020-21 Kanban Board), and 2 others: Reduce parser cache retention temporarily for DiscussionTools - https://phabricator.wikimedia.org/T280605 (10Marostegui) @Kormat don't forget to truncate pc1010 after a few days, to make sure we leave it cl... [14:28:17] 10DBA: Productionize pc2011-pc2014 and pc1011-pc1014 - https://phabricator.wikimedia.org/T284825 (10Marostegui) [14:28:29] 10DBA: Productionize pc2011-pc2014 and pc1011-pc1014 - https://phabricator.wikimedia.org/T284825 (10Marostegui) 05Open→03Stalled [14:29:05] 10DBA: Productionize pc2011-pc2014 and pc1011-pc1014 - https://phabricator.wikimedia.org/T284825 (10Marostegui) [14:30:07] 10DBA: Productionize pc2011-pc2014 and pc1011-pc1014 - https://phabricator.wikimedia.org/T284825 (10Marostegui) For the record, I did assign them a partman recipe in order to get them installed with the proper one partitioning scheme. I haven't added them to `site.pp` or anywhere else. [14:36:26] 10DBA, 10Patch-For-Review: Productionize pc2011-pc2014 and pc1011-pc1014 - https://phabricator.wikimedia.org/T284825 (10Marostegui) Per my chat with Papaul, I have just added them to `site.pp` as `insetup`. [16:12:17] 10DBA, 10DiscussionTools, 10Editing-team (Tracking), 10Performance-Team (Radar): Post-deployment: evaluate impact on site performance - https://phabricator.wikimedia.org/T280606 (10Krinkle) Late arrival, one additional operational metric to monitor: * Daily purge of parser cache MUST take less than an act... [16:12:24] 10DBA, 10DiscussionTools, 10Editing-team (Tracking), 10Performance-Team (Radar): Post-deployment: evaluate impact on site performance - https://phabricator.wikimedia.org/T280606 (10Krinkle) [19:20:01] 10DBA, 10Patch-For-Review, 10Platform Team Initiatives (Revision Storage Schema Improvements), 10Schema-change: Drop index rev_page_id (rev_page, rev_id) - https://phabricator.wikimedia.org/T163532 (10Ladsgroup) Thanks. I will and I will also create the patch for removing it in codebase too. [21:50:50] 10DBA: Productionize pc2011-pc2014 and pc1011-pc1014 - https://phabricator.wikimedia.org/T284825 (10Papaul) [23:55:23] 10DBA, 10DiscussionTools, 10Performance-Team, 10Editing-team (FY2020-21 Kanban Board), and 2 others: Reduce parser cache retention temporarily for DiscussionTools - https://phabricator.wikimedia.org/T280605 (10ppelberg) >>! In T280605#7151500, @Krinkle wrote: > Done from our side @ppelberg. We'll monitor o...