[04:55:28] 10DBA, 10Patch-For-Review: Upgrade s5 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283235 (10Marostegui) >>! In T283235#7145224, @Marostegui wrote: > db1130 (candidate master) reimaged - checking tables now. The check came back clean. [05:07:18] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) All done [05:07:39] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) 05Open→03Resolved [05:11:32] 10DBA, 10Patch-For-Review: Upgrade s5 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283235 (10Marostegui) Merged the above patch to start generating the backups from the buster host. [05:11:44] 10DBA, 10Patch-For-Review: Upgrade s5 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283235 (10Marostegui) [05:15:41] 10Blocked-on-schema-change, 10DBA: Rename name_title index on page to page_name_title - https://phabricator.wikimedia.org/T284375 (10Marostegui) [05:16:02] 10Blocked-on-schema-change, 10DBA: Rename name_title index on page to page_name_title - https://phabricator.wikimedia.org/T284375 (10Marostegui) a:03Marostegui [05:50:04] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) a:03Marostegui [05:50:51] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) [06:52:22] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) s6 codfw is entirely done. I have altered two eqiad hosts and will leave it like that till Monday to make sure there are no queries forcing that index. [... [06:52:24] 10Blocked-on-schema-change, 10DBA: Rename name_title index on page to page_name_title - https://phabricator.wikimedia.org/T284375 (10Marostegui) s6 codfw is entirely done. I have altered two eqiad hosts and will leave it like that till Monday to make sure there are no queries forcing that index. [] dbstore100... [06:53:04] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) Even though this change can be done online, it is extremely hard do to it with the host pooled and with replication running due to the traffic flowing to... [06:55:14] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) [06:55:16] 10Blocked-on-schema-change, 10DBA: Rename name_title index on page to page_name_title - https://phabricator.wikimedia.org/T284375 (10Marostegui) [07:23:58] 10DBA, 10Patch-For-Review: Switchover s5 from db1100 to db1130 - https://phabricator.wikimedia.org/T284529 (10Marostegui) [07:26:49] 10DBA, 10Patch-For-Review: Switchover s5 from db1100 to db1130 - https://phabricator.wikimedia.org/T284529 (10Marostegui) [07:28:08] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Ladsgroup) Thank you so much! [08:06:46] all backups looking good :-) [08:07:03] only s6 failed, probably because of interaction with schema changes or something [08:07:06] have a nice day! [09:02:58] 10DBA, 10Patch-For-Review: Switchover s3 from db1123 to db1157 - https://phabricator.wikimedia.org/T284648 (10LSobanski) p:05Triage→03Medium [09:14:38] Amir1: there's something strange with db1096 s6 after the schema changes [09:14:41] I am investigating [09:14:52] https://grafana.wikimedia.org/d/000000273/mysql?viewPanel=3&orgId=1&var-server=db1098&var-port=13316&from=now-24h&to=now [09:15:08] My guess is the optimizer being stupid [09:15:11] But still checking [09:15:31] Sorry, db1098 [09:15:38] db1096 also got the schema change and is working fine [09:16:20] oh oh [09:16:29] is it the revision index? [09:16:41] no, it must be either change_tag or page [09:19:29] found it [09:19:54] let me create a paste [09:22:22] One of the new indexes failed to be created (due to traffic) so it is missing and causing full scans [09:22:24] I will fix that [09:22:31] https://phabricator.wikimedia.org/P16400 [09:26:12] https://phabricator.wikimedia.org/P16400#84628 jeeeez [09:29:05] db1098:3316 is in a very weird state, I am going to rebuild it from another host [09:29:12] and then apply the changes [10:07:22] marostegui: that sounds delegateable [10:07:42] root@db1098:/srv/sqldata.s6# du -sh . [10:07:42] 516G . [10:07:43] almost done! [10:07:50] ok :) [10:07:56] thanks though :* [10:28:49] Amir1: https://phabricator.wikimedia.org/P16400#84645 fixed, going to start repooling the host [10:29:32] wow [10:29:33] thanks [10:29:40] Sorry I was at meetings [10:29:50] no worries! nothing you could've done! [10:30:04] I'm Emotional Support SRE :D [10:30:08] haha [10:30:15] aww <3 [11:38:47] 10DBA, 10wikitech.wikimedia.org, 10cloud-services-team (Kanban): Audit labswiki grants - https://phabricator.wikimedia.org/T282074 (10Marostegui) 05Open→03Resolved I have deployed the following grants on s6: ` | wikiadmin | 208.80.154.160 | | wikiuser | 208.80.154.160 | | wikiadmin | 208.80.155.109 | |... [11:38:50] 10DBA, 10wikitech.wikimedia.org: Move database for wikitech (labswiki) to a main cluster section - https://phabricator.wikimedia.org/T167973 (10Marostegui) [11:43:55] 10DBA, 10wikitech.wikimedia.org: Move database for wikitech (labswiki) to a main cluster section - https://phabricator.wikimedia.org/T167973 (10Marostegui) [14:47:21] I just realised that tendril crashes last night [14:47:25] [Wed Jun 9 21:20:53 2021] mysqld[20558]: segfault at 0 ip 00007fcb3eb1906f sp 00007fc4ee94b698 error 4 in libc-2.24.so[7fcb3e9f0000+195000]