[04:20:42] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) [04:22:51] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) [04:23:00] 10DBA: DB maintenance work to do while eqiad is passive (June 2021) - https://phabricator.wikimedia.org/T285139 (10Marostegui) [04:23:10] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) 05Open→03Resolved All done [04:34:40] 10Blocked-on-schema-change, 10DBA, 10ContentTranslation, 10Language-Team (Language-2021-April-June): Update cx-notification-log table in Production - https://phabricator.wikimedia.org/T284644 (10Marostegui) [04:35:10] 10Blocked-on-schema-change, 10DBA, 10ContentTranslation, 10Language-Team (Language-2021-April-June): Update cx-notification-log table in Production - https://phabricator.wikimedia.org/T284644 (10Marostegui) 05Open→03Resolved This is all done [05:11:25] 10Blocked-on-schema-change, 10DBA: Schema change for making cuc_id in cu_changes unsigned - https://phabricator.wikimedia.org/T283093 (10Marostegui) [05:11:38] 10DBA: DB maintenance work to do while eqiad is passive (June 2021) - https://phabricator.wikimedia.org/T285139 (10Marostegui) [05:11:49] 10Blocked-on-schema-change, 10DBA: Schema change for making cuc_id in cu_changes unsigned - https://phabricator.wikimedia.org/T283093 (10Marostegui) 05Open→03Resolved All done [05:13:23] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) [05:16:32] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) [05:22:03] 10DBA: job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007 - https://phabricator.wikimedia.org/T278621 (10Marostegui) [05:24:06] 10DBA: job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007 - https://phabricator.wikimedia.org/T278621 (10Marostegui) [05:24:22] 10DBA: job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007 - https://phabricator.wikimedia.org/T278621 (10Marostegui) a:03Marostegui [05:27:15] 10DBA: job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007 - https://phabricator.wikimedia.org/T278621 (10Marostegui) [05:36:12] PROBLEM - MariaDB sustained replica lag on db2078 is CRITICAL: 208.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2078&var-port=13325 [05:36:27] 10DBA: job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007 - https://phabricator.wikimedia.org/T278621 (10Marostegui) [05:40:00] RECOVERY - MariaDB sustained replica lag on db2078 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=db2078&var-port=13325 [05:43:02] 10DBA: job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007 - https://phabricator.wikimedia.org/T278621 (10Marostegui) [05:48:31] 10DBA: job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007 - https://phabricator.wikimedia.org/T278621 (10Marostegui) [06:31:59] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) [06:32:04] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Marostegui) 05Open→03Resolved All done [07:21:52] 10DBA, 10ops-eqiad: Upgrade db1104 firmware - https://phabricator.wikimedia.org/T286226 (10Marostegui) [07:22:05] 10DBA, 10ops-eqiad: Upgrade db1104 firmware - https://phabricator.wikimedia.org/T286226 (10Marostegui) p:05Triage→03High a:03Kormat [09:50:57] 10Blocked-on-schema-change, 10DBA: Schema change for renaming page_timestamp index on revision table to rev_page_timestamp - https://phabricator.wikimedia.org/T283499 (10Marostegui) [10:53:05] 10DBA: Move db1124 and db1125 to misc services temporarily - https://phabricator.wikimedia.org/T286042 (10Marostegui) db1124 is now in m2, leaving it for 24h before adding it to the proxy. [10:53:18] 10DBA: Move db1124 and db1125 to misc services temporarily - https://phabricator.wikimedia.org/T286042 (10Marostegui) [11:09:49] 10DBA: job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007 - https://phabricator.wikimedia.org/T278621 (10Marostegui) [11:13:07] 10Blocked-on-schema-change, 10DBA: Schema change for renaming page_timestamp index on revision table to rev_page_timestamp - https://phabricator.wikimedia.org/T283499 (10Marostegui) s1 progress [] dbstore1003 [] db1184 [x] db1169 [] db1164 [] db1163 [] db1154 [] db1140 [] db1139 [x] db1135 [] db1134 [] db1133... [11:13:48] 10Blocked-on-schema-change, 10DBA: Schema change for renaming page_timestamp index on revision table to rev_page_timestamp - https://phabricator.wikimedia.org/T283499 (10Marostegui) [14:19:58] 10DBA: job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007 - https://phabricator.wikimedia.org/T278621 (10Marostegui) [14:24:23] 10DBA: job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007 - https://phabricator.wikimedia.org/T278621 (10Marostegui) [14:27:25] 10DBA: job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007 - https://phabricator.wikimedia.org/T278621 (10Marostegui) s3 codfw [] db2149 [] db2139 [] db2127 [] db2109 [] db2105 [x] db2094 [x] db2074 [14:27:42] 10DBA: job_cmd is varbinary(255) in production while being varbinary(60) in code since 2007 - https://phabricator.wikimedia.org/T278621 (10Marostegui) [15:32:20] 10DBA, 10Data-Services: Prepare and check storage layer for jvwikisource - https://phabricator.wikimedia.org/T286245 (10Marostegui) p:05Triage→03Medium Let us know when the wiki is created so we can sanitize it