[01:07:22] PROBLEM - MariaDB sustained replica lag on pc2010 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [01:09:10] RECOVERY - MariaDB sustained replica lag on pc2010 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [01:18:12] PROBLEM - MariaDB sustained replica lag on pc2010 is CRITICAL: 3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [01:21:52] PROBLEM - MariaDB sustained replica lag on pc2010 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [01:27:18] RECOVERY - MariaDB sustained replica lag on pc2010 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [01:49:10] PROBLEM - MariaDB sustained replica lag on pc2010 is CRITICAL: 2.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [02:01:55] RECOVERY - MariaDB sustained replica lag on pc2010 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [02:14:36] PROBLEM - MariaDB sustained replica lag on pc2010 is CRITICAL: 3 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [02:15:55] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10Ladsgroup) [02:23:40] RECOVERY - MariaDB sustained replica lag on pc2010 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [02:33:37] marostegui: morning! You told me to remind you about T163532 later. Here's the ping. It can obviously wait if you're busy [02:33:38] T163532: Drop index rev_page_id (rev_page, rev_id) - https://phabricator.wikimedia.org/T163532 [02:36:18] PROBLEM - MariaDB sustained replica lag on pc2010 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [02:39:44] RECOVERY - MariaDB sustained replica lag on pc2010 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [03:21:10] PROBLEM - MariaDB sustained replica lag on pc2010 is CRITICAL: 2.4 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [03:28:04] RECOVERY - MariaDB sustained replica lag on pc2010 is OK: (C)2 ge (W)1 ge 0.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [03:58:00] PROBLEM - MariaDB sustained replica lag on pc2010 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [04:07:02] RECOVERY - MariaDB sustained replica lag on pc2010 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [04:25:34] PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 2.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [04:36:14] Amir1: thanks for the reminder! I am going to depool an s1 host and do it there, will report back in the task [04:36:30] RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [04:36:48] 10DBA, 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: dbstore1004 85% disk space used. - https://phabricator.wikimedia.org/T283125 (10Marostegui) 05Open→03Resolved Thanks! [04:38:57] 10DBA: Rename dbstore1004 to db1183 and place it on s7 - https://phabricator.wikimedia.org/T284622 (10Marostegui) [04:39:08] 10DBA: Rename dbstore1004 to db1183 and place it on s7 - https://phabricator.wikimedia.org/T284622 (10Marostegui) p:05Triage→03Medium [04:47:07] Amir1: https://phabricator.wikimedia.org/T163532#7144661 [04:56:13] marostegui: Thanks <3 [04:56:36] I assume dropping this index would give us some free space given that it's revision table [04:56:57] yeah, definitely useful to drop stuff from there if it is not used [04:58:15] oh also an update on the image table mess. The main patch got merged. I'm going to test it in depth in beta cluster this week and next and after that, the patches that uses them will be merged (already being worked on by Tim and Petr). [04:58:29] hopefully in three weeks we can start cleaning up commons [04:58:36] Amir1: ooooooh sweet [04:58:44] can't wait for that [04:59:37] The main patch that just got merged https://gerrit.wikimedia.org/r/c/mediawiki/core/+/693298 [04:59:42] you can see the chain [05:00:05] and this would be important https://gerrit.wikimedia.org/r/c/mediawiki/extensions/PdfHandler/+/698373 [05:14:08] Wow the first one :-/ [05:21:35] that's why I merged it after the branch cut so I would have two weeks to test in beta cluster and co [05:54:41] marostegui: oh stupid thing. Can you check size of revision table on db1135 and compare it with another host? To see how much dropping the index gives us [06:06:12] 10DBA: Rename dbstore1004 to db1183 and place it on s7 - https://phabricator.wikimedia.org/T284622 (10Marostegui) a:03Kormat [06:06:26] kormat: as requested :p [06:06:40] Amir1: Wilco a bit later! [06:07:04] Thanks. No rush obviously [06:07:17] Unless the db is going to run away which I hope you catch it ASAP [07:49:08] PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [07:52:32] PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 2.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [08:01:04] RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [08:14:46] PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 3.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [08:19:56] RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [08:37:06] PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 3.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [08:42:22] RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [08:46:26] 10Blocked-on-schema-change, 10DBA: Schema change for renaming several indexes in change_tag table - https://phabricator.wikimedia.org/T284619 (10LSobanski) p:05Triage→03Medium [09:13:14] PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 7.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [09:19:54] PROBLEM - MariaDB sustained replica lag on pc2010 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [09:27:10] PROBLEM - MariaDB sustained replica lag on pc2010 is CRITICAL: 2.6 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [09:30:52] RECOVERY - MariaDB sustained replica lag on pc2010 is OK: (C)2 ge (W)1 ge 0 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2010&var-port=9104 [09:31:28] PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 3.2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [09:36:34] PROBLEM - MariaDB sustained replica lag on pc2007 is CRITICAL: 2 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2007&var-port=9104 [09:38:24] RECOVERY - MariaDB sustained replica lag on pc2007 is OK: (C)2 ge (W)1 ge 0.4 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2007&var-port=9104 [09:40:34] RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0.6 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [10:00:36] PROBLEM - MariaDB sustained replica lag on pc2008 is CRITICAL: 2.8 ge 2 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [10:01:00] marostegui: for all our sanity's sake: https://gerrit.wikimedia.org/r/c/operations/puppet/+/698954 [10:03:19] or increase the threshold :) [10:03:22] haha [10:03:29] but yeah something is needed I agree :D [10:03:35] +1ed [10:06:04] RECOVERY - MariaDB sustained replica lag on pc2008 is OK: (C)2 ge (W)1 ge 0.8 https://wikitech.wikimedia.org/wiki/MariaDB/troubleshooting%23Replication_lag https://grafana.wikimedia.org/d/000000273/mysql?orgId=1&var-server=pc2008&var-port=9104 [10:07:47] 10DBA, 10Patch-For-Review: Upgrade s5 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283235 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db1130.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/2021060... [10:29:04] 10DBA, 10Patch-For-Review: Upgrade s5 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283235 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1130.eqiad.wmnet'] ` and were **ALL** successful. [10:30:34] 10DBA, 10Patch-For-Review: Upgrade s5 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283235 (10Marostegui) db1130 (candidate master) reimaged - checking tables now. [10:31:03] 10DBA, 10Patch-For-Review: Upgrade s5 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283235 (10Marostegui) [10:42:05] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) [10:46:13] 10DBA, 10Patch-For-Review, 10Platform Team Initiatives (Revision Storage Schema Improvements), 10Schema-change: Drop index rev_page_id (rev_page, rev_id) - https://phabricator.wikimedia.org/T163532 (10Marostegui) [10:49:11] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) s5 progress [x] dbstore1003 [x] db1161 [x] db1154 [x] db1150 [x] db1145 [x] db1144 [] db1130 [x] db1113 [x] db1110 [] db1100 [x] db1096 [x] clouddb1021 [x] clou... [11:28:17] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) [11:29:00] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) [11:31:58] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) s2 eqiad [x] dbstore1007 [x] dbstore1004 [x] db1182 [x] db1171 [x] db1170 [x] db1162 [x] db1156 [x] db1155 [x] db1146 [x] db1129 [x] db1122 [x] db1105 [x] db1102... [11:32:36] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) [11:32:44] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) [11:35:24] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) s8 eqiad [x] dbstore1005 [x] db1178 [x] db1177 [x] db1172 [x] db1167 [x] db1154 [x] db1126 [x] db1116 [x] db1114 [x] db1111 [x] db1109 [x] db1104 [x] db1101 [x]... [11:35:42] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) [11:37:28] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) [11:39:49] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) s7 eqiad [x] dbstore1003 [x] db1181 [x] db1174 [x] db1170 [x] db1158 [x] db1155 [x] db1136 [x] db1127 [x] db1116 [x] db1101 [x] db1098 [x] clouddb1021 [x] cloud... [11:40:05] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) [11:42:51] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) [11:50:34] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) s4 eqiad [x] dbstore1007 [x] dbstore1004 [] db1160 [] db1155 [x] db1150 [] db1149 [] db1148 [] db1147 [] db1146 [x] db1145 [] db1144 [] db1143 [] db1142 [x] db11... [11:59:21] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) [12:08:08] what part of puppet do i need to patch to make marostegui stop spamming the channel? :P [12:08:37] hahaha [12:08:43] <3 [12:10:08] remove me from the admin group in puppet! [12:21:37] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) [12:23:08] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) [12:26:16] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) s1 eqiad [x] dbstore1003 [x] db1184 [x] db1169 [x] db1164 [x] db1163 [x] db1154 [x] db1140 [x] db1139 [x] db1135 [x] db1134 [x] db1133 [x] db1119 [x] db1118 [x]... [12:26:54] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) [12:30:10] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) [12:39:13] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) s3 eqiad [x] dbstore1007 [x] dbstore1004 [] db1179 [] db1175 [] db1171 [] db1166 [] db1157 [] db1154 [] db1123 [] db1112 [] db1102 [] clouddb1021 [] clouddb1017... [13:03:56] 10DBA: Switchover s3 from db1123 to db1157 - https://phabricator.wikimedia.org/T284648 (10Kormat) [13:04:28] 10DBA: Switchover s3 from db1123 to db1157 - https://phabricator.wikimedia.org/T284648 (10Kormat) [13:04:58] 10DBA: Switchover s3 from db1123 to db1157 - https://phabricator.wikimedia.org/T284648 (10Kormat) [13:06:01] 10DBA: Switchover s3 from db1123 to db1157 - https://phabricator.wikimedia.org/T284648 (10Kormat) [13:18:33] 10DBA, 10Patch-For-Review: Switchover s3 from db1123 to db1157 - https://phabricator.wikimedia.org/T284648 (10Kormat) [13:22:12] 10DBA, 10Patch-For-Review: Switchover s3 from db1123 to db1157 - https://phabricator.wikimedia.org/T284648 (10Marostegui) The procedure looks good to me! [13:34:45] FYI kormat and sobanski, ma just told me there's a bot running doing some changes against ast.wiktionary (s3), might take around 1 day or so. There's max repllag set to 5 for it, just in case if you see something strange. [13:35:17] I'm keeping an eye on https://replag.toolforge.org/ just in case [13:35:26] stays on zero so I guess it's good [13:36:09] thanks for the heads-up 👍 [13:37:12] I wish I could speed the process but if I use put_throttle = 1 I assume databases are going to complain :-) [13:38:10] ma: this might be good too: https://grafana.wikimedia.org/d/000000278/mysql-aggregated?viewPanel=6&orgId=1&from=now-15m&to=now&var-site=eqiad&var-group=core&var-shard=s3&var-role=All [13:38:21] to monitor that I mean [13:42:19] 10Blocked-on-schema-change, 10DBA: Schema change for adding oi_timestamp on oldimage table - https://phabricator.wikimedia.org/T284221 (10Marostegui) [13:49:39] 10DBA: Switchover s5 from db1100 to db1130 - https://phabricator.wikimedia.org/T284529 (10Kormat) Procedure looks good :) [13:54:37] marostegui: looks nicer and it has autoupdate :) [16:26:44] 10DBA, 10DiscussionTools, 10Editing-team (Tracking), 10Performance-Team (Radar): Post-deployment: evaluate impact on site performance - https://phabricator.wikimedia.org/T280606 (10ppelberg) >>! In T280606#7063026, @Esanders wrote: > Sounds good. Do you need any more input from the Editing team at this poi... [17:02:48] 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Jelto) @Dzahn and I checked the backups on `gitlab1001`. Backups are enabled again and there is one new backup for today. Currently the backup for one... [17:33:30] 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Sergey.Trofimovsky.SF) A bit of a context: the plan was to hold 7 days worth of backups locally, but in order to test local retention and its interacti... [17:44:05] 10DBA, 10DiscussionTools, 10Editing-team (Tracking), 10Performance-Team (Radar): Post-deployment: evaluate impact on site performance - https://phabricator.wikimedia.org/T280606 (10Krinkle) [17:44:26] 10DBA, 10DiscussionTools, 10Editing-team (Tracking), 10Performance-Team (Radar): Post-deployment: evaluate impact on site performance - https://phabricator.wikimedia.org/T280606 (10Krinkle) No further input is needed at this time. [17:45:17] 10DBA, 10DiscussionTools, 10Performance-Team, 10Editing-team (FY2020-21 Kanban Board), and 2 others: Reduce parser cache retention temporarily for DiscussionTools - https://phabricator.wikimedia.org/T280605 (10Krinkle) [17:46:48] 10DBA, 10DiscussionTools, 10Performance-Team, 10Editing-team (FY2020-21 Kanban Board), and 2 others: Reduce parser cache retention temporarily for DiscussionTools - https://phabricator.wikimedia.org/T280605 (10Krinkle) > - Reduce overall ParserCache expiry in wmf-config (reduce fom 30 days to 22 days). Pri... [18:08:13] 10DBA, 10DiscussionTools, 10Editing-team (Tracking), 10Performance-Team (Radar): Post-deployment: evaluate impact on site performance - https://phabricator.wikimedia.org/T280606 (10ppelberg) >>! In T280606#7146751, @Krinkle wrote: > No further input is needed at this time. Noted. Thank you. [20:34:30] 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10wkandek) 3 days duration confirmed as sufficient. [21:15:02] marostegui: further on the ast.wiktionary job, it's going to take days, not just one, to finish the task. Sorry :-(