[04:19:36] 10DBA, 10Patch-For-Review: Switchover s5 from db1100 to db1130 - https://phabricator.wikimedia.org/T284529 (10Marostegui) [04:29:58] 10DBA, 10Patch-For-Review: Switchover s5 from db1100 to db1130 - https://phabricator.wikimedia.org/T284529 (10Marostegui) [04:31:23] 10DBA, 10Patch-For-Review: Switchover s5 from db1100 to db1130 - https://phabricator.wikimedia.org/T284529 (10Marostegui) [05:03:52] 10DBA, 10Patch-For-Review: Switchover s5 from db1100 to db1130 - https://phabricator.wikimedia.org/T284529 (10Marostegui) [05:06:17] 10DBA, 10Patch-For-Review: Switchover s5 from db1100 to db1130 - https://phabricator.wikimedia.org/T284529 (10Marostegui) [05:11:05] 10DBA: Switchover s5 from db1100 to db1130 - https://phabricator.wikimedia.org/T284529 (10Marostegui) Switchover was done. Read only start: 05:00:37 Read only stop: 05:01:23 Total read only time: 46 seconds [05:11:31] 10DBA: Switchover s5 from db1100 to db1130 - https://phabricator.wikimedia.org/T284529 (10Marostegui) [05:13:26] 10DBA, 10Patch-For-Review: Upgrade s5 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283235 (10Marostegui) [05:13:53] 10DBA, 10Patch-For-Review: Upgrade s5 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283235 (10Marostegui) 05Stalledβ†’03Open [05:13:55] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) [05:13:57] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) [05:14:00] 10Blocked-on-schema-change, 10DBA: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Marostegui) [05:24:28] s5 is now in orchestrator [05:25:11] πŸŽ‰ [05:41:55] 10Blocked-on-schema-change, 10DBA: Schema change for watchlist.wl_notificationtimestamp going binary(14) from varbinary(14) - https://phabricator.wikimedia.org/T268392 (10Marostegui) [05:42:06] 10Blocked-on-schema-change, 10DBA: Schema change to turn user_last_timestamp.user_newtalk to binary(14) - https://phabricator.wikimedia.org/T266486 (10Marostegui) [05:42:15] 10Blocked-on-schema-change, 10DBA: Schema change for dropping default of img_timestamp and making it binary(14) - https://phabricator.wikimedia.org/T273360 (10Marostegui) [05:53:06] 10Blocked-on-schema-change, 10DBA: Schema change to make rc_id unsigned and rc_timestamp BINARY - https://phabricator.wikimedia.org/T276150 (10Marostegui) [05:53:17] 10Blocked-on-schema-change, 10DBA: Schema change to make rc_id unsigned and rc_timestamp BINARY - https://phabricator.wikimedia.org/T276150 (10Marostegui) db1100 (old s5 master) done [05:54:48] 10Blocked-on-schema-change, 10DBA: Schema change for making cuc_id in cu_changes unsigned - https://phabricator.wikimedia.org/T283093 (10Marostegui) db1100 (old s5 master) done [05:55:07] 10DBA: Switchover s5 from db1100 to db1130 - https://phabricator.wikimedia.org/T284529 (10Marostegui) [05:55:22] 10Blocked-on-schema-change, 10DBA: Schema change for making cuc_id in cu_changes unsigned - https://phabricator.wikimedia.org/T283093 (10Marostegui) [05:56:21] 10DBA, 10Patch-For-Review: Upgrade s5 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283235 (10Marostegui) [06:01:40] 10DBA: Switchover s5 from db1100 to db1130 - https://phabricator.wikimedia.org/T284529 (10Marostegui) 05Openβ†’03Resolved [06:01:42] 10DBA, 10Patch-For-Review: Upgrade s5 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283235 (10Marostegui) [06:03:56] 10DBA, 10Patch-For-Review: Upgrade s5 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283235 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by marostegui on cumin1001.eqiad.wmnet for hosts: ` ['db1100.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/2021062... [06:24:57] 10DBA, 10Patch-For-Review: Upgrade s5 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283235 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['db1100.eqiad.wmnet'] ` and were **ALL** successful. [06:25:53] 10DBA, 10Patch-For-Review: Upgrade s5 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283235 (10Marostegui) Old master upgraded to Buster and 10.4 - running mysqlcheck now. [06:38:57] jynus: I saw you pushed the change to remove s5 stretch backup host, is there anything else required to mark that step as done on https://phabricator.wikimedia.org/T283235 ? [07:40:21] 10DBA, 10Orchestrator, 10User-Kormat: Enable report_host for mariadb - https://phabricator.wikimedia.org/T266483 (10Marostegui) [07:42:27] that patch, but I wanted to wait a bit, like for s3 [07:46:09] sure, no rush! [08:31:06] 10Blocked-on-schema-change, 10DBA: Schema change for renaming page_timestamp index on revision table to rev_page_timestamp - https://phabricator.wikimedia.org/T283499 (10Marostegui) So far I haven't seen any errors coming from queries forcing this specific index, so I have change it on db1169 as well. If nothi... [08:32:12] sobanski: i've been looking at the purgeParserCache queries, and they're _weeird_ [08:33:37] like, a Lot more expensive than they need to be [09:02:21] Interesting [09:04:46] 10Blocked-on-schema-change, 10DBA, 10Patch-For-Review: Schema change for renaming several indexes in logging table - https://phabricator.wikimedia.org/T270620 (10Marostegui) 05Openβ†’03Stalled Stalling this till the above is pushed [09:17:56] 10DBA, 10MediaWiki-Parser, 10Performance-Team, 10Parsoid (Tracking), 10Patch-For-Review: purgeParserCache.php should not take over 24 hours for its daily run - https://phabricator.wikimedia.org/T282761 (10Kormat) The queries that purgeParserCache sends are... weird, and more expensive than necessary. E.... [09:20:19] 10DBA, 10MediaWiki-Parser, 10Performance-Team, 10Parsoid (Tracking), 10Patch-For-Review: purgeParserCache.php should not take over 24 hours for its daily run - https://phabricator.wikimedia.org/T282761 (10Kormat) I did some digging into the codebase and (assuming my lack of php knowledge didn't lead me a... [09:38:46] 10DBA, 10MediaWiki-Parser, 10Performance-Team, 10Parsoid (Tracking), 10Patch-For-Review: purgeParserCache.php should not take over 24 hours for its daily run - https://phabricator.wikimedia.org/T282761 (10Marostegui) In terms of expensiveness, looking at the table schema, it is going to be down to the am... [09:41:54] 10DBA, 10MediaWiki-Parser, 10Performance-Team, 10Parsoid (Tracking), 10Patch-For-Review: purgeParserCache.php should not take over 24 hours for its daily run - https://phabricator.wikimedia.org/T282761 (10Marostegui) The above tests were done in a 100% idle host by the way. [10:21:29] 10DBA, 10Patch-For-Review: Upgrade s3 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283131 (10Kormat) db1123 had a clean mysqlcheck run, repooling it now. [10:21:42] 10DBA, 10Patch-For-Review: Upgrade s3 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283131 (10Kormat) [12:37:06] 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Jelto) >>! In T274463#7159798, @jbond wrote: > > I can explain this, I have created a gitlab module which is not used in production, but is running in... [12:41:28] 10DBA, 10Patch-For-Review: Upgrade s3 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283131 (10Kormat) a:05Kormatβ†’03jcrespo @jcrespo : assigning to you for the final backups step. Please resolve the task when that's done. Thanks! [13:16:48] 10DBA, 10MediaWiki-Parser, 10Performance-Team, 10Parsoid (Tracking), 10Patch-For-Review: purgeParserCache.php should not take over 24 hours for its daily run - https://phabricator.wikimedia.org/T282761 (10Kormat) Current status: the purge has been running for ~28h, and is at 15% progress. That extrapolat... [13:23:05] ^ 😭 [13:33:13] ParserCache: It's The Bestβ„’ [13:48:27] marostegui: for https://gerrit.wikimedia.org/r/c/operations/puppet/+/700898, i'm going to disable puppet, and start deploying it selectively [13:49:06] rip [13:49:18] +1 :) [14:01:31] 10DBA, 10MediaWiki-Parser, 10Performance-Team, 10Parsoid (Tracking), 10Patch-For-Review: purgeParserCache.php should not take over 24 hours for its daily run - https://phabricator.wikimedia.org/T282761 (10LSobanski) @Krinkle @aaron do you happen to know if there is a specific reason to do the deletes in... [14:07:36] 10DBA, 10MediaWiki-Parser, 10Performance-Team, 10Parsoid (Tracking), 10Patch-For-Review: purgeParserCache.php should not take over 24 hours for its daily run - https://phabricator.wikimedia.org/T282761 (10Daimona) FWIW, looking at the code above I am under the impression that the SELECT+DELETE is only ne... [14:07:51] 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) On backup1001, we can now see our fileset in bconsole: (sudo bconsole -> restore -> 5 -> 96) ` Select the Client (1-228): 96 Automatically sel... [14:08:25] 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10jcrespo) https://grafana.wikimedia.org/d/413r2vbWk/bacula?orgId=1&from=1624370763674&to=1624370871766&var-site=eqiad&var-job=gitlab1001.wikimedia.org-D... [14:10:29] 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) ` [backup1001:~] $ echo "show job" | sudo bconsole | grep gitlab Job: name=gitlab1001.wikimedia.org-Daily-production-gitlab JobType=66 level=Inc... [14:12:53] 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10jcrespo) All of the above is correct, but this is easier to type, I think: ` # check_bacula.py gitlab1001.wikimedia.org-Daily-production-gitlab 2021-0... [14:16:14] 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10jbond) >>! In T274463#7169289, @Jelto wrote: >>>! In T274463#7159798, @jbond wrote: >> >> I can explain this, I have created a gitlab module which is... [14:19:38] 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10jcrespo) > When the backup configuration is in place in GitLab and Bacular we can do a restore test. BTW this is ready, at least one from Bacula -> cl... [14:19:45] 10Data-Persistence-Backup, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: Backups for GitLab - https://phabricator.wikimedia.org/T274463 (10Dzahn) I was able to test restore for the config backup: ` root@gitlab1001:/var/tmp/bacula-restores/etc/gitlab/config_backup/latest# file latest.tar... [14:31:49] 10DBA, 10Patch-For-Review: Upgrade s5 to Debian Buster and MariaDB 10.4 - https://phabricator.wikimedia.org/T283235 (10Marostegui) db1100 had all the tables checked and clean. Started replication [14:32:51] marostegui: tested the monitoring on 4 different hosts, seems to work fine. rolling out to everywhere now. [14:33:21] kormat: sounds good [14:33:29] * marostegui turns off his phone [14:35:08] smart. :) [14:44:20] kormat: some codfw hosts reporting on icinga but with unkown for pt-heartbeat [14:44:26] ie: db2071 es2030 [14:44:56] i hate everything [14:46:11] oh come onnn icinfa [14:46:32] ignore it, it's just looking for attention [14:46:39] XDDD [14:48:16] i forgot to re-run puppet on alert1001 afterwards [14:48:47] the UNKOWN, Unable to read output failures are because the host no longer has the old check configured, but icinga is still looking for it [15:00:12] ah, the typical icinga doing its thing! [15:02:08] there is one other weird thing not related to pt-heartbeat: "MariaDB read only s8" CHECK_NRPE: Receive header underflow - only 0 bytes received (4 expected). on db2082 [15:04:04] oh, it went away now, I think [15:04:49] it may be just pending of a puppet run [16:58:23] 10DBA, 10MediaWiki-Parser, 10Performance-Team, 10Parsoid (Tracking), 10Patch-For-Review: purgeParserCache.php should not take over 24 hours for its daily run - https://phabricator.wikimedia.org/T282761 (10Krinkle) >>! In T282761#7169015, @Kormat wrote: > What I was expecting is: > ` > DELETE FROM `pc134`... [19:45:59] 10DBA, 10MediaWiki-Parser, 10Performance-Team, 10Parsoid (Tracking), 10Patch-For-Review: purgeParserCache.php should not take over 24 hours for its daily run - https://phabricator.wikimedia.org/T282761 (10LSobanski) >>>! In T282761#7169546, @LSobanski wrote: >> Do you happen to know if there is a specif... [19:54:03] 10DBA, 10MediaWiki-Parser, 10Performance-Team, 10Parsoid (Tracking), 10Patch-For-Review: purgeParserCache.php should not take over 24 hours for its daily run - https://phabricator.wikimedia.org/T282761 (10Krinkle) I'm not sure I follow. Is the hypothesis that using by-exptime instead of by-primarykey que...