[00:05:00] Project beta-update-databases-eqiad build #65915: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65915/ [01:05:00] Project beta-update-databases-eqiad build #65916: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65916/ [02:05:01] Project beta-update-databases-eqiad build #65917: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65917/ [03:05:01] Project beta-update-databases-eqiad build #65918: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65918/ [04:05:00] Project beta-update-databases-eqiad build #65919: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65919/ [05:05:01] Project beta-update-databases-eqiad build #65920: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65920/ [06:05:01] Project beta-update-databases-eqiad build #65921: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65921/ [06:24:57] bah the output of the beta-update-databases-eqiad job is buffered :-( [07:05:00] Project beta-update-databases-eqiad build #65922: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65922/ [08:05:00] Project beta-update-databases-eqiad build #65923: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65923/ [09:05:01] Project beta-update-databases-eqiad build #65924: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65924/ [10:05:00] Project beta-update-databases-eqiad build #65925: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65925/ [10:19:03] 10Gerrit, 10wikitech.wikimedia.org, 10LDAP: Rename account Zoranzoki21 to Kizule on Gerrit - https://phabricator.wikimedia.org/T260647 (10Kizule) >>! In T260647#8707253, @Kizule wrote: > Oh, I found that I have some Toolforge projects. I'll add account Kizule as a maintainer, once my request gets approved on... [11:05:00] Project beta-update-databases-eqiad build #65926: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65926/ [11:30:29] 10Gerrit, 10wikitech.wikimedia.org, 10LDAP: Rename account Zoranzoki21 to Kizule on Gerrit - https://phabricator.wikimedia.org/T260647 (10taavi) >>! In T260647#8707211, @Kizule wrote: > So, I'd like to get account Zoranzoki21 merged with Kizule, if it is possible. Merging accounts is unfortunately not possi... [11:46:17] 10Gerrit, 10wikitech.wikimedia.org, 10LDAP: Rename account Zoranzoki21 to Kizule on Gerrit - https://phabricator.wikimedia.org/T260647 (10Kizule) > Merging accounts is unfortunately not possible. I thinked that it is. Merging on Wikitech on way that edits get moved to Kizule, old account on Wikitech gets del... [12:05:00] Project beta-update-databases-eqiad build #65927: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65927/ [13:05:00] Project beta-update-databases-eqiad build #65928: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65928/ [14:03:34] 10Release-Engineering-Team (Seen), 10Wikidata, 10wdwb-tech, 10MW-1.35-notes (1.35.0-wmf.16; 2020-01-21), and 2 others: Stop using jsonlint and instead use eslint-plugin-json for the linting - https://phabricator.wikimedia.org/T220036 (10Kizule) I'm hoping that we will be able to close this really soon (per... [14:05:00] Project beta-update-databases-eqiad build #65929: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65929/ [14:37:03] Project beta-update-databases-eqiad build #65930: 15ABORTED in 17 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65930/ [14:42:50] 10Beta-Cluster-Infrastructure: deployment-prep hourly databases update times out - https://phabricator.wikimedia.org/T332507 (10hashar) [14:43:17] !log deployment-prep: databases update hourly job times out, filed T332507 [14:43:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:43:21] T332507: deployment-prep hourly databases update times out - https://phabricator.wikimedia.org/T332507 [14:59:06] 10Beta-Cluster-Infrastructure: deployment-prep hourly databases update times out - https://phabricator.wikimedia.org/T332507 (10hashar) Nothing really suspicious in mediawiki/core. But for extensions PoolCounter has been removed by 52fa8a4ccc712566842aafb4275acd0a78d1d05e for T324631 ` commit 52fa8a4ccc712566842... [15:13:09] 10Beta-Cluster-Infrastructure: deployment-prep hourly databases update times out - https://phabricator.wikimedia.org/T332507 (10Umherirrender) {T327336} is marked as resolved and it is not refered by mediawiki-config [16:05:01] Project beta-update-databases-eqiad build #65931: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65931/ [17:05:00] Project beta-update-databases-eqiad build #65932: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65932/ [18:05:00] Project beta-update-databases-eqiad build #65933: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65933/ [19:05:01] Project beta-update-databases-eqiad build #65934: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65934/ [20:05:01] Project beta-update-databases-eqiad build #65935: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65935/ [20:19:21] Project beta-update-databases-eqiad build #65936: 15ABORTED in 58 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65936/ [20:26:48] Project beta-update-databases-eqiad build #65937: 15ABORTED in 6 min 48 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/65937/ [20:39:30] 10Beta-Cluster-Infrastructure: deployment-prep hourly databases update times out - https://phabricator.wikimedia.org/T332507 (10TheresNoTime) **deployment-db11** ` Mar 19 20:10:30 deployment-db11 mysqld[14965]: 2023-03-19 20:10:30 5858284 [Warning] Access denied for user 'wikiadmin'@'172.16.%' to database 'mains... [20:42:36] TheresNoTime: databases are evil [20:43:31] they are, and my experience with mysql replicas and the such is very lacking.. I wouldn't feel comfortable trying to fix it [20:43:54] TheresNoTime: reclone [20:44:01] zabe: has done it a few times [20:53:23] I think I fixed it [20:53:24] .... [20:54:35] How TheresNoTime [20:55:05] 10Beta-Cluster-Infrastructure: deployment-prep hourly databases update times out - https://phabricator.wikimedia.org/T332507 (10TheresNoTime) //killing// the stuck mysqld process on deployment-db13 and restarting the service has brought mysql back up on db13.. it's now replicating (it's //very// behind but catch... [21:00:02] RhinosF1: luck, and ^ [21:00:50] Cool [21:01:00] Hammers also work [21:06:32] 10Beta-Cluster-Infrastructure: deployment-prep hourly databases update times out - https://phabricator.wikimedia.org/T332507 (10TheresNoTime) It's caught up, and https://integration.wikimedia.org/ci/view/Beta/job/beta-update-databases-eqiad/65938/console is running successfully (//so far..//) [22:10:20] !log (beta) `GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, REFERENCES, INDEX, ALTER, CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE VIEW, SHOW VIEW, CREATE ROUTINE, ALTER ROUTINE, EVENT, TRIGGER ON mainstash.* TO 'wikiadmin'@'172.16.%';` T322469 [22:10:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:10:23] T322469: DBError: Access denied for user 'wikiadmin'@'172.16.%' to database 'mainstash' on Beta Cluster - https://phabricator.wikimedia.org/T322469 [22:15:34] 10Beta-Cluster-Infrastructure, 10Performance-Team (Radar), 10Wikimedia-production-error: DBError: Access denied for user 'wikiadmin'@'172.16.%' to database 'mainstash' on Beta Cluster - https://phabricator.wikimedia.org/T322469 (10TheresNoTime) >>! In T322469#8708351, @Stashbot wrote: > {nav icon=file, name=... [22:34:15] !log (beta) `samtar@deployment-kafka-logging01:~$ sudo systemctl restart kafka.service`, noticed beta-logs wasn't showing logs from ~14:00UTC in T332507 [22:34:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:34:18] T332507: deployment-prep hourly databases update times out - https://phabricator.wikimedia.org/T332507 [22:40:00] 10Beta-Cluster-Infrastructure: deploymentwiki site still present in sites table - https://phabricator.wikimedia.org/T331739 (10TheresNoTime) 05Open→03Resolved a:03TheresNoTime [23:08:09] 10Beta-Cluster-Infrastructure: Puppet failing on deployment-sessionstore04 due to lack of free memory - https://phabricator.wikimedia.org/T332524 (10TheresNoTime)