[03:39:45] PROBLEM - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is CRITICAL: CRITICAL - Certificate gerrit.wikimedia.org expires in 7 day(s) (Sat 28 May 2022 08:33:22 PM GMT +0000). https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [03:42:01] RECOVERY - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is OK: OK - Certificate gerrit.wikimedia.org will expire on Wed 27 Jul 2022 08:27:52 PM GMT +0000. https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [05:17:37] (03CR) 10DannyS712: bootstrap: add script to install/update local version of scap (035 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/793410 (https://phabricator.wikimedia.org/T307086) (owner: 10Jaime Nuche) [08:26:47] 10Continuous-Integration-Config, 10phan, 10PHP 8.1 support: Phan 3.2.6 crashed on composer-php81-docker test - https://phabricator.wikimedia.org/T308692 (10Reedy) 05Open→03Resolved [09:11:56] 10Gerrit, 10SRE: Icinga Check SSL might have a time based race condition - https://phabricator.wikimedia.org/T308908 (10hashar) [09:13:22] 10Continuous-Integration-Config, 10phan, 10PHP 8.1 support: Phan 3.2.6 crashed on composer-php81-docker test - https://phabricator.wikimedia.org/T308692 (10hashar) a:05hashar→03TheresNoTime Reassigning to @TheresNoTime who found out the reason above (T308692#7939339). I have merely pushed the button ;) [10:05:16] 10Gerrit, 10SRE: Icinga Check SSL might have a time based race condition - https://phabricator.wikimedia.org/T308908 (10RhinosF1) {T293826} maybe? [10:07:13] PROBLEM - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is CRITICAL: CRITICAL - Certificate gerrit.wikimedia.org expires in 7 day(s) (Sat 28 May 2022 08:33:22 PM GMT +0000). https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [10:09:25] RECOVERY - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is OK: OK - Certificate gerrit.wikimedia.org will expire on Wed 27 Jul 2022 08:27:52 PM GMT +0000. https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [14:11:56] !log Icinga reports `Gerrit Health Check SSL Expiry` errors filed as T308908 [14:11:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:11:58] T308908: Icinga Check SSL might have a time based race condition - https://phabricator.wikimedia.org/T308908 [15:10:31] maintenance-disconnect-full-disks build 387984 integration-agent-docker-1036 (/: 30%, /srv: 95%, /var/lib/docker: 52%): OFFLINE due to disk space [15:15:58] maintenance-disconnect-full-disks build 387985 integration-agent-docker-1036 (/: 30%, /srv: 49%, /var/lib/docker: 50%): RECOVERY disk space OK [15:24:14] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Jenkins: quibble-vendor-mysql-php72-selenium-docker: "cannot create directory ‘log’: Permission denied" - https://phabricator.wikimedia.org/T308927 (10TheresNoTime) [15:24:44] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure: quibble-vendor-mysql-php72-selenium-docker: "cannot create directory ‘log’: Permission denied" - https://phabricator.wikimedia.org/T308927 (10TheresNoTime) [15:31:32] I feel like I may have broken something - https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/793797 - doing a "recheck" at the same time as getting a +2 has left the gate-and-submit in a stuck state I think? I've manually submitted that revert [15:31:54] oh, no.. the jobs are running now.. [15:32:33] something weird definitely happened there, the gerrit interface shows both of our comments under my review [15:35:44] wasn't that a bug reported earlier [15:35:56] with comments posted very close together [15:37:30] T308369 [15:37:30] T308369: Gerrit attributed my comments to jenkins-bot - https://phabricator.wikimedia.org/T308369 [15:38:13] either way, as long as I've not broken anything (have checked, all seems good), I'm happy :) [15:38:27] yes [15:40:08] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10ci-test-error (WMF-deployed Build Failure): quibble-vendor-mysql-php72-selenium-docker: "cannot create directory ‘log’: Permission denied" - https://phabricator.wikimedia.org/T308927 (10Majavah) p:05Tri... [15:53:11] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10ci-test-error (WMF-deployed Build Failure): quibble-vendor-mysql-php72-selenium-docker: "cannot create directory ‘log’: Permission denied" - https://phabricator.wikimedia.org/T308927 (10hashar) 05Open→... [16:40:30] maintenance-disconnect-full-disks build 388002 integration-agent-docker-1036 (/: 30%, /srv: 96%, /var/lib/docker: 53%): OFFLINE due to disk space [16:45:43] maintenance-disconnect-full-disks build 388003 integration-agent-docker-1036 (/: 30%, /srv: 49%, /var/lib/docker: 50%): RECOVERY disk space OK [16:50:40] hey! I already pinged about this in -operations, but repeating here: requesting RelEng approval to deploy https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/793797/ ASAP to fix GlobalRename leaving unattached accounts [17:19:04] TheresNoTime: There was a change earlier this week that made Zuul no longer run a CI job if there is already a +2 [17:19:18] I suspect that may have broken 'recheck'? cc hashar [17:19:28] taavi: I can approve it soon. Just wait until I settle [17:19:41] The purpose of that change was that when you re-+2 something, yuou only want 'gate' not 'gate' and 'test' both. [17:25:39] taavi: in the meantime. I send an email to global renamers mailing list [17:25:56] Amir1: thanks! lmk when I can deploy [17:27:03] meanwhile I'm looking into how big of a mess it has created [17:32:35] taavi: let me know if you need anything, I'm deploying something unimportant, happy to help or make space anytime [19:23:42] (03CR) 10jerkins-bot: [V: 04-1] build: Updating composer dependencies [tools/release] - 10https://gerrit.wikimedia.org/r/794727 (owner: 10Libraryupgrader) [19:38:25] zuul seems to be giving lots of "This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository" errors [19:49:42] I am blocked on working on my hackathon project because it seems Jenkins is complaining about cross-repo dependencies not been mergable, but I do not have any: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/793934 [19:49:49] anything I can do to fix this? [19:50:26] it's a zuul bug, you can't really do anything but recheck again [19:50:46] it failed twice [19:50:56] and I cannot recheck myself :-( [19:50:56] hmm [19:51:16] unless somebody adds me to https://gerrit.wikimedia.org/g/integration/config//%2B/HEAD/zuul/layout.yaml [19:53:11] (03PS1) 10Zabe: zuul: Add Mitar to the allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/794756 [19:53:19] (03CR) 10jerkins-bot: [V: 04-1] zuul: Add Mitar to the allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/794756 (owner: 10Zabe) [19:53:27] (03PS2) 10Zabe: zuul: Add Mitar to the allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/794756 [19:53:34] (03CR) 10jerkins-bot: [V: 04-1] zuul: Add Mitar to the allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/794756 (owner: 10Zabe) [19:54:37] (03CR) 10Zabe: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/794756 (owner: 10Zabe) [19:54:57] aparently it got worse :/ [19:55:38] thanks for trying! [19:57:06] really unfortunate timing :-( [19:57:49] does Zull's logs say anything? [19:58:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos - https://phabricator.wikimedia.org/T308943 (10Zabe) [20:09:15] 10Phabricator (Upstream): upstream request timeout on Phabricator - https://phabricator.wikimedia.org/T308946 (10AlexisJazz) [20:11:06] 10Phabricator (Upstream): upstream request timeout on Phabricator - https://phabricator.wikimedia.org/T308946 (10AlexisJazz) [20:14:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos - https://phabricator.wikimedia.org/T308943 (10Legoktm) Same here: https://gerrit.wikimedia.org/r/794759 - my guess is... [20:28:04] 10Phabricator (Search): upstream request timeout on Phabricator - https://phabricator.wikimedia.org/T308946 (10Aklapper) p:05Triage→03Low Cannot reproduce on https://phabricator.wikimedia.org/search/ . https://phabricator.wikimedia.org/search/query/_ndFuPQejqYu/#R lists results. [20:30:59] Project beta-scap-sync-world build #52230: 04FAILURE in 6 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/52230/ [20:36:09] Yippee, build fixed! [20:36:10] Project beta-scap-sync-world build #52231: 09FIXED in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/52231/ [20:52:47] (03PS3) 10Samtar: zuul: Add Mitar to the allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/794756 (owner: 10Zabe) [20:52:53] (03CR) 10jenkins-bot: zuul: Add Mitar to the allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/794756 (owner: 10Zabe) [20:54:10] (03PS4) 10Samtar: zuul: Add Mitar to the allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/794756 (owner: 10Zabe) [20:54:16] (03CR) 10jenkins-bot: zuul: Add Mitar to the allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/794756 (owner: 10Zabe) [20:54:49] well I'm stumped [20:55:01] TheresNoTime, see T308943 [20:55:01] T308943: CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos - https://phabricator.wikimedia.org/T308943 [20:57:56] hey TheresNoTime [20:58:09] wotcha [20:59:03] TheresNoTime: did you have a chance to get to hackathon? [21:16:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos - https://phabricator.wikimedia.org/T308943 (10TheresNoTime) `zuul-merger` is [[ https://icinga.wikimedia.org/cgi-bin/i... [21:22:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos - https://phabricator.wikimedia.org/T308943 (10TheresNoTime) p:05Triage→03Unbreak! Looking closer, this is affectin... [21:39:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos - https://phabricator.wikimedia.org/T308943 (10TheresNoTime) [21:56:39] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos - https://phabricator.wikimedia.org/T308943 (10bd808) There are a lot of errors similar to this in /var/log/zuul/merger... [22:00:04] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos - https://phabricator.wikimedia.org/T308943 (10hashar) a:03hashar o/ checking [22:00:53] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos - https://phabricator.wikimedia.org/T308943 (10TheresNoTime) (https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Blu... [22:05:30] 10Phabricator (Search): upstream request timeout on Phabricator - https://phabricator.wikimedia.org/T308946 (10AlexisJazz) >>! In T308946#7947338, @Aklapper wrote: > Cannot reproduce on https://phabricator.wikimedia.org/search/ . https://phabricator.wikimedia.org/search/query/_ndFuPQejqYu/#R lists results. I do... [22:06:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos - https://phabricator.wikimedia.org/T308943 (10hashar) From the Gerrit log https://logstash.wikimedia.org/app/dashboard... [22:06:35] PROBLEM - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is CRITICAL: CRITICAL - Certificate gerrit.wikimedia.org expires in 6 day(s) (Sat 28 May 2022 08:33:22 PM GMT +0000). https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [22:11:05] RECOVERY - Gerrit Health Check SSL Expiry on gerrit.wikimedia.org is OK: OK - Certificate gerrit.wikimedia.org will expire on Wed 27 Jul 2022 08:27:52 PM GMT +0000. https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [22:13:15] (03PS5) 10Samtar: zuul: Add Mitar to the allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/794756 (owner: 10Zabe) [22:14:23] looks good hashar.. [22:15:08] TheresNoTime: yeah it should be good now ;) [22:16:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos - https://phabricator.wikimedia.org/T308943 (10hashar) 05Open→03Resolved Should be good now after I have restarted... [22:16:28] (03CR) 10Samtar: [C: 03+1] "🥳 CI issues fixed, may as well +1 this" [integration/config] - 10https://gerrit.wikimedia.org/r/794756 (owner: 10Zabe) [22:18:07] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos - https://phabricator.wikimedia.org/T308943 (10TheresNoTime) Just confirming that https://gerrit.wikimedia.org/r/c/inte... [22:18:35] (03PS6) 10Zabe: zuul: Add Mitar to the allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/794756 [22:35:43] hashar: how many SSH connections does the bot normally have open? Having an icinga alert for "connections == 4" sounds like a good idea ^^ [22:43:04] so what is the process of getting https://gerrit.wikimedia.org/r/c/integration/config/+/794756/ merged and deployed? it would help me use CI more effectively [22:49:58] Hey Mitar :) I gave it a +1 code review when I did the recheck after the CI broke, but really it's just a matter of waiting for someone with +2 access to review it [22:50:40] I see [22:52:40] It shouldn't take too long :) [22:54:50] (03CR) 10Samtar: zuul: Add Mitar to the allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/794756 (owner: 10Zabe) [23:00:55] legoktm: ref T274359, h/nowlan was going to get it deployed a little while back, but there's a few concerns about what it could break (: [23:00:56] T274359: Mobile REST API delivers year old+ content for very select pages - https://phabricator.wikimedia.org/T274359 [23:01:41] I can imagine [23:02:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: CI fails with 'This change or one of its cross-repo dependencies was unable to be automatically merged' for a lot of repos - https://phabricator.wikimedia.org/T308943 (10hashar) Thanks for the confirmation @TheresNoTime ! [23:02:55] (03CR) 10Legoktm: [C: 03+2] zuul: Add Mitar to the allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/794756 (owner: 10Zabe) [23:03:18] If my name wasn't on the patch I'd be all for the "just go for it and see who screams" approach! [23:04:49] (03Merged) 10jenkins-bot: zuul: Add Mitar to the allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/794756 (owner: 10Zabe) [23:05:14] so after https://gerrit.wikimedia.org/r/c/integration/config/+/794756/ is merged, does new configuration have to be deployed? [23:05:48] !log deployed https://gerrit.wikimedia.org/r/c/integration/config/+/794756/ [23:05:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:05:54] Mitar: you should be set now! [23:07:05] awesome! [23:09:59] Seen a few more instances of T308927, but it could just been old jobs from around the same time.. [23:09:59] T308927: quibble-vendor-mysql-php72-selenium-docker: "cannot create directory ‘log’: Permission denied" - https://phabricator.wikimedia.org/T308927 [23:14:07] TheresNoTime: maybe it would be less daunting if the blacklist were removed gradually instead of all in one go? [23:14:50] I suppose we could just remove ANI and test that? [23:14:56] like, I doubt "Talk:United_States_presidential_election,_2016" is still as big a problem as it used to be. And I do know that "Cyberbot is creating 90% of null edits" was fixed a while back [23:15:26] That's... yeah that's much smarter than me removing it all in one go.... [23:33:11] (03CR) 10Samtar: "Recheck" [tools/release] - 10https://gerrit.wikimedia.org/r/794727 (owner: 10Libraryupgrader) [23:36:00] (03CR) 10Samtar: [C: 03+2] build: Updating composer dependencies [tools/release] - 10https://gerrit.wikimedia.org/r/794727 (owner: 10Libraryupgrader)