[01:33:03] (03PS1) 10Krinkle: zuul: Add CI jobs for less.php (php74+ including php82) [integration/config] - 10https://gerrit.wikimedia.org/r/879687 [01:33:26] (03CR) 10Krinkle: [C: 03+2] zuul: Add CI jobs for less.php (php74+ including php82) [integration/config] - 10https://gerrit.wikimedia.org/r/879687 (owner: 10Krinkle) [01:34:30] (03Merged) 10jenkins-bot: zuul: Add CI jobs for less.php (php74+ including php82) [integration/config] - 10https://gerrit.wikimedia.org/r/879687 (owner: 10Krinkle) [01:35:49] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/879687 [01:35:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [04:08:41] https://gitenterprise.me/2023/01/13/2023-new-year-and-opportunities-for-gerritforge-and-gerrit-code-review/ [04:37:30] 10GitLab (CI & Job Runners), 10Patch-For-Review: Allow variants of rustlang/rust:nightly images on GitLab CI - https://phabricator.wikimedia.org/T326515 (10Legoktm) [04:37:36] 10GitLab (CI & Job Runners), 10mwbot-rs: Unable to run Rust coverage reports in GitLab CI - https://phabricator.wikimedia.org/T325947 (10Legoktm) [10:23:31] (03PS1) 10Dreamy Jazz: Run tests for mediawiki/extensions/ArrayFunctions [integration/config] - 10https://gerrit.wikimedia.org/r/879754 [10:39:11] (03CR) 10Hashar: [C: 03+2] Run tests for mediawiki/extensions/ArrayFunctions [integration/config] - 10https://gerrit.wikimedia.org/r/879754 (owner: 10Dreamy Jazz) [10:40:19] (03Merged) 10jenkins-bot: Run tests for mediawiki/extensions/ArrayFunctions [integration/config] - 10https://gerrit.wikimedia.org/r/879754 (owner: 10Dreamy Jazz) [10:41:29] !log Reloading Zuul for https://gerrit.wikimedia.org/r/c/integration/config/+/879754 [10:41:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:42:13] (03CR) 10Dreamy Jazz: "Thanks for the quick merge!" [integration/config] - 10https://gerrit.wikimedia.org/r/879754 (owner: 10Dreamy Jazz) [10:42:15] Krinkle: thanks for the Gerrit link. We will finalise the design document for the transition to project-specific change numbers in Gerrit v3.8. ;) [10:42:19] big renumbering ahead! :-] [10:43:08] (03CR) 10Hashar: [C: 03+2] "You are welcome! I have done a `recheck` on the extension patch https://gerrit.wikimedia.org/r/c/mediawiki/extensions/ArrayFunctions/+/879" [integration/config] - 10https://gerrit.wikimedia.org/r/879754 (owner: 10Dreamy Jazz) [12:11:18] (03PS1) 10Daniel Kinzler: New features for pre-commit-phpcs [tools/code-utils] - 10https://gerrit.wikimedia.org/r/879774 [13:17:08] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: Align and refactor GitLab restore scripts - https://phabricator.wikimedia.org/T326315 (10Jelto) Restore on GitLab replica `gitlab2002` failed because the rsync transfer of the backup took more time and was overlapping the beginning of the re... [13:57:19] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.40-release: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146 (10Bugreporter2) Can I put in a specific request that Extension:TemplateStyles should be in... [14:48:14] 10Release-Engineering-Team, 10SRE, 10SRE-OnFire, 10serviceops-collab, 10Sustainability: Remove old scap repositories from deploy1002 - https://phabricator.wikimedia.org/T309162 (10hashar) Not sure about the comparison but on deploy1002 we have roughly: `lines=10 $ (cd /srv/deployment && find . -mindepth... [14:57:26] (03CR) 10Hashar: [C: 03+1] "I will let Derick review and give the final approval ;)" [tools/code-utils] - 10https://gerrit.wikimedia.org/r/879774 (owner: 10Daniel Kinzler) [16:01:15] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group, 10Epic, 10MW-1.40-release: Expand the set of bundled extensions and skins in MediaWiki 1.40 - https://phabricator.wikimedia.org/T317146 (10Dinoguy1000) @Bugreporter2 Good point, somehow it wasn't listed on [[ https://www.mediaw... [16:11:44] I just got two “No valid crumb was included in the request” Jetty errors when following three build/console links from Zuul (the third request still worked) [16:12:13] reloading the page fixed the problem in one tab, but I don’t remember seeing this error before, so I’ve left the other tab open for now in case someone wants to look into it ^^ [16:12:21] was something Jetty-related upgraded recently? [16:12:42] screenshot: https://usercontent.irccloud-cdn.com/file/dXiDkwli/image.png [16:22:43] Lucas_WMDE: we haven't touched Jenkins since november iirc [16:22:53] ok [16:22:56] the requests are served via our ats/varnish stack [16:23:05] so maybe some header get mangled in the process sometime [16:23:34] I guess it could just be random [16:24:03] I’m okay with ignoring it, just wanted to mention it in case it was relevant to something ^^ [16:24:10] or your CRSF token has expired somehow [16:24:17] might be worth to logout / clear cookie and log back again [16:24:23] I do see the errors on the server side [16:24:33] well, it works after reloading (and I’m still logged in) [16:24:35] Jan 13 16:10:43 contint2001 jenkins[17543]: WARNING: [hudson.security.csrf.CrumbFilter doFilter] Found invalid crumb xxxxxxxx. If you are calling this URL with a script, please use the API Token instead. More information: https://www.jenkins.io/redirect/crumb-cannot-be-used-for-script [16:24:39] it just happened on two initial page loads [16:24:44] Jan 13 16:10:43 contint2001 jenkins[17543]: WARNING: [hudson.security.csrf.CrumbFilter doFilter] No valid crumb was included in request for /ci/job/quibble-vendor-mysql-php74-noselenium-docker/76991/logText/progressiveHtml by Lucas Werkmeister (WMDE). Returning 403. [16:24:50] hm, interesting [16:24:52] so your token was "wrong" [16:25:18] and there is a second request for /ci/job/wmf-quibble-selenium-php74-docker/15950/logText/progressiveHtml which had the same token given [16:25:20] and it got rejected [16:25:42] I think that matches the two tabs where I got the error [16:25:49] (15950 was definitely one of them) [16:26:00] maybe it had expired [16:26:07] afaict that's just what happens if your session has timed out [16:26:13] if the token has a short time to live [16:26:20] if you open a tab and let it run in the background [16:26:24] after sometime the token is expired [16:26:36] but jenkins still issues background http requests to refresh the console view [16:26:42] using the now expired token [16:26:55] ^^^ don't quote me on that, it is purely an hypothesis ;) [16:27:07] hm, but this was when I initially loaded the logs [16:27:30] :-\ [16:28:08] might be worth filing in our phabricator and one (== me) can dig in upstream bug tracker [16:30:03] ok, sure [16:31:42] there are some doc upstream at https://docs.cloudbees.com/docs/cloudbees-ci-kb/latest/client-and-managed-masters/instance-navigation-often-fails-with-crumb-issuer-errors [16:31:59] which mentions the check could fail if the proxy does not preserve the client IP or the X-Forwarded-For is miss configured [16:32:17] 10Continuous-Integration-Infrastructure: Spontaneous “No valid crumb was included in the request” Jetty error when loading Jenkins logs - https://phabricator.wikimedia.org/T326970 (10Lucas_Werkmeister_WMDE) [16:32:22] created ^ [16:32:31] :-] [16:32:39] no promise on that one, but it is good to have it filed! [16:32:47] * hashar enters a meeting [16:33:49] 10Continuous-Integration-Infrastructure: Spontaneous “No valid crumb was included in the request” Jetty error when loading Jenkins logs - https://phabricator.wikimedia.org/T326970 (10Lucas_Werkmeister_WMDE) I think the three builds would have been [1](https://integration.wikimedia.org/ci/job/quibble-vendor-mysql... [16:33:50] ok, sure! [16:44:32] 10Continuous-Integration-Infrastructure: Spontaneous “No valid crumb was included in the request” Jetty error when loading Jenkins logs - https://phabricator.wikimedia.org/T326970 (10hashar) [16:45:40] 10Continuous-Integration-Infrastructure, 10Jenkins: Spontaneous “No valid crumb was included in the request” Jetty error when loading Jenkins logs - https://phabricator.wikimedia.org/T326970 (10hashar) That might be an issue with our ATS/Varnish cache stripping the XFF header, not passing the client IP or stri... [20:04:59] 10Phabricator, 10serviceops-collab: create aphlict2001 (Phabricator realtime notifications codfw) - https://phabricator.wikimedia.org/T322369 (10Dzahn) created VM with: ` dzahn@cumin2002:~$ sudo cookbook sre.ganeti.makevm --vcpus 1 --memory 2 --disk 20 --network private --cluster codfw --group D aphlict2001 ` [20:43:39] (03CR) 10D3r1ck01: [C: 03+2] New features for pre-commit-phpcs [tools/code-utils] - 10https://gerrit.wikimedia.org/r/879774 (owner: 10Daniel Kinzler) [20:43:45] (03CR) 10D3r1ck01: [C: 03+2] "Nice!" [tools/code-utils] - 10https://gerrit.wikimedia.org/r/879774 (owner: 10Daniel Kinzler) [20:44:09] (03CR) 10D3r1ck01: [C: 03+2] New features for pre-commit-phpcs (031 comment) [tools/code-utils] - 10https://gerrit.wikimedia.org/r/879774 (owner: 10Daniel Kinzler) [20:44:29] (03Merged) 10jenkins-bot: New features for pre-commit-phpcs [tools/code-utils] - 10https://gerrit.wikimedia.org/r/879774 (owner: 10Daniel Kinzler) [20:57:44] 10Continuous-Integration-Config, 10Release-Engineering-Team (Seen): Wikimedia Portals needs libpng-dev for npm-browser-node-6 tests - https://phabricator.wikimedia.org/T186117 (10Jdforrester-WMF) 05Open→03Declined The portals repo now uses the Pipeline to self-manage its CI instead.