[00:00:45] 06Release-Engineering-Team, 06Content-Transform-Team, 10Quibble, 10Catalyst (PatchDemo), and 3 others: Not adding trailing / to autoload patches breaks Parsoid CI - https://phabricator.wikimedia.org/T400299#11109281 (10cscott) Ok, then it should be safe to re-apply the original patch. ^ [05:35:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [05:35:39] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T402611 (10phaultfinder) 03NEW [05:40:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [05:57:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [05:57:36] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T402611#11109439 (10phaultfinder) [06:02:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [09:27:13] 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 07Performance Issue: CI jobs hanging for ~20 mins - https://phabricator.wikimedia.org/T402606#11109895 (10Lucas_Werkmeister_WMDE) I don’t think this job was hanging? (Timestamps below are in UTC.) ` … 23:08:36 Running com... [09:39:27] (03CR) 10Hashar: [C:03+2] "Thanks for the review!" [integration/config] - 10https://gerrit.wikimedia.org/r/1180552 (owner: 10Hashar) [09:40:54] (03Merged) 10jenkins-bot: jjb: simplify branchdeploy jobs definition [integration/config] - 10https://gerrit.wikimedia.org/r/1180552 (owner: 10Hashar) [10:10:10] 10GitLab (Project Migration), 06Community-Tech, 06translatewiki.net, 10WS Export, 13Patch-For-Review: Migrate ws-export repo from GitHub to GitLab - https://phabricator.wikimedia.org/T395398#11110063 (10Samwilson) [10:10:29] 10GitLab (Project Migration), 06Community-Tech, 06translatewiki.net, 10WS Export, 13Patch-For-Review: Migrate ws-export repo from GitHub to GitLab - https://phabricator.wikimedia.org/T395398#11110065 (10Samwilson) 05In progress→03Resolved This is all now done. [11:41:55] 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure), 07Performance Issue: CI jobs hanging for ~20 mins - https://phabricator.wikimedia.org/T402606#11110356 (10Reedy) Maybe not.. But the Math ones are still slow (which may be another reason) and hitting the 1 hour timeout... [14:18:20] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, 10Wikifunctions, and 3 others: ${{SCRIPT, template="wikimedia.template"}} is not sending a correct template - https://phabricator.wikimedia.org/T400286#11110768 (10DSantamaria) p:05Triage→03High [14:19:06] 10Gerrit, 10ProveIt-Gadget: Users of ProvieIt gadget get a 403 Forbidden fetching i18n files from Gerrit/Gitiles - https://phabricator.wikimedia.org/T394916#11110770 (10Sophivorus) @Iniquity I just modified the development version as mentioned: English messages are hard-coded and the translations are fetch... [14:44:54] hey Releng! I'm having a blubber problem where the environment variable I set in `base` only seems to work for the first variant. My blubberfile is at https://phabricator.wikimedia.org/P81707 . The copy command works if I hard-code the version number, any suggestions? [14:55:22] inflatador: my guess is that `copies` does not expand env variables in its `source` parameter (or any other parameter) [14:55:57] whereas in the first usage the builder `command` is being passed env vars [14:57:16] ah...OK. Maybe I can make a symlink in the `compile` container that I can use to copy [15:00:03] and for that wget, you should probably follow it with a sha checksum verification. But I am picky :] [15:01:33] I agree, although I guess I don't understand why you'd need to if you're already downloading over HTTPS [15:02:00] (03CR) 10Hashar: [C:03+2] "Success!" [integration/config] - 10https://gerrit.wikimedia.org/r/1179783 (https://phabricator.wikimedia.org/T388056) (owner: 10Jeena Huneidi) [15:02:05] If I'm missing something LMK, that is a serious question ;) [15:02:34] inflatador: my fringe theory is the remote tarball could get corrupted/replaced/ whatever [15:03:26] (03Merged) 10jenkins-bot: Update wikilambda daily selenium job [integration/config] - 10https://gerrit.wikimedia.org/r/1179783 (https://phabricator.wikimedia.org/T388056) (owner: 10Jeena Huneidi) [15:03:45] for the integration/config images, we have the artifacts checked with gpg or some sha256sum [15:03:58] cause who knows! [15:04:18] anyway, as I said, that is probably a nitpick [15:04:19] :] [15:04:30] hashar makes sense, more about corruption than security [15:05:27] wait until you have to deal with npmjs :] [15:06:21] then I agree, that opensearch-project organization on GitHub is unlikely to be hijacked [15:06:47] and we are then less likely to rebuild the image during the short (hopefully) time frame during which it remains hijacked [15:06:50] oh yeah, my thought isn't that it wouldn't be hijacked, just that the hijacker could change the checksums too [15:07:27] well with a sha 256 checksum, good luck finding a collision. I am not sure how possible it is [15:07:28] anyway [15:07:47] there is a reason I am not doing security, it is an endless trail of paranoia [15:07:48] yeah, I agree with that too. But I'd be the one who thought it was a problem on my end and "fixed" it ;p [15:08:50] anyway, it costs nothing to add a checksum so may as well [15:09:13] [The certificate of the remote website https://bankofcountry.country/ is invalid? Someone is trying to tamper your connection and you should reject it. Accept? (y/N)] [15:09:18] yeah [15:09:23] as I said, I am picky :] [15:09:40] and for that one it probably does not matter much [15:31:50] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, 10Wikifunctions, and 3 others: ${{SCRIPT, template="wikimedia.template"}} is not sending a correct template - https://phabricator.wikimedia.org/T400286#11111010 (10vaughnwalters) For my purposes, emails are being sent on e... [16:04:43] 10Continuous-Integration-Infrastructure, 07Zuul, 10Wikidata, 10ci-test-error (WMF-deployed Build Failure), and 2 others: Wikibase secondary CI broken: repo/sql/postgres/archives/patch-wb_changes-change_timestamp.sql does not match expected SQL - https://phabricator.wikimedia.org/T400918#11111100 (10Lucas_We... [16:45:21] hashar: fyi https://github.com/linux-test-project/lcov/issues/432 [16:45:35] lcov for php-compile doesn't work entirely te same on debian 13 vs debian 11 [16:55:07] (03PS1) 10Krinkle: dockerfiles: update php8{1,2,3,4}-compile to publish lcov.info file [integration/config] - 10https://gerrit.wikimedia.org/r/1181162 [17:15:12] (03PS1) 10RLazarus: helm-linter: Bump for envoy 1.26.8 and switch to bullseye-wikimedia [integration/config] - 10https://gerrit.wikimedia.org/r/1181166 (https://phabricator.wikimedia.org/T402584) [17:16:41] (03PS1) 10RLazarus: jjb: Update to helm-linter:0.7.3 to pick up envoy-future 1.26.8 [integration/config] - 10https://gerrit.wikimedia.org/r/1181167 (https://phabricator.wikimedia.org/T402584) [17:32:25] (03PS1) 10Krinkle: CoveragePage: Add support for lvoc.info files [integration/docroot] - 10https://gerrit.wikimedia.org/r/1181169 [17:35:13] (03PS2) 10Krinkle: CoveragePage: Add support for lvoc.info files [integration/docroot] - 10https://gerrit.wikimedia.org/r/1181169 [17:36:52] (03CR) 10Krinkle: [C:03+2] CoveragePage: Add support for lvoc.info files [integration/docroot] - 10https://gerrit.wikimedia.org/r/1181169 (owner: 10Krinkle) [17:37:23] (03Merged) 10jenkins-bot: CoveragePage: Add support for lvoc.info files [integration/docroot] - 10https://gerrit.wikimedia.org/r/1181169 (owner: 10Krinkle) [18:18:26] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for harshkush07 - https://phabricator.wikimedia.org/T402675 (10Harsh_Kushwaha07) 03NEW [18:20:23] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for harshkush07 - https://phabricator.wikimedia.org/T402676 (10Harsh_Kushwaha07) 03NEW [18:21:01] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Ebenezer_Rao - https://phabricator.wikimedia.org/T402677 (10EbenezerRao) 03NEW [18:21:29] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for harshkush07 - https://phabricator.wikimedia.org/T402678 (10Harsh_Kushwaha07) 03NEW [18:21:32] 10GitLab (Account Approval), 06Release-Engineering-Team, 06RoadToWiki: Requesting GitLab account activation for Ebenezer_Rao - https://phabricator.wikimedia.org/T402677#11111462 (10EbenezerRao) [18:21:41] 10GitLab (Account Approval), 06Release-Engineering-Team, 06RoadToWiki: Requesting GitLab account activation for harshkush07 - https://phabricator.wikimedia.org/T402678#11111463 (10Harsh_Kushwaha07) [18:26:30] 10GitLab (Account Approval), 06Release-Engineering-Team, 06RoadToWiki: Requesting GitLab account activation for harshkush07 - https://phabricator.wikimedia.org/T402678#11111478 (10Pppery) [18:26:32] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for harshkush07 - https://phabricator.wikimedia.org/T402676#11111481 (10Pppery) →14Duplicate dup:03T402678 [18:26:35] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for harshkush07 - https://phabricator.wikimedia.org/T402675#11111482 (10Pppery) →14Duplicate dup:03T402678 [18:41:02] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Harshkush07 - https://phabricator.wikimedia.org/T402681 (10Harsh_Kushwaha07) 03NEW [18:41:25] 10GitLab (Account Approval), 06Release-Engineering-Team, 06RoadToWiki: Requesting GitLab account activation for Harshkush07 - https://phabricator.wikimedia.org/T402681#11111550 (10Harsh_Kushwaha07) [18:44:34] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for harshkush07 - https://phabricator.wikimedia.org/T402675#11111554 (10JJMC89) [18:45:36] 10GitLab (Account Approval), 06Release-Engineering-Team, 06RoadToWiki: Requesting GitLab account activation for Harshkush07 - https://phabricator.wikimedia.org/T402681#11111555 (10Aklapper) Please stop creating more and more duplicates. [18:45:44] 10GitLab (Account Approval), 06Release-Engineering-Team, 06RoadToWiki: Requesting GitLab account activation for Harshkush07 - https://phabricator.wikimedia.org/T402681#11111558 (10Aklapper) →14Duplicate dup:03T402678 [18:45:45] 10GitLab (Account Approval), 06Release-Engineering-Team, 06RoadToWiki: Requesting GitLab account activation for harshkush07 - https://phabricator.wikimedia.org/T402678#11111556 (10JJMC89) [18:45:48] 10GitLab (Account Approval), 06Release-Engineering-Team, 06RoadToWiki: Requesting GitLab account activation for harshkush07 - https://phabricator.wikimedia.org/T402678#11111560 (10Aklapper) [18:46:19] 10GitLab (Account Approval), 06Release-Engineering-Team, 06RoadToWiki: Requesting GitLab account activation for Harshkush07 - https://phabricator.wikimedia.org/T402681#11111566 (10Harsh_Kushwaha07) I need to contribute to Videocuttool https://gitlab.wikimedia.org/cloudvps-repos/videocuttool/VideoCutTool... [18:51:47] 10GitLab (Account Approval), 06Release-Engineering-Team, 06RoadToWiki: Requesting GitLab account activation for Harshkush07 - https://phabricator.wikimedia.org/T402681#11111599 (10Adityakumar0545) Hi @Aklapper, This is Aditya, Mentor from #RoadToWiki. We are currently onboarding new members to contribute... [18:56:08] 10GitLab (Account Approval), 06Release-Engineering-Team, 06RoadToWiki: Requesting GitLab account activation for Ebenezer_Rao - https://phabricator.wikimedia.org/T402677#11111648 (10Adityakumar0545) I am vouching for this User @EbenezerRao. This is Aditya, Mentor from #RoadToWiki. We are currently onboarding... [18:56:12] (03PS1) 10Krinkle: CoveragePage: Improve alignment of clover/lcov links [integration/docroot] - 10https://gerrit.wikimedia.org/r/1181180 [18:56:18] (03CR) 10Krinkle: [C:03+2] CoveragePage: Improve alignment of clover/lcov links [integration/docroot] - 10https://gerrit.wikimedia.org/r/1181180 (owner: 10Krinkle) [18:56:47] (03Merged) 10jenkins-bot: CoveragePage: Improve alignment of clover/lcov links [integration/docroot] - 10https://gerrit.wikimedia.org/r/1181180 (owner: 10Krinkle) [18:57:15] (03CR) 10Krinkle: [C:03+2] dockerfiles: update php8{1,2,3,4}-compile to publish lcov.info file [integration/config] - 10https://gerrit.wikimedia.org/r/1181162 (owner: 10Krinkle) [18:58:31] 10GitLab (Account Approval), 06Release-Engineering-Team, 06RoadToWiki: Requesting GitLab account activation for harshkush07 - https://phabricator.wikimedia.org/T402678#11111663 (10Adityakumar0545) Apologies for any inconvenience. This is Aditya, Mentor from #RoadToWiki. We are currently onboarding new membe... [18:58:37] (03Merged) 10jenkins-bot: dockerfiles: update php8{1,2,3,4}-compile to publish lcov.info file [integration/config] - 10https://gerrit.wikimedia.org/r/1181162 (owner: 10Krinkle) [19:02:36] (03CR) 10Krinkle: [C:03+2] "Screenshots:" [integration/docroot] - 10https://gerrit.wikimedia.org/r/1181180 (owner: 10Krinkle) [19:04:04] !log Updating docker-pkg files on contint primary for https://gerrit.wikimedia.org/r/1181162 [19:04:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:12:07] 10GitLab (Account Approval), 06Release-Engineering-Team, 06RoadToWiki: Requesting GitLab account activation for Ebenezer_Rao - https://phabricator.wikimedia.org/T402677#11111715 (10thcipriani) 05Open→03Resolved a:03thcipriani Thanks @Adityakumar0545 @EbenezerRao you should be able to login to Git... [19:13:04] 10GitLab (Account Approval), 06Release-Engineering-Team, 06RoadToWiki: Requesting GitLab account activation for harshkush07 - https://phabricator.wikimedia.org/T402678#11111720 (10thcipriani) 05Open→03Resolved a:03thcipriani Thanks @Adityakumar0545 @Harsh_Kushwaha07 you should be able to login to... [19:13:34] 10GitLab (Account Approval), 06Release-Engineering-Team, 06RoadToWiki: Requesting GitLab account activation for Ebenezer_Rao - https://phabricator.wikimedia.org/T402677#11111723 (10Adityakumar0545) Thanks @thcipriani for the assitance! [19:15:14] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Harshkush07 - https://phabricator.wikimedia.org/T402681#11111743 (10Adityakumar0545) [19:16:23] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for harshkush07 - https://phabricator.wikimedia.org/T402678#11111755 (10Adityakumar0545) [19:16:32] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Ebenezer_Rao - https://phabricator.wikimedia.org/T402677#11111756 (10Adityakumar0545) [19:22:19] (03PS1) 10Krinkle: utils: Use tox in docker-updates to avoid ModuleNotFoundError [integration/config] - 10https://gerrit.wikimedia.org/r/1181182 [19:23:42] (03CR) 10CI reject: [V:04-1] utils: Use tox in docker-updates to avoid ModuleNotFoundError [integration/config] - 10https://gerrit.wikimedia.org/r/1181182 (owner: 10Krinkle) [19:24:36] (03PS2) 10Krinkle: utils: Use tox in docker-updates to avoid ModuleNotFoundError [integration/config] - 10https://gerrit.wikimedia.org/r/1181182 [19:25:28] (03PS1) 10Krinkle: jjb: Apply php8X-compile updates to publish lcov.info file [integration/config] - 10https://gerrit.wikimedia.org/r/1181183 [19:25:33] (03PS2) 10Krinkle: jjb: Apply php8X-compile updates to publish lcov.info file [integration/config] - 10https://gerrit.wikimedia.org/r/1181183 [19:25:54] (03CR) 10CI reject: [V:04-1] utils: Use tox in docker-updates to avoid ModuleNotFoundError [integration/config] - 10https://gerrit.wikimedia.org/r/1181182 (owner: 10Krinkle) [19:27:35] (03CR) 10Krinkle: "The lack of .py extension made it escape the lint step previously. Fun." [integration/config] - 10https://gerrit.wikimedia.org/r/1181182 (owner: 10Krinkle) [19:30:30] (03CR) 10Krinkle: [C:03+2] "`" [integration/config] - 10https://gerrit.wikimedia.org/r/1181183 (owner: 10Krinkle) [19:31:52] (03Merged) 10jenkins-bot: jjb: Apply php8X-compile updates to publish lcov.info file [integration/config] - 10https://gerrit.wikimedia.org/r/1181183 (owner: 10Krinkle) [19:41:30] Yay, https://doc.wikimedia.org/cover/ now displays a link to php-excimer and indicates its percentage [19:41:48] the report was already there before, but because it lacked a clover.xml file, it wasn't linked to from anywhere. [19:42:03] so it was basically a hidden subdirectory you could guess by URL only [19:42:25] we now generate the index based on glob */clover.xml and */lcov.info combined. [21:00:38] 10Beta-Cluster-Infrastructure, 10GitLab (CI & Job Runners), 06collaboration-services, 10m3api: Unable to connect to Beta Cluster from WMCS GitLab CI runners - https://phabricator.wikimedia.org/T397591#11112003 (10LucasWerkmeister) [21:52:33] (03CR) 10Nanocloud: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/1181183 (owner: 10Krinkle) [21:54:29] (03CR) 10Nanocloud: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/1181183 (owner: 10Krinkle) [23:44:40] (03CR) 10Krinkle: "Is this problem limited to the future when we re-use mwext job for mwskin repos? Or is there a bug today as well?" [integration/config] - 10https://gerrit.wikimedia.org/r/1180908 (https://phabricator.wikimedia.org/T288396) (owner: 10Hashar) [23:54:39] 10Continuous-Integration-Infrastructure (Zuul upgrade), 10Release-Engineering-Team (Doing 😎), 07Essential-Work: Investigate zuul migrate script used for the Zuul v3 OpenStack migration - https://phabricator.wikimedia.org/T383273#11112238 (10thcipriani) [23:56:57] 10Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.45.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T396376#11112241 (10thcipriani) [23:57:52] 10Release-Engineering-Team (Doing 😎), 07Essential-Work: Catalyst: VisualEditor/VisualEditor checking out master when different branch specified - https://phabricator.wikimedia.org/T400925#11112242 (10thcipriani)