[01:27:41] 10GitLab (CI & Job Runners), 10mwbot-rs: GitLab CI keeps SIGKILLing Rust jobs - https://phabricator.wikimedia.org/T349786 (10CodeReviewBot) legoktm updated https://gitlab.wikimedia.org/repos/mwbot-rs/rust-ci-pipeline/-/merge_requests/10 Reduce memory consumption by lowering concurrency [01:28:51] 10GitLab (CI & Job Runners), 10mwbot-rs: GitLab CI keeps SIGKILLing Rust jobs - https://phabricator.wikimedia.org/T349786 (10Legoktm) Thanks @dancy, that was the clue I needed! Rust tries to automatically parallelize for the number of CPUs, so reducing the concurrency to 1 seems to keep it under the memory limit. [01:29:15] 10GitLab (CI & Job Runners), 10mwbot-rs: GitLab CI keeps SIGKILLing Rust jobs - https://phabricator.wikimedia.org/T349786 (10CodeReviewBot) legoktm merged https://gitlab.wikimedia.org/repos/mwbot-rs/rust-ci-pipeline/-/merge_requests/10 Reduce memory consumption by lowering concurrency [07:23:24] Project beta-scap-sync-world build #126747: 04FAILURE in 8 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126747/ [07:28:57] Project beta-scap-sync-world build #126748: 04STILL FAILING in 2 min 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126748/ [07:29:21] 10Phabricator, 10Legalpad, 10WMF-Legal, 10WMF-NDA-Requests, 10User-AKlapper: Clarify if NDAs (to access #WMF-NDA protected Phab tasks) are on paper or in Legalpad's L2 or both - https://phabricator.wikimedia.org/T349595 (10Aklapper) Sure! Also added `@JbuattiWMF`. [07:37:41] Project beta-scap-sync-world build #126749: 04STILL FAILING in 2 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126749/ [07:46:15] Project beta-scap-sync-world build #126750: 04STILL FAILING in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126750/ [07:56:26] Project beta-scap-sync-world build #126751: 04STILL FAILING in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126751/ [08:06:31] Project beta-scap-sync-world build #126752: 04STILL FAILING in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126752/ [08:13:42] <_joe_> I need some assistance to edit access rights on a gerrit repo [08:14:15] <_joe_> I want to grant a specific user push rights to master [08:27:02] Project beta-scap-sync-world build #126753: 04STILL FAILING in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126753/ [08:29:44] (03PS1) 10Giuseppe Lavagetto: Review access change [docker-images/production-images] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/969164 [08:30:46] Only one specific user to have push rights? Hmm [08:31:17] Project beta-scap-sync-world build #126754: 04STILL FAILING in 2 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126754/ [08:41:15] Project beta-scap-sync-world build #126755: 04STILL FAILING in 6 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126755/ [09:07:53] Project beta-scap-sync-world build #126756: 04STILL FAILING in 22 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126756/ [09:08:00] 10Phabricator (Upstream), 10Upstream: Disable "Contact Numbers" settings panel (and SMS support functionality that we do not use) - https://phabricator.wikimedia.org/T240109 (10Aklapper) a:03Aklapper [09:12:46] Project beta-scap-sync-world build #126757: 04STILL FAILING in 2 min 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126757/ [09:16:26] Project beta-scap-sync-world build #126758: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126758/ [09:26:29] Project beta-scap-sync-world build #126759: 04STILL FAILING in 1 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126759/ [09:29:36] 10Phabricator (Upstream), 10Release-Engineering-Team (Social Piranhas 🐟), 10Upstream: Disable "Contact Numbers" settings panel (and SMS support functionality that we do not use) - https://phabricator.wikimedia.org/T240109 (10Aklapper) 05Open→03Stalled [09:36:32] Project beta-scap-sync-world build #126760: 04STILL FAILING in 1 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126760/ [09:46:20] Project beta-scap-sync-world build #126761: 04STILL FAILING in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126761/ [09:56:23] Project beta-scap-sync-world build #126762: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126762/ [10:04:26] 10Phabricator (Upstream), 10Upstream: Listing tickets "Closed By: Current Viewer" while logged out causes PhabricatorTypeaheadInvalidTokenException: This datasource ("PhabricatorPeopleUserFunctionDatasource") can not evaluate the function "viewer(...)". - https://phabricator.wikimedia.org/T217984 (10Aklapper)... [10:04:30] 10Phabricator (Upstream), 10Upstream: Listing tickets "Closed By: Current Viewer" while logged out causes PhabricatorTypeaheadInvalidTokenException: This datasource ("PhabricatorPeopleUserFunctionDatasource") can not evaluate the function "viewer(...)". - https://phabricator.wikimedia.org/T217984 (10Aklapper) [10:06:22] Project beta-scap-sync-world build #126763: 04STILL FAILING in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126763/ [10:16:24] Project beta-scap-sync-world build #126764: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126764/ [10:26:28] Project beta-scap-sync-world build #126765: 04STILL FAILING in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126765/ [10:31:32] 10Phabricator (Upstream), 10Upstream: Call to a member function updateDatasourceTokens() on null at PhabricatorProjectFulltextEngine.php:20 - https://phabricator.wikimedia.org/T345563 (10Aklapper) [10:31:34] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream, 10User-brennen: Hovercards: (RuntimeException) Undefined index: objectPHID - https://phabricator.wikimedia.org/T346876 (10Aklapper) [10:36:29] Project beta-scap-sync-world build #126766: 04STILL FAILING in 1 min 20 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126766/ [10:46:28] Project beta-scap-sync-world build #126767: 04STILL FAILING in 1 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126767/ [10:48:49] 10Phabricator (Upstream), 10CSS, 10Upstream: Custom Policy window overflows when task name is too long - https://phabricator.wikimedia.org/T316308 (10Aklapper) `` expands its width this case (it set to `width: 100%`) and somehow the total width of the entire overlay dialog c... [10:56:21] Project beta-scap-sync-world build #126768: 04STILL FAILING in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126768/ [11:06:28] Project beta-scap-sync-world build #126769: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126769/ [11:15:27] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services, 10Patch-For-Review: Migrate to using new GitLab CI runner authentication scheme - https://phabricator.wikimedia.org/T344951 (10CodeReviewBot) jelto opened https://gitlab.wikimedia.org/repos/releng/gitlab... [11:16:26] Project beta-scap-sync-world build #126770: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126770/ [11:20:21] 10Release-Engineering-Team, 10docker-pkg, 10serviceops: Attach opencontainers image metadata to docker images - https://phabricator.wikimedia.org/T345070 (10JMeybohm) [11:21:08] 10Release-Engineering-Team, 10docker-pkg, 10serviceops: Attach opencontainers image metadata to docker images - https://phabricator.wikimedia.org/T345070 (10JMeybohm) [11:26:31] Project beta-scap-sync-world build #126771: 04STILL FAILING in 1 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126771/ [11:36:40] Project beta-scap-sync-world build #126772: 04STILL FAILING in 1 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126772/ [11:46:28] Project beta-scap-sync-world build #126773: 04STILL FAILING in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126773/ [11:53:22] (03PS1) 10Sophivorus: Zuul: [mediawiki/extensions/CloudflarePurge] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/969318 [11:56:29] Project beta-scap-sync-world build #126774: 04STILL FAILING in 1 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126774/ [12:06:16] Project beta-scap-sync-world build #126775: 04STILL FAILING in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126775/ [12:16:23] Project beta-scap-sync-world build #126776: 04STILL FAILING in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126776/ [12:26:19] Project beta-scap-sync-world build #126777: 04STILL FAILING in 1 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126777/ [12:36:27] Project beta-scap-sync-world build #126778: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126778/ [12:46:19] Project beta-scap-sync-world build #126779: 04STILL FAILING in 1 min 11 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126779/ [12:54:21] 10GitLab (Project Migration), 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services, and 3 others: Migrate active repositories in Phabricator Differential to GitLab - https://phabricator.wikimedia.org/T191182 (10Aklapper) Acting as a Phabricator admin, I unlocked most non... [12:56:14] Project beta-scap-sync-world build #126780: 04STILL FAILING in 1 min 6 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126780/ [13:06:15] Project beta-scap-sync-world build #126781: 04STILL FAILING in 1 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126781/ [13:16:15] Project beta-scap-sync-world build #126782: 04STILL FAILING in 1 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126782/ [13:26:23] Project beta-scap-sync-world build #126783: 04STILL FAILING in 1 min 15 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126783/ [13:36:24] Project beta-scap-sync-world build #126784: 04STILL FAILING in 1 min 16 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126784/ [13:46:16] Project beta-scap-sync-world build #126785: 04STILL FAILING in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126785/ [13:56:15] Project beta-scap-sync-world build #126786: 04STILL FAILING in 1 min 7 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126786/ [14:06:16] Project beta-scap-sync-world build #126787: 04STILL FAILING in 1 min 5 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126787/ [14:16:20] Project beta-scap-sync-world build #126788: 04STILL FAILING in 1 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126788/ [14:26:31] Project beta-scap-sync-world build #126789: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126789/ [14:44:30] _joe_: I can do that for you. Hit me w/ details. [14:45:32] <_joe_> dancy: care to review https://gerrit.wikimedia.org/r/c/operations/docker-images/production-images/+/969164 ? [14:46:41] OK [14:47:12] Project beta-scap-sync-world build #126790: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126790/ [14:50:19] (03CR) 10Ahmon Dancy: [V: 03+2 C: 03+2] Review access change [docker-images/production-images] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/969164 (owner: 10Giuseppe Lavagetto) [14:50:31] _joe_: Merged. Give 'er a test. [14:52:32] Project beta-scap-sync-world build #126791: 04STILL FAILING in 1 min 35 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126791/ [14:54:03] I'm going to attempt to restart deployment-snapshot03.deployment-prep.eqiad1.wikimedia.cloud using the Horizon UI [14:56:24] Project beta-scap-sync-world build #126792: 04STILL FAILING in 1 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126792/ [14:57:04] How to restart the interesting little jenkins-bot here? https://gerrit.wikimedia.org/r/c/mediawiki/extensions/RegexFun/+/842004 [14:58:47] !log Rebooted deployment-snapshot03.deployment-prep.eqiad1.wikimedia.cloud to fix beta-scap-sync-world [14:58:50] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:59:20] valerio-bozzolan: Try commenting "recheck" in Gerrit. [15:04:57] 10Release-Engineering-Team (Social Piranhas 🐟), 10Patch-For-Review, 10Release, 10Train Deployments: 1.42.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T348355 (10dancy) 05Open→03Resolved [15:10:12] 10Beta-Cluster-Infrastructure: beta cluster down - https://phabricator.wikimedia.org/T329592 (10dcaro) [15:16:46] Yippee, build fixed! [15:16:46] Project beta-scap-sync-world build #126793: 09FIXED in 11 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/126793/ [15:21:15] <_joe_> dancy: thanks! (sorry was in a meeting at the time) [15:37:27] dancy: Thanks. But I'm somehow inclined to think that it's stuck [15:38:06] Probably I'm not allow-listed https://www.mediawiki.org/wiki/Continuous_integration/Allow_list [15:40:28] * dancy takes a look [15:41:43] In the meantime, I'll start the recheck [15:46:03] Would anyone have an idea or theory why the integration-agent-docker-*.integration.eqiad1.wikimedia.cloud machines would pull some Git repos from Phab Diffusion more than once per minute? [15:46:37] (...and which project tag to add in Phab as I'm about to create a ticket about this? :P ) [15:50:36] 10Release-Engineering-Team, 10RESTBase, 10serviceops-radar: RESTBase scap deployment failed - https://phabricator.wikimedia.org/T349318 (10thcipriani) >>! In T349318#9284718, @Jgiannelos wrote: > We just run a succesful scap deploy on restbase. 🎉 happy to hear it :) [15:51:03] (03PS1) 10Ahmon Dancy: Zuul: Add Valerio Bozzolan to email_allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/969374 [15:52:52] (03CR) 10Jforrester: Zuul: Add Valerio Bozzolan to email_allowlist (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/969374 (owner: 10Ahmon Dancy) [15:53:26] dancy: I'll fix and deploy? [15:53:49] (03PS2) 10Ahmon Dancy: Zuul: Add Valerio Bozzolan to email_allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/969374 [15:54:01] Alright fixed but happy to turn over deployment to you [15:54:02] (03CR) 10Jforrester: [C: 03+2] Zuul: Add Valerio Bozzolan to email_allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/969374 (owner: 10Ahmon Dancy) [15:54:05] *Already [15:54:09] Thanks! [15:55:13] (03CR) 10Ahmon Dancy: Zuul: Add Valerio Bozzolan to email_allowlist (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/969374 (owner: 10Ahmon Dancy) [15:55:27] (03CR) 10Jforrester: [C: 03+2] Zuul: Add Valerio Bozzolan to email_allowlist (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/969374 (owner: 10Ahmon Dancy) [15:55:38] (03Merged) 10jenkins-bot: Zuul: Add Valerio Bozzolan to email_allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/969374 (owner: 10Ahmon Dancy) [15:55:52] !log Zuul: Add Valerio Bozzolan to email_allowlist [15:55:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:56:07] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/CloudflarePurge] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/969318 (owner: 10Sophivorus) [15:56:09] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/CloudflarePurge] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/969318 (owner: 10Sophivorus) [15:57:28] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CloudflarePurge] Add basic CI [integration/config] - 10https://gerrit.wikimedia.org/r/969318 (owner: 10Sophivorus) [15:57:40] !log Zuul: [mediawiki/extensions/CloudflarePurge] Add basic CI [15:57:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:57:59] (03PS2) 10Jforrester: Add VisualEditor as dependency to AtMentions [integration/config] - 10https://gerrit.wikimedia.org/r/969114 (owner: 10Hslater) [15:58:02] (03CR) 10Jforrester: [C: 03+2] Add VisualEditor as dependency to AtMentions [integration/config] - 10https://gerrit.wikimedia.org/r/969114 (owner: 10Hslater) [15:59:27] (03Merged) 10jenkins-bot: Add VisualEditor as dependency to AtMentions [integration/config] - 10https://gerrit.wikimedia.org/r/969114 (owner: 10Hslater) [15:59:38] !log Zuul: Add VisualEditor as dependency to AtMentions [15:59:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:00:17] (03PS2) 10Jforrester: Zuul: Archive the WikiLexicalData extension [integration/config] - 10https://gerrit.wikimedia.org/r/968949 (https://phabricator.wikimedia.org/T271379) (owner: 10Zoranzoki21) [16:00:24] (03CR) 10Jforrester: [C: 03+2] Zuul: Archive the WikiLexicalData extension [integration/config] - 10https://gerrit.wikimedia.org/r/968949 (https://phabricator.wikimedia.org/T271379) (owner: 10Zoranzoki21) [16:00:29] (03PS2) 10Jforrester: Zuul: Archive the RevisionCommentSupplement extension [integration/config] - 10https://gerrit.wikimedia.org/r/968991 (https://phabricator.wikimedia.org/T343616) (owner: 10Zoranzoki21) [16:00:31] (03CR) 10Jforrester: [C: 03+2] Zuul: Archive the RevisionCommentSupplement extension [integration/config] - 10https://gerrit.wikimedia.org/r/968991 (https://phabricator.wikimedia.org/T343616) (owner: 10Zoranzoki21) [16:03:06] (03Merged) 10jenkins-bot: Zuul: Archive the WikiLexicalData extension [integration/config] - 10https://gerrit.wikimedia.org/r/968949 (https://phabricator.wikimedia.org/T271379) (owner: 10Zoranzoki21) [16:03:11] (03Merged) 10jenkins-bot: Zuul: Archive the RevisionCommentSupplement extension [integration/config] - 10https://gerrit.wikimedia.org/r/968991 (https://phabricator.wikimedia.org/T343616) (owner: 10Zoranzoki21) [16:03:24] !log Zuul: Archive the WikiLexicalData extension for T271379 [16:03:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:03:28] T271379: Archive the WikiLexicalData extension - https://phabricator.wikimedia.org/T271379 [16:03:34] !log Zuul: Archive the RevisionCommentSupplement extension for T343616 [16:03:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:03:38] T343616: Archive the RevisionCommentSupplement extension - https://phabricator.wikimedia.org/T343616 [16:06:25] 10Continuous-Integration-Config, 10Diffusion, 10Phabricator: integration-agent-docker machines excessively pull some Wikibase related Git repos in Diffusion - https://phabricator.wikimedia.org/T349921 (10Aklapper) [16:17:49] 10GitLab, 10Release-Engineering-Team (Social Piranhas 🐟): Create development environment for hacking on Zuul/GitLab - https://phabricator.wikimedia.org/T349923 (10dduvall) [16:25:24] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Language-Team (Language-2023-October-December), 10affects-translatewiki.net: jenkins.job=mediawiki-i18n-check-docker fails without a real error - https://phabricator.wikimedia.org/T331098 (10thcipriani) >>! In T331098#9214975, @Nikerabbit wrote... [16:33:38] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Language-Team (Language-2023-October-December), 10affects-translatewiki.net: jenkins.job=mediawiki-i18n-check-docker fails without a real error - https://phabricator.wikimedia.org/T331098 (10Nikerabbit) Yeah we push a lot of patches to Gerrit i... [17:29:44] 10GitLab, 10Release-Engineering-Team (Social Piranhas 🐟): Create development environment for hacking on Zuul/GitLab - https://phabricator.wikimedia.org/T349923 (10dduvall) Created a new repo at https://gitlab.wikimedia.org/repos/releng/gitlab-dev for GitLab/Zuul hacking. It needs a GitLab runner which I'll wor... [17:29:59] 10GitLab, 10Release-Engineering-Team (Social Piranhas 🐟): Create development environment for hacking on Zuul/GitLab - https://phabricator.wikimedia.org/T349923 (10dduvall) p:05Triage→03Medium [18:39:41] 10Beta-Cluster-Infrastructure, 10Cloud-VPS: New Cloud VPS instance has "Failed to start Execute cloud user/final script." in its error log - https://phabricator.wikimedia.org/T349937 (10Tgr) [18:40:53] 10Beta-Cluster-Infrastructure, 10Cloud-VPS: New Cloud VPS instance has "Failed to start Execute cloud user/final script." in its error log - https://phabricator.wikimedia.org/T349937 (10Tgr) I also got this error as part of the login message, but it seems unrelated: ` -bash: warning: setlocale: LC_ALL: cannot... [18:49:52] 10Phabricator, 10Legalpad, 10WMF-Legal, 10WMF-NDA-Requests, 10User-AKlapper: Clarify if NDAs (to access #WMF-NDA protected Phab tasks) are on paper or in Legalpad's L2 or both - https://phabricator.wikimedia.org/T349595 (10KFrancis) Thank you so much! I'll have Jim review that page and I'll get back to... [19:22:11] 10WikimediaDebug, 10MediaWiki-Platform-Team, 10Observability-Tracing, 10MW-1.41-notes (1.41.0-wmf.27; 2023-09-19): Add support for request tracing to WikimediaDebug browser extension - https://phabricator.wikimedia.org/T340573 (10CDanis) Thanks so much @pmiazga ! This is great to see and the spec chosen f... [21:12:51] 10Beta-Cluster-Infrastructure, 10Cloud-VPS: New Cloud VPS instance has "Failed to start Execute cloud user/final script." in its error log - https://phabricator.wikimedia.org/T349937 (10Tgr) 05Open→03Invalid Okay so I found https://wikitech.wikimedia.org/wiki/Help:Standalone_puppetmaster#Step_2:_Setup_a_pu... [21:25:40] 10Beta-Cluster-Infrastructure, 10serviceops: Unable to upload files on Beta Commons - https://phabricator.wikimedia.org/T340908 (10Tgr) >>! In T340908#9285076, @Tgr wrote: > In production, rdb1 / rdb2 / rdb3 (which point to rdb1009 and rdb1011) use the [[https://gerrit.wikimedia.org/g/operations/puppet/+/ba56d... [22:26:45] 10Beta-Cluster-Infrastructure: Beta log host should have logrotate for error.log - https://phabricator.wikimedia.org/T349943 (10Tgr) [22:30:09] 10Beta-Cluster-Infrastructure, 10Wikifunctions: Wikifunctions spams beta error log with "socket_sendto(): Host lookup failed" - https://phabricator.wikimedia.org/T349944 (10Tgr) [22:51:14] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10Wikifunctions: Wikifunctions spams beta error log with "socket_sendto(): Host lookup failed" - https://phabricator.wikimedia.org/T349944 (10Tgr) wikifunctionswiki uses `prometheus-labmon.eqiad.wmnet` like everything else. Which is an internal domain... [22:52:35] 10Beta-Cluster-Infrastructure, 10observability: Statsd broken on the beta cluster ("socket_sendto(): Host lookup failed") - https://phabricator.wikimedia.org/T349944 (10Tgr) [23:06:50] 10Beta-Cluster-Infrastructure, 10serviceops: Unable to upload files on Beta Commons - https://phabricator.wikimedia.org/T340908 (10Tgr) It still fails with `Could not acquire locks on server rdb1.` :( I'm trying to figure out why but beta DOS-ing its own error log (T349944) doesn't help. [23:10:09] 10Beta-Cluster-Infrastructure, 10observability, 10Patch-For-Review: Statsd broken on the beta cluster ("socket_sendto(): Host lookup failed") - https://phabricator.wikimedia.org/T349944 (10Tgr) Disabling statsd got the error log under control. I'll leave it to the Observability team whether they want statsd... [23:25:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Social Piranhas 🐟), 10Zuul: Proof-of-concept: Run MediaWiki tests in latest Zuul - https://phabricator.wikimedia.org/T349214 (10thcipriani) @dancy got the first successful quibble run in the new zuul today: http://zuul.dancy.cloud.releng.te... [23:25:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Social Piranhas 🐟), 10Zuul: Proof-of-concept: Run MediaWiki tests in latest Zuul - https://phabricator.wikimedia.org/T349214 (10thcipriani) [23:25:52] 10GitLab (Misc), 10Release-Engineering-Team (Priority Backlog 📥): Investigate and document "Depends-On" in GitLab - https://phabricator.wikimedia.org/T349872 (10thcipriani) [23:27:06] 10GitLab, 10Release-Engineering-Team (Social Piranhas 🐟): Create development environment for hacking on Zuul/GitLab - https://phabricator.wikimedia.org/T349923 (10thcipriani) [23:27:09] 10GitLab (Misc), 10Release-Engineering-Team (Priority Backlog 📥): Investigate and document "Depends-On" in GitLab - https://phabricator.wikimedia.org/T349872 (10thcipriani) [23:31:37] 10Beta-Cluster-Infrastructure, 10serviceops: Unable to upload files on Beta Commons - https://phabricator.wikimedia.org/T340908 (10Tgr) `PHP Warning: RedisException: Connection timed out`, a firewall issue I guess? Redis is listening for all IPs: ` tgr@deployment-rdb01:~$ sudo lsof -iTCP -sTCP:LISTEN -n -P |... [23:40:08] 10Beta-Cluster-Infrastructure, 10serviceops: Unable to upload files on Beta Commons - https://phabricator.wikimedia.org/T340908 (10Tgr) >>! In T340908#9288678, @Tgr wrote: > iptables has a reasonable-looking rule, but it's not getting any new packets: > ` > tgr@deployment-rdb01:~$ sudo iptables --list -n -v |...