[01:35:10] 10Continuous-Integration-Infrastructure, 07Jenkins: Wikibase job in Jenkins do not include the full log - https://phabricator.wikimedia.org/T420347#11721398 (10hashar) I have reproduced the issue locally and surely it shows up in the production Jenkins logs: `counterexample WARNING: [hudson.ExpressionFactory2$... [01:55:15] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream: Wikibase job in Jenkins do not include the full log - https://phabricator.wikimedia.org/T420347#11721436 (10hashar) @Krinkle for info, in the Jenkins collapsible section plugin, whenever a section name uses a captured group that contains `$` or... [01:55:39] nothing like java hacking at 3am :b [01:56:37] I have dreamed of the solution (literally dreamed it yes), enough it woke me up and I HAD to fix it NOW [01:56:42] more or less related https://xkcd.com/386/ [02:07:56] 06Release-Engineering-Team (Seen), 10Quibble, 07Essential-Work: Jenkins collapsible sections are deeply nested in Quibble output - https://phabricator.wikimedia.org/T303056#11721463 (10hashar) 05Resolved→03Open I am reopening this. It was worked around by removing the parallel-npm-install system but **th... [02:27:42] hashar: nice! [02:28:52] Krinkle: thanks :] [07:16:31] FIRING: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:16:44] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T420432 (10phaultfinder) 03NEW [07:24:47] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T420432#11721739 (10ABran-WMF) [07:24:49] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow - https://phabricator.wikimedia.org/T420189#11721740 (10ABran-WMF) [07:24:56] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4 - https://phabricator.wikimedia.org/T420432#11721741 (10ABran-WMF) p:05Triage→03High [07:26:31] RESOLVED: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [07:28:34] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow - https://phabricator.wikimedia.org/T420189#11721745 (10ABran-WMF) Testing connection re-use with the new tweaks had th... [07:32:06] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4 - https://phabricator.wikimedia.org/T420432#11721758 (10ABran-WMF) 05Open→03Resolved a:03ABran-WMF [07:58:53] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic: Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow - https://phabricator.wikimedia.org/T420189#11721787 (10ABran-WMF) The [[ https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&ref... [08:07:41] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream: Quibble -c commands cause Jenkins Collapsible Section plugin to erase console output (Wikibase job in Jenkins do not include the full log) - https://phabricator.wikimedia.org/T420347#11721796 (10hashar) [08:08:20] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream: Quibble -c commands cause Jenkins Collapsible Section plugin to erase console output (Wikibase job in Jenkins do not include the full log) - https://phabricator.wikimedia.org/T420347#11721798 (10hashar) I have released it as 1.13.0 [08:32:53] 10Continuous-Integration-Infrastructure, 07Jenkins, 07Upstream: Quibble -c commands cause Jenkins Collapsible Section plugin to erase console output (Wikibase job in Jenkins do not include the full log) - https://phabricator.wikimedia.org/T420347#11721832 (10hashar) 05Open→03Resolved a:03hashar I h... [08:41:24] 06Gerrit-Privilege-Requests, 10GoogleLogin: Maintainership/Review request for GoogleLogin: patch 1074565 has been pending for over a year - https://phabricator.wikimedia.org/T414991#11721835 (10Pastakhov) @Aklapper is there a way to get access to the repo without sending an email to the wikitech-l mailing list... [08:55:42] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use - https://phabricator.wikimedia.org/T417998#11721863 (10hashar) 05Resolved→03Stalled Reopening as {nav Stalled} given the ATS connecti... [09:06:50] 06Gerrit-Privilege-Requests, 10GoogleLogin: Maintainership/Review request for GoogleLogin: patch 1074565 has been pending for over a year - https://phabricator.wikimedia.org/T414991#11721898 (10Peachey88) Hi @Pastakhov, that section relates to Wikiteq requesting users to be added to the wikiteq gerrit user gro... [10:39:17] 10Phabricator-Bot-Requests: Create mcp-phabricator bot account - https://phabricator.wikimedia.org/T420446 (10kostajh) 03NEW [10:42:08] 10Phabricator-Bot-Requests: Create mcp-phabricator bot account - https://phabricator.wikimedia.org/T420446#11722306 (10kostajh) It's not clear to me if a "read-only account" is possible but T85289 seems to describe the same use case. [10:42:39] 10Phabricator-Bot-Requests: Create mcp-phabricator bot account - https://phabricator.wikimedia.org/T420446#11722309 (10kostajh) [10:44:39] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11722322 (10Aklapper) My take is that implementing such a restriction would de-facto exclude new contributors, as "editing a task" also includes "commenting" (if I rememb... [12:18:10] (03Abandoned) 10Sbisson: ArticleGuidance: register dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/1254289 (owner: 10Sbisson) [12:59:22] (03CR) 10Ebernhardson: "Thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/1254282 (https://phabricator.wikimedia.org/T420407) (owner: 10Ebernhardson) [13:09:31] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic: Gerrit: Debug connection re-use on Gerrit's httpd causing Gerrit interface to be very slow - https://phabricator.wikimedia.org/T420189#11722913 (10ABran-WMF) After a conversation with @hashar where I was thinking about bumping `http... [13:41:03] (03CR) 10Ebernhardson: [C:04-1] "sadly this one is different, glent runs in the hadoop cluster and has to use java 8" [integration/config] - 10https://gerrit.wikimedia.org/r/1254341 (https://phabricator.wikimedia.org/T420407) (owner: 10Jforrester) [13:50:24] 10Phabricator-Bot-Requests: Create mcp-phabricator bot account - https://phabricator.wikimedia.org/T420446#11723098 (10Aklapper) I don't see any way to achieve creating a read-only account currently. I'd be happy to be proven wrong. May welcome an upstream ticket in https://we.phorge.it/ to discuss the use case,... [13:55:20] (03PS2) 10Jforrester: jjb: Add support for running Java 25 for search projects [integration/config] - 10https://gerrit.wikimedia.org/r/1254342 [13:55:20] (03PS2) 10Jforrester: Zuul: [search/*] Add experimental Java 25 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1254343 [13:55:35] (03Abandoned) 10Jforrester: Zuul: [search/glent] Drop Java 8 testing, add Java 17 testing [integration/config] - 10https://gerrit.wikimedia.org/r/1254341 (https://phabricator.wikimedia.org/T420407) (owner: 10Jforrester) [13:56:05] (03PS3) 10Jforrester: jjb: Drop most search maven jobs on Java 8, now unused; split Glent into own [integration/config] - 10https://gerrit.wikimedia.org/r/1254335 (https://phabricator.wikimedia.org/T420407) [14:33:12] (03CR) 10Jforrester: [C:03+2] jjb: Drop most search maven jobs on Java 8, now unused; split Glent into own [integration/config] - 10https://gerrit.wikimedia.org/r/1254335 (https://phabricator.wikimedia.org/T420407) (owner: 10Jforrester) [14:34:52] (03Merged) 10jenkins-bot: jjb: Drop most search maven jobs on Java 8, now unused; split Glent into own [integration/config] - 10https://gerrit.wikimedia.org/r/1254335 (https://phabricator.wikimedia.org/T420407) (owner: 10Jforrester) [14:36:49] (03CR) 10Jforrester: [C:03+2] jjb: Add support for running Java 25 for search projects [integration/config] - 10https://gerrit.wikimedia.org/r/1254342 (owner: 10Jforrester) [14:38:19] (03CR) 10CI reject: [V:04-1] jjb: Add support for running Java 25 for search projects [integration/config] - 10https://gerrit.wikimedia.org/r/1254342 (owner: 10Jforrester) [14:38:34] (03PS3) 10Jforrester: jjb: Add support for running Java 25 for search projects [integration/config] - 10https://gerrit.wikimedia.org/r/1254342 [14:39:01] (03CR) 10Jforrester: jjb: Add support for running Java 25 for search projects [integration/config] - 10https://gerrit.wikimedia.org/r/1254342 (owner: 10Jforrester) [14:39:57] (03PS4) 10Jforrester: jjb: Add support for running Java 25 for search projects [integration/config] - 10https://gerrit.wikimedia.org/r/1254342 [14:40:09] (03CR) 10Jforrester: [C:03+2] jjb: Add support for running Java 25 for search projects [integration/config] - 10https://gerrit.wikimedia.org/r/1254342 (owner: 10Jforrester) [14:41:43] (03Merged) 10jenkins-bot: jjb: Add support for running Java 25 for search projects [integration/config] - 10https://gerrit.wikimedia.org/r/1254342 (owner: 10Jforrester) [14:42:35] (03PS3) 10Jforrester: Zuul: [search/*] Add experimental Java 25 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1254343 [14:50:03] (03CR) 10Jforrester: [C:03+2] Zuul: [search/*] Add experimental Java 25 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1254343 (owner: 10Jforrester) [14:50:28] (03Abandoned) 10Jforrester: Zuul: Temporarily make wikilambda-catalyst-end-to-end non-voting again [integration/config] - 10https://gerrit.wikimedia.org/r/1251377 (owner: 10Jforrester) [14:50:47] (03CR) 10Jforrester: [C:03+2] Zuul: [GrowthExperiments] drop duplicate VisualEditor dep [integration/config] - 10https://gerrit.wikimedia.org/r/1253488 (owner: 10Hashar) [14:51:56] (03Merged) 10jenkins-bot: Zuul: [search/*] Add experimental Java 25 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1254343 (owner: 10Jforrester) [14:52:19] (03Merged) 10jenkins-bot: Zuul: [GrowthExperiments] drop duplicate VisualEditor dep [integration/config] - 10https://gerrit.wikimedia.org/r/1253488 (owner: 10Hashar) [14:52:34] !log Zuul: [search/*] Add experimental Java 25 jobs [14:52:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:52:42] !log Zuul: [GrowthExperiments] drop duplicate VisualEditor dep [14:52:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:52:45] (03CR) 10Jforrester: [C:03+2] utils: zuul-dependencies: list deps repositories [integration/config] - 10https://gerrit.wikimedia.org/r/1253449 (owner: 10Hashar) [14:54:00] (03CR) 10Jforrester: Zuul: Handle wmf/next the same way as wmf/branch_cut_pretest (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1250762 (owner: 10Jforrester) [14:54:14] (03Merged) 10jenkins-bot: utils: zuul-dependencies: list deps repositories [integration/config] - 10https://gerrit.wikimedia.org/r/1253449 (owner: 10Hashar) [14:54:16] (03PS3) 10Hashar: Zuul: Handle wmf/next the same way as wmf/branch_cut_pretest [integration/config] - 10https://gerrit.wikimedia.org/r/1250762 (owner: 10Jforrester) [14:54:25] (03CR) 10Jforrester: [C:03+2] Zuul: Handle wmf/next the same way as wmf/branch_cut_pretest [integration/config] - 10https://gerrit.wikimedia.org/r/1250762 (owner: 10Jforrester) [14:54:47] (03PS2) 10Jforrester: Zuul: [mediawiki/extensions/TestKitchen] Don't drop from release branches [integration/config] - 10https://gerrit.wikimedia.org/r/1248853 [14:56:16] (03Merged) 10jenkins-bot: Zuul: Handle wmf/next the same way as wmf/branch_cut_pretest [integration/config] - 10https://gerrit.wikimedia.org/r/1250762 (owner: 10Jforrester) [14:56:44] !log Zuul: Handle wmf/next the same way as wmf/branch_cut_pretest [14:56:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:39:31] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T420479#11723682 (10thcipriani) p:05Triage→03Medium a:03hashar [15:39:40] (03CR) 10Jforrester: Add Python 3.12 and 3.13 testing [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250540 (https://phabricator.wikimedia.org/T419675) (owner: 10Hashar) [15:40:18] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T420480#11723694 (10thcipriani) p:05Triage→03Medium a:03jnuche [15:41:01] James_F: for that python 3.12/3.13, one of the test fail because one of the python lib doess less calls to time.time() than earlier versions [15:41:02] 00:02:26.797 E assert [">>> Start: ..., in 4.000 s"] == [">>> Start: ... in 48.000 s"] [15:41:05] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T420481#11723703 (10thcipriani) p:05Triage→03Medium a:03dancy [15:41:05] I need to fix the test [15:41:12] Ack. [15:41:13] it is mocking time.time [15:41:34] I will probably rewrite it to assert against the full text [15:41:41] aka join all logged messages by newlines [15:41:45] and compare to a block of text [15:41:51] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T420482#11723708 (10thcipriani) p:05Triage→03Medium a:03dduvall [15:41:53] or do a regex match [15:41:57] I don't know yet :-] [15:42:32] I should have kept that change as a WIP when I noticed one test failed [15:43:10] thcipriani: Sorry for being picky about T420483 [15:43:17] T420483: 1.46.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T420483 [15:44:54] James_F: thank you for flagging <3 (I do usually make the tasks for weeks without train and then decline. AND I probably would have missed earth day :D) [15:45:13] TBF it's you that noticed Earth Day meant no train, I think. :-) [15:45:53] I contain the multitudes: detailed focused and sloppy, both [15:46:03] I blame the weather. [15:46:14] too nice here. Distracting. [16:08:43] 10GitLab: GitLab Private Repository Request for: aranyap - https://phabricator.wikimedia.org/T420485 (10aranyap) 03NEW [16:10:13] 10Beta-Cluster-Infrastructure, 10Cassandra, 06Data-Persistence: Cassandra killed by oom-killer and prometheus scrapes failing intermittently on deployment-sessionstore06 - https://phabricator.wikimedia.org/T415021#11723846 (10bd808) 05Open→03Resolved a:03bd808 So here is my sense making: Before the... [16:12:23] 10Beta-Cluster-Infrastructure, 10Cassandra, 06Data-Persistence: Cassandra killed by oom-killer and prometheus scrapes failing intermittently on deployment-sessionstore06 - https://phabricator.wikimedia.org/T415021#11723870 (10bd808) [16:12:25] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-sessionstore06 is down - https://phabricator.wikimedia.org/T420287#11723872 (10bd808) →14Duplicate dup:03T415021 [16:20:34] 10GitLab (Auth & Access), 10Phabricator, 06Release-Engineering-Team (Radar), 06cloud-services-team, 10Toolforge: Look for ways to consolidate "we trust this human" access lists - https://phabricator.wikimedia.org/T364516#11723934 (10brennen) [16:30:39] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11724035 (10Nux) Note that Flyspray has options like that: {F73035455} Not that I would recommend moving to Flyspray (which is mostly dead now), but if Phorge doesn't se... [16:57:52] Hi, releng folks. I maintain a small open-source project, www.storymaps.io, which allows teams to plan work, and one of its features is the ability to pull and push data to Phabricator. However, for WMF staff, it's not possible to hit the Phabricator API remotely. We have to use a workaround. However, I can use the API locally in my own browser session, which tells me the WMF Phabricator API is not disabled, just locked down. Who [16:57:52] would be the best team to speak to about this block to find out whether it's a blanket rule or if there is an allow list I could apply to join? Thanks in advance! [16:59:08] jgleeson: What error message/code are you getting? [16:59:21] It could be blocked by blanket 3rd party IP blocks etc [17:06:09] ^ that'd be my first guess [17:06:53] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11724266 (10Dzahn) I would just like to add that editing other people's tickets is a super common feature that we use all over the place. As a minimum this would have to... [17:09:14] I'm pretty sure I use the phab api outside my browser, but still from my home network [17:10:53] or just a default library u-a block [17:12:49] yeah... it definitely feels like one of the generic-y catch alls [17:20:02] oh right. lemme try again and I'll find the error [17:23:53] oh wow it worked [17:23:55] sorry all [17:23:58] https://usercontent.irccloud-cdn.com/file/kPZ5RSXe/image.png [17:24:41] I tested this a bunch of times when writing the integration, and it kept failing [17:24:48] https://phabricator.wikimedia.org/T420493 [17:24:53] https://phabricator.wikimedia.org/T420494 [17:25:07] but now it is, so I must have been doing something wrong [17:25:20] I guess I need to remove that notice... [17:25:27] sorry! [17:34:27] heh [17:40:40] 10Scap: scap backport fails too fast when trying to re-merge a patch that had failed CI - https://phabricator.wikimedia.org/T420501 (10Catrope) 03NEW [17:53:15] https://en.wikipedia.beta.wmcloud.org/wiki/Main_Page [17:53:22] Did you know ... [17:53:22] that the Beta Cluster now runs PHP 8.1? [17:53:24] nope [17:59:02] Reedy: I'm demoing this tool to part of the foundation soon. I'm pretty sure for that demo, it will decide not to work again :) [17:59:48] quick, record a video ;) [18:00:31] :) [18:04:40] 10Scap: scap backport fails too fast when trying to re-merge a patch that had failed CI - https://phabricator.wikimedia.org/T420501#11724756 (10dancy) p:05Triage→03Low [18:06:53] 10Scap: scap backport fails too fast when trying to re-merge a patch that had failed CI - https://phabricator.wikimedia.org/T420501#11724781 (10dancy) Why didn't `_approve_changes` recognize 1253604 as merged at 23:22:01(i.e., why was status != "MERGED")? The merge happened at 23:20:57 (earlier) [18:07:47] 18:07:32 18:07:32 /usr/bin/sudo -u root -- /usr/local/sbin/restart-php-fpm-all (ran as mwdeploy@deployment-parsoid14.deployment-prep.eqiad1.wikimedia.cloud) returned [255]: ssh: Could not resolve hostname deployment-parsoid14.deployment-prep.eqiad1.wikimedia.cloud: Name or service not known [18:14:00] 16:40 bd808: Deleted deployment-parsoid14.deployment-prep.eqiad1.wikimedia.cloud (T385849) [18:14:01] T385849: Determine usefulness of questionable hosts found in 2025-02-06 audit - https://phabricator.wikimedia.org/T385849 [18:14:31] 06Release-Engineering-Team (Seen): Replace Go templates with Rego policy DSL in the JWT authorizer - https://phabricator.wikimedia.org/T346331#11724816 (10dduvall) p:05Low→03Medium [18:15:26] Something in the scap/puppet config need updating to remove references to that host? [18:15:28] 06Release-Engineering-Team (Doing 😎): Replace Go templates with Rego policy DSL in the JWT authorizer - https://phabricator.wikimedia.org/T346331#11724824 (10dduvall) 05Open→03In progress a:03dduvall [18:15:53] yea, seems like that [18:16:28] but the hostname does not appear in puppet repo [18:16:30] `/etc/dsh/group/parsoid_php` is the file that is mentioning deployment-parsoid14 [18:16:44] (on deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud) [18:16:50] ah, and dsh groups are actually conftool now, at least in prod [18:16:56] 10Beta-Cluster-Infrastructure: beta-scap-sync-world tries to communicate with deployment-parsoid14.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420509 (10Reedy) 03NEW [18:17:37] the equivalent in prod would be https://config-master.wikimedia.org/pybal/eqiad/parsoid-php [18:17:45] and having to remove it from conftool data [18:18:54] 10Beta-Cluster-Infrastructure: beta-scap-sync-world tries to communicate with deployment-parsoid14.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420509#11724855 (10Reedy) p:05Triage→03Low [18:20:37] 10Beta-Cluster-Infrastructure: beta-scap-sync-world tries to communicate with deployment-parsoid14.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420509#11724864 (10Reedy) [18:45:36] 10Beta-Cluster-Infrastructure: beta-scap-sync-world tries to communicate with deployment-parsoid14.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420509#11724982 (10dancy) a:03dancy `deployment-parsoid14` was being referenced from /etc/dsh/group/parsoid_php`. On `deployment-deploy... [18:47:23] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11724988 (10Novem_Linguae) >>! In T420132#11722322, @Aklapper wrote: > My take is that implementing such a restriction would de-facto exclude new contributors, as "editin... [18:47:32] Project beta-scap-sync-world build #249763: 04FAILURE in 2 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/249763/ [18:48:22] ^ https://phabricator.wikimedia.org/T420509 [18:50:08] Project beta-scap-sync-world build #249764: 04STILL FAILING in 2 min 9 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/249764/ [18:50:32] Was that last run ahead of your puppet fixes? [18:50:37] Or did they not work? :-( [18:51:34] 10Beta-Cluster-Infrastructure: beta-scap-sync-world tries to communicate with deployment-parsoid14.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420509#11725002 (10Jdforrester-WMF) New error: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/249764/console ` 00:02:08.8... [18:53:13] The puppet cleanup was incomplete. I see one more bit that needs fixing [18:53:19] Aha. [18:57:03] https://gerrit.wikimedia.org/r/c/operations/puppet/+/1255012 [18:57:28] Makes sense. [18:57:52] In the meantime I'll touch a blank file to avoid errors from the CI job [18:58:09] Project beta-scap-sync-world build #249765: 04STILL FAILING in 2 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/249765/ [19:00:32] Yippee, build fixed! [19:00:32] Project beta-scap-sync-world build #249766: 09FIXED in 2 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/249766/ [19:03:11] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: beta-scap-sync-world tries to communicate with deployment-parsoid14.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420509#11725050 (10dancy) As a workaround until https://gerrit.wikimedia.org/r/c/operations/puppet/+/1255012 is merg... [19:11:18] Are these 504s I'm getting related? [19:15:40] James_F: dancy ^^ [19:15:59] Where are you getting 504s from? [19:16:00] What 504s? [19:16:08] trying to login to beta [19:16:12] Error: 504, Connection Timed Out at 2026-03-18 19:15:35 GMT [19:16:18] via deployment-cache-text08.deployment-prep.eqiad1.wikimedia.cloud, ATS/9.2.11 [19:16:52] Hmm. https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page is not working for me [19:17:05] good that it's not just me then :) [19:18:08] It just loaded for me on refresh. [19:18:45] Working for me now too. I wonder if we had a scrape attack during that time [19:18:50] Could be. [19:20:57] It is faster again for me too [19:22:26] Project beta-scap-sync-world build #249769: 04FAILURE in 2 min 25 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/249769/ [19:24:12] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11725164 (10SomeRandomDeveloper) >>! In T420132#11724988, @Novem_Linguae wrote: >>>! In T420132#11722322, @Aklapper wrote: >> My take is that implementing such a restrict... [19:25:38] Hmm. I still see parsoid_php listed in `mw_web_clusters` in the `[eqiad1.wikimedia.cloud]` section of `/etc/scap.cfg` on `deployment-deploy04.deployment-prep.eqiad1.wikimedia.cloud` [19:26:08] Even though https://gerrit.wikimedia.org/r/c/operations/puppet/+/1255012 is merged and I have run `sudo run-puppet-agent` on that host [19:26:37] I'm confused by that so I'm taking a mental break [19:28:15] 10Phabricator: Make it so that only Trusted-Contributors can edit things on other people's tickets - https://phabricator.wikimedia.org/T420132#11725169 (10Pppery) Indeed, editing a task does not include commenting on it. [19:39:32] Yippee, build fixed! [19:39:32] Project beta-scap-sync-world build #249770: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/249770/ [19:55:45] 10Phabricator: Hovering over long truncated strings - https://phabricator.wikimedia.org/T420522 (10Reedy) 03NEW [19:56:31] FIRING: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [19:56:41] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T420523 (10phaultfinder) 03NEW [20:01:31] RESOLVED: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:01:57] 10Phabricator: Hovering over long truncated strings may result in some vertical movement - https://phabricator.wikimedia.org/T420522#11725277 (10Reedy) p:05Triage→03Low [20:04:06] 10Phabricator: Hovering over long truncated strings may result in some vertical movement - https://phabricator.wikimedia.org/T420522#11725281 (10Reedy) [20:08:33] 10Phabricator: Hovering over long truncated strings may result in some vertical movement - https://phabricator.wikimedia.org/T420522#11725289 (10matmarex) a:03matmarex [20:08:36] 10Phabricator, 07CSS, 07Mobile: Simplified patch status comments have inconsistent margin on mobile - https://phabricator.wikimedia.org/T416964#11725290 (10matmarex) a:03matmarex [20:08:40] 10Phabricator, 07CSS: Gerrit patches are easy to miss in Phabricator history stream - https://phabricator.wikimedia.org/T416510#11725293 (10matmarex) a:03matmarex [20:20:31] FIRING: ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip6) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:20:43] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T420523#11725328 (10phaultfinder) [20:23:33] Project beta-code-update-eqiad build #592221: 04FAILURE in 32 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/592221/ [20:23:56] Project mediawiki-core-doxygen build #18693: 04FAILURE in 5 min 11 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18693/ [20:24:19] Gerrit seems unhappy now.. Getting quite a few 502s [20:25:31] FIRING: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:25:39] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T420523#11725339 (10phaultfinder) [20:32:18] 10GitLab: GitLab Private Repository Request for: CSP Research - https://phabricator.wikimedia.org/T420485#11725377 (10Peachey88) [20:35:31] RESOLVED: [2x] ProbeDown: Service gerrit2003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit2003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [20:35:39] Yippee, build fixed! [20:35:39] Project beta-code-update-eqiad build #592222: 09FIXED in 2 min 38 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/592222/ [21:31:26] (03PS7) 10JHathaway: puppet: use bind mounts for local runs [integration/config] - 10https://gerrit.wikimedia.org/r/1135116 [21:31:51] 06Release-Engineering-Team, 10MediaWiki-extensions-WikimediaMaintenance, 10Wiki-Setup (Create): addWiki.php fails with CannotReplaceActiveServiceException for DBLoadBalancerFactory - https://phabricator.wikimedia.org/T420531#11725608 (10Jdforrester-WMF) [21:32:13] Yippee, build fixed! [21:32:14] Project mediawiki-core-doxygen build #18694: 09FIXED in 14 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18694/ [21:34:53] 06Release-Engineering-Team, 10MediaWiki-extensions-WikimediaMaintenance, 10Wiki-Setup (Create): addWiki.php fails with CannotReplaceActiveServiceException for DBLoadBalancerFactory - https://phabricator.wikimedia.org/T420531#11725617 (10Jdforrester-WMF) @Zabe: You created a new wiki two weeks ago with {T4159... [21:37:46] 06Release-Engineering-Team, 10MediaWiki-extensions-WikimediaMaintenance, 10Wiki-Setup (Create): addWiki.php fails with CannotReplaceActiveServiceException for DBLoadBalancerFactory - https://phabricator.wikimedia.org/T420531#11725634 (10Jdforrester-WMF) Similar to {T341102} I imagine. [21:40:05] (03PS8) 10JHathaway: puppet: use bind mounts for local runs [integration/config] - 10https://gerrit.wikimedia.org/r/1135116 [21:41:25] 06Release-Engineering-Team, 10MediaWiki-extensions-WikimediaMaintenance, 10Wiki-Setup (Create): addWiki.php fails with CannotReplaceActiveServiceException for DBLoadBalancerFactory - https://phabricator.wikimedia.org/T420531#11725651 (10Zabe) >>! In T420531#11725615, @Jdforrester-WMF wrote: > @Zabe: You crea... [21:42:48] (03CR) 10JHathaway: "picking this up again after it got lost in my backlog, how do I coordinate merging this with I3c765f0c1ae3d987a7cec598bed7f1e684bc506b" [integration/config] - 10https://gerrit.wikimedia.org/r/1135116 (owner: 10JHathaway) [21:48:25] 06Release-Engineering-Team, 10MediaWiki-extensions-WikimediaMaintenance, 10Wiki-Setup (Create): addWiki.php fails with CannotReplaceActiveServiceException for DBLoadBalancerFactory - https://phabricator.wikimedia.org/T420531#11725674 (10Jdforrester-WMF) >>! In T420531#11725651, @Zabe wrote: >>>! In T420531#1... [21:51:37] 10Gerrit, 06Release-Engineering-Team, 14Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 07ci-test-error (WMF-deployed Build Failure), 13Patch-For-Review: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when interacting with rep... - https://phabricator.wikimedia.org/T246763#11725686 [22:07:40] A_smart_kitten thanks :) [22:12:05] Neriah: oh, re my comment in T246763? (or something else?) [22:12:06] T246763: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when interacting with repos - https://phabricator.wikimedia.org/T246763 [22:13:07] ya [22:13:36] though now looking at IRC for the first time so far today, I'm wondering if those CI 502s for your patch (that I commented on that task about) were actually related to that task, or if they were just Gerrit having some problems at that sort of time :p [22:13:59] but yw either way :] [22:18:37] 10Phabricator-Bot-Requests: Create mcp-phabricator bot account - https://phabricator.wikimedia.org/T420446#11725808 (10kostajh) Thanks for your analysis. Another approach might be to look at not requiring an API token at all for searching. This doesn't seem like that big of a stretch, given that we allow searchi... [22:21:14] A_smart_kitten for example https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1250562 [22:24:10] Neriah: yeah, given that that was also around the same time as the failures at https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CheckUser/+/1254998, and given r.eedy's message above ~2 hours ago, I'm wondering if Gerrit was just having some general problems at that sort of time for whatever reason, that also impacted CI [22:24:25] T420523 seems to be the automatically-opened task for that [22:24:26] T420523: ProbeDown - https://phabricator.wikimedia.org/T420523 [22:27:00] Oops, it seems the first message I wrote didn't get sent... [22:27:12] (before the link) [22:27:21] oh no worries, feel free to resend [22:31:02] It was deleted 🤦‍♂️ [22:31:05] I don't get along with this system... [22:31:13] ah, sorry :/ [22:31:18] generally speaking, I'm not sure it was a general problem. [22:31:34] because I'm pretty sure there were tests that did go through at the time. [22:59:19] (03open) 10dduvall: image: Derive exported variable names from `CI_JOB_NAME_SLUG` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/165 [22:59:28] (03update) 10dduvall: image: Derive exported variable names from `CI_JOB_NAME_SLUG` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/165 [23:01:50] (03update) 10dduvall: image: Derive exported variable names from `CI_JOB_NAME_SLUG` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/165 [23:02:16] (03update) 10dduvall: image: Derive exported variable names from `CI_JOB_NAME_SLUG` [repos/releng/kokkuri] - 10https://gitlab.wikimedia.org/repos/releng/kokkuri/-/merge_requests/165