[00:01:05] *if [00:50:59] 10Phabricator, 10MediaWiki-extensions-CentralAuth, 06Stewards-and-global-tools: CentralAuth locks should disable linked Phabricator account - https://phabricator.wikimedia.org/T338384#10360235 (10Pppery) [01:57:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10360310 (10cscott) Ther... [05:44:00] bd808: nice, thanks! [08:04:05] (03open) 10jelto: gitlab-runner: bump image version to alpine-v17.4.2 [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/415 (https://phabricator.wikimedia.org/T380906) [08:32:38] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [08:35:35] FIRING: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [08:42:38] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [08:45:35] RESOLVED: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [08:53:36] 06Release-Engineering-Team, 10WMF-General-or-Unknown: wgBetaFeaturesAllowList lists to ask Greg G - https://phabricator.wikimedia.org/T380929#10360581 (10Aklapper) It should be a place (which could describe a team) and not any individual names. Maybe https://www.mediawiki.org/wiki/Wikimedia_Release_Engineering... [08:55:28] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 13Patch-For-Review: Reports: Order Over-due Tasks by oldest date instead of task ID - https://phabricator.wikimedia.org/T380907#10360582 (10Aklapper) [09:34:47] 10Deployments, 06Release-Engineering-Team, 06serviceops, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958 (10hashar) 03NEW p:05Triage→03Medium [09:39:35] 10Deployments, 06serviceops-radar, 06SRE: Confusing failed httpbb check for totoro.wikimedia.org during scap deployment - https://phabricator.wikimedia.org/T364880#10360638 (10hashar) 05Open→03Resolved a:03RLazarus I am marking this one resolved since the confusing https://totoro.wikimedia.org. URL... [09:57:21] 10Deployments, 06Release-Engineering-Team, 06serviceops, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10360675 (10hashar) Some logs from php-fpm being restarted | Time | type | level | message | program |------|------|-------|--------... [10:16:30] 10Deployments, 06Release-Engineering-Team, 06serviceops, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10360717 (10akosiaris) This is probably what you are looking for ` [2024-11-27T09:25:11.887Z] "GET /wiki/Wikidata:Main_Page HTTP/1.... [10:34:18] 10Deployments, 06Release-Engineering-Team, 06serviceops, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10360770 (10Lucas_Werkmeister_WMDE) FWIW, I had two similar errors on Monday, no idea if the logs are still around. [IRC](https://wm... [10:46:05] 10Deployments, 06Release-Engineering-Team, 06serviceops, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10360825 (10hashar) > How many times has it been reproduced up to now? I had one yesterday when promoting group 0 which came from b... [10:48:51] 10Deployments, 06Release-Engineering-Team, 06serviceops, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10360844 (10hashar) [10:48:53] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.5 deployment blockers - https://phabricator.wikimedia.org/T375664#10360845 (10hashar) [10:49:51] 10GitLab: GitLab Private Repository Request for: WikiSwarm - https://phabricator.wikimedia.org/T380965 (10Joe) 03NEW p:05Triage→03High [10:50:02] 10Deployments, 06Release-Engineering-Team, 06serviceops, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10360861 (10hashar) I have removed this from the list of train subtasks, it only got there cause I have used {nav Create Subtask} to... [10:51:13] 10GitLab: GitLab Private Repository Request for: WikiSwarm - https://phabricator.wikimedia.org/T380965#10360862 (10Joe) [10:55:53] hashar: I've been looking at speeding up the selenium test runs further. What are you feelings about sharding-based approaches ( https://gerrit.wikimedia.org/r/c/integration/config/+/1097353 , https://webdriver.io/docs/sharding/ ) - is that something we can experiment with? [10:57:19] they already run in parallel? [10:59:26] ah [10:59:55] codders: we had them running in parallel with https://phabricator.wikimedia.org/T226869 [11:00:13] https://gerrit.wikimedia.org/r/q/bug:T226869 [11:00:14] T226869: Run browser tests in parallel - https://phabricator.wikimedia.org/T226869 [11:00:18] though some repositories are still pending [11:00:31] they run in parallel as in they use wdio maxInstances [11:00:39] cause that highlights a bunch of race conditions / concurrency trubles [11:00:54] so some did not get moved [11:01:05] but they still take 18 mins in total. I'm looking at strategies to reduce that: https://phabricator.wikimedia.org/T374003 [11:01:25] sharding avoids race conditions entirely - each instance runs the tests linearly and gets its own wiki [11:01:27] I think some repositories have a super large package-lock.json [11:02:12] and surely there are plenty of tests we should not run in the first place [11:02:19] similar to the `Standalone` tests in PHPUnit [11:02:37] anyway, I'd like to avoid introducing more Jenkins jobs [11:03:05] and instead have the parallelism carried either in quibble / mediawiki or the test suite [11:03:16] okay. why's that? [11:03:37] I feel like sooner or later we're going to hit the limits of what a single agent can parallelise [11:03:39] cause I want to get rid of Jenkins [11:03:56] and the less complexity it has, the easier it will be to migrate out of it [11:04:18] but overall, I have doubt about whether those browser tests have anyusefulness [11:04:27] they can probably be filtered out [11:05:05] Do we have a plan for CI after Jenkins? Is there somewhere it makes sense to run CI jobs besides Jenkins? [11:05:50] aka do we need to run the 80 or so Cite tests for patches affecting other repositories or shouldn't that be a in a dedicated job that only run for Cite itself [11:06:11] okay. but that would also mean creating a new jenkins job, right? [11:06:14] yeah the plan is to upgrade Zuul [11:06:54] which no more relies on Jenkins but on Ansible to define tasks (which themselves can really be just about spinning up a container that has all the logic, to be determinated) [11:07:14] ah. interesting [11:07:20] if we get a job dedicated to Cite, for sure that is another job, but it would only trigger for patch to Cite [11:07:27] so that is less jobs running concurrently overall [11:07:28] :) [11:07:45] is your concern the number of jobs defined? or the number of jobs running concurrently? [11:08:02] whereas if we get multiple wmf-quibble-selenium jobs, that is more jobs running in parallel and that causes issues on the infra [11:08:04] or at least could [11:08:11] so yeah number of jobs running concurrently is my concern [11:08:13] well [11:08:20] concurrency IS my concern overall :-b [11:08:40] same as for running PHPUnit tests in parallel, that got me worried, but apparently it went well [11:08:45] you don't like more jenkins jobs either, but you'd take more jenkins jobs if it avoids concurrency [11:08:45] so I am probably a bit too paranoid [11:09:43] also if we have more jobs triggered for a change, that reduces the number of changes tested in parallel given the number of jobs that can run in parallel is fixed (that is the number of executors available on the Jenkins agent) [11:09:45] something like that [11:10:12] there is surely a bunch of math equations that can assist in doing the capacity planning [11:10:29] that's true. my priority is reducing the cycle time for patches for developers. But it's also not clear to me that we're anywhere near capacity most of the time [11:10:30] (I think in Math the field is named "queue theory" ) [11:10:40] yeah - I skipped that course. *sooo boring* [11:10:59] I used to work in telecom [11:11:06] that is a must know :b [11:11:13] (I think it was more the lecturer than the content, to be fair) [11:11:31] but I agree it is not a very exciting course :) [11:12:05] oh we hjave Cypress as well [11:12:05] eek [11:12:12] anyway [11:12:26] I'd look at moving some repos/tests out of the wmf-quibble-selenium job [11:13:02] and have that one solely for testing interactions accross extensions / core / Vector|MinervaNeue [11:13:22] if eg Cite and Wikibase takes a while, we can offload them [11:13:30] s/we/one/ [11:13:36] anyway, I gotta prepare lunch [11:13:40] that's a bold plan. I'll stick it on my list of options - thanks! [11:14:05] 10Deployments, 06Release-Engineering-Team, 06serviceops, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10360917 (10Tgr) Happened twice to me on Monday: https://pastebin.com/yV7PLW5P https://pastebin.com/H86xifkd [11:14:17] if we get some timing about how long it take for `npm ci` and running the actual tests for each repositories, that might help [11:14:38] kosta had a patch to run the npm install in parallel [11:14:48] ahead of running `npm run selenium-test` (or whatever is the entry point) [11:15:01] yeah. we talked about that too [11:15:13] I can try gather some data [11:15:21] ah --parallel-npm-install [11:15:53] which got enabled and then removed https://gerrit.wikimedia.org/r/c/integration/config/+/771820 [11:16:02] not sure why, the commit message or the linked tasks might have the explanation [11:16:33] I'll take a look - thanks for the tip! [11:18:33] what I would love is a way to generate a timeline of quibble tasks/commands [11:18:38] similar to `systemd-analyze` https://lafibre.info/images/stats/202303_systemd-analyze_plot_lafibre.svg [11:19:15] or OpenTelemetry trace view https://opentelemetry.io/docs/demo/screenshots/jaeger-trace-view.png [11:21:19] possibly generated as a web browser har archive [11:28:12] that would be very cool [11:29:04] my concern about moving tests out of the `wmf-quibble-selenium` job is that it exacerbates the issue that we already have, that merges to `mediawiki/core` breaks extensions in some unexpected way, and then we wake up in the european morning and have to debug some CI failure before we can get on with our jobs [11:29:27] It makes more sense to me to allocate CPU resources than to have developers spend more time firefighting CI failures [11:39:49] Is there already a task / plan for the Zuul upgrade work? (I don't see anything tagged https://phabricator.wikimedia.org/tag/zuul/ ) [12:04:59] (03PS1) 10Lucas Werkmeister (WMDE): Zuul: [purtle] Test in PHP 8.3 too [integration/config] - 10https://gerrit.wikimedia.org/r/1098494 (https://phabricator.wikimedia.org/T368271) [12:21:28] (03CR) 10Hashar: [C:03+2] Zuul: [purtle] Test in PHP 8.3 too [integration/config] - 10https://gerrit.wikimedia.org/r/1098494 (https://phabricator.wikimedia.org/T368271) (owner: 10Lucas Werkmeister (WMDE)) [12:23:04] (03Merged) 10jenkins-bot: Zuul: [purtle] Test in PHP 8.3 too [integration/config] - 10https://gerrit.wikimedia.org/r/1098494 (https://phabricator.wikimedia.org/T368271) (owner: 10Lucas Werkmeister (WMDE)) [13:41:03] 10Continuous-Integration-Infrastructure, 10Quibble: Parallel npm install times out in Quibble - https://phabricator.wikimedia.org/T303270#10361488 (10Lucas_Werkmeister_WMDE) >>! In T303270#7760110, @hashar wrote: > I lost myself in cpython threading module and found https://github.com/python/cpython/commit/872... [13:53:48] 10Deployments, 06Release-Engineering-Team, 06serviceops, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10361543 (10akosiaris) This [grafana explore panel](https://w.wiki/CDKe) aligns pretty well with the errors in the last 3 days. {F... [14:13:35] (03PS1) 10Máté Szabó: zuul: Test ReportIncident with EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/1098516 (https://phabricator.wikimedia.org/T380599) [14:26:51] (03CR) 10Kosta Harlan: [C:03+1] zuul: Test ReportIncident with EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/1098516 (https://phabricator.wikimedia.org/T380599) (owner: 10Máté Szabó) [14:27:34] (03CR) 10Kosta Harlan: [C:03+1] zuul: Test ReportIncident with EventLogging (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1098516 (https://phabricator.wikimedia.org/T380599) (owner: 10Máté Szabó) [14:29:33] (03PS1) 10Kosta Harlan: Revert "jjb: Remove parallel-npm-install flag" [integration/config] - 10https://gerrit.wikimedia.org/r/1098522 [14:29:41] (03CR) 10CI reject: [V:04-1] Revert "jjb: Remove parallel-npm-install flag" [integration/config] - 10https://gerrit.wikimedia.org/r/1098522 (owner: 10Kosta Harlan) [14:29:42] (03PS2) 10Kosta Harlan: Revert "jjb: Remove parallel-npm-install flag" [integration/config] - 10https://gerrit.wikimedia.org/r/1098522 (https://phabricator.wikimedia.org/T303270) [14:29:50] (03CR) 10CI reject: [V:04-1] Revert "jjb: Remove parallel-npm-install flag" [integration/config] - 10https://gerrit.wikimedia.org/r/1098522 (https://phabricator.wikimedia.org/T303270) (owner: 10Kosta Harlan) [14:31:54] (03PS3) 10Kosta Harlan: Revert "jjb: Remove parallel-npm-install flag" [integration/config] - 10https://gerrit.wikimedia.org/r/1098522 (https://phabricator.wikimedia.org/T303270) [14:33:09] 10Deployments, 06Release-Engineering-Team, 06serviceops, 07Wikimedia-production-error: httpb fails upon deployment of 1.44.0-wmf.5 - https://phabricator.wikimedia.org/T380958#10361743 (10Aklapper) >>! In T380958#10360717, @akosiaris wrote: > How many times has it been reproduced up to now? I also experienc... [14:45:56] 10Continuous-Integration-Infrastructure, 10Shellbox, 07Test-Coverage: Shellbox coverage not published since June 2022 - https://phabricator.wikimedia.org/T378423#10361801 (10hashar) a:03hashar The last coverage was generated on Jun 27 9:53:11 UTC 2022. We eventually switched to php 7.2 /upgraded composer/d... [14:51:19] (03CR) 10Máté Szabó: zuul: Test ReportIncident with EventLogging (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/1098516 (https://phabricator.wikimedia.org/T380599) (owner: 10Máté Szabó) [14:51:20] (03PS2) 10Máté Szabó: zuul: Test ReportIncident with EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/1098516 (https://phabricator.wikimedia.org/T380599) [14:55:25] (03CR) 10Hashar: [C:03+2] zuul: Test ReportIncident with EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/1098516 (https://phabricator.wikimedia.org/T380599) (owner: 10Máté Szabó) [14:57:04] (03Merged) 10jenkins-bot: zuul: Test ReportIncident with EventLogging [integration/config] - 10https://gerrit.wikimedia.org/r/1098516 (https://phabricator.wikimedia.org/T380599) (owner: 10Máté Szabó) [15:01:19] (03CR) 10Kosta Harlan: "deploy with caution. It might need a revert. :)" [integration/config] - 10https://gerrit.wikimedia.org/r/1098522 (https://phabricator.wikimedia.org/T303270) (owner: 10Kosta Harlan) [15:06:20] (03CR) 10Hashar: [C:03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/1098516 (https://phabricator.wikimedia.org/T380599) (owner: 10Máté Szabó) [15:12:48] > Could not resolve host: gerrit.wikimedia.org [15:12:50] it’s back :( [15:12:55] (https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-selenium/56225/console, https://integration.wikimedia.org/ci/job/mediawiki-quibble-apitests-vendor-php74/27838/console) [15:15:54] filed T380991 for it [15:15:55] T380991: Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991 [15:16:58] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991 (10Lucas_Werkmeister_WM... [15:19:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10361978 (10Lucas_Werkmei... [15:59:01] 10Continuous-Integration-Infrastructure, 10Quibble: Add support for multi-DB testing to MediaWiki - https://phabricator.wikimedia.org/T379925#10362231 (10hashar) For T342763, @Urbanecm proposed a patch to Quibble to have it spin up a second database ( https://gerrit.wikimedia.org/r/c/integration/quibble/+/9499... [15:59:10] (03CR) 10Hashar: "That is followed up in T379925: Add support for multi-DB testing to MediaWiki" [integration/quibble] - 10https://gerrit.wikimedia.org/r/949986 (https://phabricator.wikimedia.org/T342763) (owner: 10Urbanecm) [16:00:00] Lucas_WMDE: that is the WMCS infra rather than just Gerrit [16:00:19] my guess is there is ongoing work on it [16:11:14] 06Release-Engineering-Team, 10WMF-General-or-Unknown: wgBetaFeaturesAllowList lists to ask Greg G - https://phabricator.wikimedia.org/T380929#10362278 (10Pppery) Also all but one entry has a date more than 6 months ago so the other part of the change should be enforced or removed. [16:12:08] 06Release-Engineering-Team, 10WMF-General-or-Unknown: wgBetaFeaturesAllowList permission/sign-off process - https://phabricator.wikimedia.org/T380929#10362283 (10greg) [16:12:31] 06Release-Engineering-Team, 10WMF-General-or-Unknown: wgBetaFeaturesAllowList permission/sign-off process - https://phabricator.wikimedia.org/T380929#10362285 (10greg) (renaming so I'm not always pinged on IRC :) ) [16:16:50] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991#10362307 (10bd808) My... [16:24:07] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991#10362357 (10Lucas_Werkm... [16:27:12] 06Release-Engineering-Team, 10WMF-General-or-Unknown: wgBetaFeaturesAllowList permission/sign-off process - https://phabricator.wikimedia.org/T380929#10362370 (10hashar) The `$wgBetaFeaturesAllowList` flag was introduced by @Jdforrester-WMF back in 2014 with https://gerrit.wikimedia.org/r/c/operations/mediawik... [16:29:14] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991#10362375 (10fnegri) +1... [16:31:18] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991#10362378 (10bd808) >>!... [16:32:18] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991#10362402 (10bd808)... [16:32:42] 06Release-Engineering-Team, 10WMF-General-or-Unknown: Improve wgBetaFeaturesAllowList permission/sign-off process - https://phabricator.wikimedia.org/T380929#10362407 (10Pppery) [16:33:12] 06Release-Engineering-Team, 10WMF-General-or-Unknown: Improve wgBetaFeaturesAllowList permission/sign-off process - https://phabricator.wikimedia.org/T380929#10362409 (10Jdforrester-WMF) We already have T352825; should we merge this into that? [16:33:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10362413 (10bd808) >>! In T3... [16:34:04] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10362400 (10bd808) [16:34:23] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org - https://phabricator.wikimedia.org/T374830#10362405 (10bd808) 05Resolv... [16:34:30] 06Project-Admins, 05acl*Project-Admins: Create Group for Wikimedians of Kerala User Group - https://phabricator.wikimedia.org/T381004 (10Ranjithsiji) 03NEW [16:34:51] 06Project-Admins: Create Group for Wikimedians of Kerala User Group - https://phabricator.wikimedia.org/T381004#10362428 (10taavi) [16:35:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10362432 [16:35:30] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10362433 [16:45:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10362518 [16:47:32] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10362520 [17:10:57] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991#10362670 (10Andrew... [17:12:26] 10Beta-Cluster-Infrastructure: Re-create deployment-cumin - https://phabricator.wikimedia.org/T380678#10362691 (10taavi) 05Open→03Resolved a:03taavi [17:29:51] 06Project-Admins: Create Group for Wikimedians of Kerala User Group - https://phabricator.wikimedia.org/T381004#10362774 (10Aklapper) Hi, thanks for your interest! What does that mean for https://gitlab.com/groups/wikimedians-kerala/-/issues , https://github.com/Wikimedians-of-Kerala/12th-wikidata-birthday-pop/i... [17:35:19] (03approved) 10mszabo: releases.json: Upgrade phpunit/phpunit to 9.6.21 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/36 (owner: 10reedy) [17:35:32] (03update) 10mszabo: releases.json: Upgrade phpunit/phpunit to 9.6.21 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/36 (owner: 10reedy) [17:35:55] (03merge) 10mszabo: releases.json: Upgrade phpunit/phpunit to 9.6.21 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/36 (owner: 10reedy) [17:42:38] FIRING: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [17:44:34] (03open) 10mszabo: releases: Add @vue/test-utils and Jest [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/44 [17:44:35] FIRING: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [17:46:11] (03update) 10mszabo: releases: Add @vue/test-utils and Jest [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/44 [17:46:22] (03update) 10mszabo: releases: Add @vue/test-utils and Jest [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/44 [17:48:33] (03PS1) 10Hslater: Add NotifyMe as a dependency of BlueSpiceBookshelf & BlueSpicePageAssignments [integration/config] - 10https://gerrit.wikimedia.org/r/1098576 [17:49:35] FIRING: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [17:49:50] FIRING: [2x] DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [17:51:11] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991#10362840 (10Andrew... [17:52:33] RESOLVED: DatasourceError: Queue (Jenkins jobs + Zuul functions) - https://grafana.wikimedia.org/alerting/grafana/b9a8470a-ebab-46f7-9be2-22b5e74a528b/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [17:54:35] RESOLVED: DatasourceNoData: - https://alerts.wikimedia.org/?q=alertname%3DDatasourceNoData [18:45:48] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991#10363016 (10Andrew... [18:46:04] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991#10363013 (10Andrew... [18:51:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10363043 [19:06:53] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991#10363163 (10Andrew) 0... [19:08:37] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991#10363177 (10Andrew) ht... [19:09:23] 10Beta-Cluster-Infrastructure, 10CampaignEvents, 06Campaigns-Product-Team, 10MediaWiki-extensions-CentralAuth, and 2 others: Users deleted through Special:CentralAuth can still log in on betacluster - https://phabricator.wikimedia.org/T381028#10363175 (10vaughnwalters) [19:10:50] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991#10363188 (10Andrew) A... [19:12:44] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991#10363190 (10Andrew) p:... [19:13:55] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991#10363191 (10Andrew) Ju... [19:18:15] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991#10363203 (10Andrew) al... [19:22:00] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991#10363225 (10bd808) > A... [19:34:36] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10363330 [19:34:42] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs failing with: Could not resolve host: gerrit.wikimedia.org (2024-11-27) - https://phabricator.wikimedia.org/T380991#10363336 (10Andrew) >>... [20:00:45] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10363600 [20:03:48] 06Release-Engineering-Team, 10Scap, 06collaboration-services: scap / puppet dependency issue on reimaged phabricator hosts - https://phabricator.wikimedia.org/T378769#10363614 (10Dzahn) Deployed Jaime's fix. No issues on any of the 180 scap::target hosts with a notable exception. It broke puppet on the 2 p... [20:13:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 06cloud-services-team, 10Cloud-VPS, 10ci-test-error (WMF-deployed Build Failure): Various CI jobs running in the integration Cloud VPS project failing due to transient DNS lookup... - https://phabricator.wikimedia.org/T374830#10363633 [20:39:10] 10Beta-Cluster-Infrastructure, 10CampaignEvents, 06Campaigns-Product-Team, 10MediaWiki-extensions-CentralAuth, and 2 others: Users deleted through Special:CentralAuth can still log in on betacluster - https://phabricator.wikimedia.org/T381028#10363708 (10Reedy) Deleting a global (user) account isn't the sa... [20:56:11] 10Beta-Cluster-Infrastructure, 10CampaignEvents, 06Campaigns-Product-Team, 10MediaWiki-extensions-CentralAuth, and 2 others: Users deleted through Special:CentralAuth can still log in on betacluster - https://phabricator.wikimedia.org/T381028#10363778 (10Urbanecm) >>! In T381028#10363708, @Reedy wrote: > D... [20:56:23] (03open) 10jforrester: releases: Bump jsdoc from 4.0.2 to 4.0.4 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/45 [20:56:35] (03merge) 10jforrester: releases: Bump jsdoc from 4.0.2 to 4.0.4 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/45 [20:59:01] 10Beta-Cluster-Infrastructure, 10CampaignEvents, 06Campaigns-Product-Team, 10MediaWiki-extensions-CentralAuth, and 2 others: Users deleted through Special:CentralAuth can still log in on betacluster - https://phabricator.wikimedia.org/T381028#10363804 (10Reedy) ` "CentralAuthStrict": { "description":... [21:18:25] 10Beta-Cluster-Infrastructure, 10CampaignEvents, 06Campaigns-Product-Team, 10MediaWiki-extensions-CentralAuth, and 2 others: Users deleted through Special:CentralAuth can still log in on betacluster - https://phabricator.wikimedia.org/T381028#10363845 (10Tgr) >>! In T381028#10363778, @Urbanecm wrote: > Tha... [21:32:45] 10Beta-Cluster-Infrastructure, 10CampaignEvents, 06Campaigns-Product-Team, 10MediaWiki-extensions-CentralAuth, and 2 others: Users deleted through Special:CentralAuth can still log in on betacluster - https://phabricator.wikimedia.org/T381028#10363885 (10vaughnwalters) Okay if this is expected behavior. th... [22:31:30] 10Beta-Cluster-Infrastructure, 10CampaignEvents, 06Campaigns-Product-Team, 10MediaWiki-extensions-CentralAuth, and 2 others: Users deleted through Special:CentralAuth can still log in on betacluster - https://phabricator.wikimedia.org/T381028#10363998 (10Tgr) >>! In T381028#10363845, @Tgr wrote: > It doesn... [22:50:44] (03open) 10volker-e: releases: Bump Codex to 1.17.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/46 [22:52:11] (03approved) 10jforrester: releases: Bump Codex to 1.17.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/46 (owner: 10volker-e) [22:52:15] (03update) 10volker-e: releases: Bump Codex to 1.17.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/46 [22:52:21] (03update) 10volker-e: releases: Bump Codex to 1.17.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/46 [22:52:24] (03merge) 10jforrester: releases: Bump Codex to 1.17.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/46 (owner: 10volker-e)