[01:22:47] some phab admin wants to delete https://phabricator.wikimedia.org/T303115#7755655 ? [05:06:08] ^ has been done [10:27:27] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed (Feb 2022) - https://phabricator.wikimedia.org/T302699 (10dom_walden) >>! In T302699#7740763, @dom_walden wrote: > ` > AH00288: scoreboard is full, not at MaxRequestWorker... [10:32:51] 10Beta-Cluster-Infrastructure: Setup prometheus-mysqld-exporter on beta cluster - https://phabricator.wikimedia.org/T301989 (10Majavah) >>! In T301989#7741352, @dom_walden wrote: >>>! In T301989#7739896, @Majavah wrote: >> The exporter is already running on the db servers. However it's not scraped on any Prometh... [10:45:23] 10Beta-Cluster-Infrastructure, 10Analytics, 10Beta-Cluster-reproducible: 502, connect failed for intake-analytics.wikimedia.beta.wmflabs.org (Mar 2022) - https://phabricator.wikimedia.org/T303160 (10AlexisJazz) [10:53:11] !log restarted apache on deployment-mediawiki11 # T302699 [10:53:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:53:14] T302699: Beta cluster down: Error: 502, Next Hop Connection Failed (Feb 2022) - https://phabricator.wikimedia.org/T302699 [11:03:59] 10Beta-Cluster-Infrastructure, 10MediaWiki-API, 10Traffic, 10Beta-Cluster-reproducible: api.php not working (503, Backend fetch failed) - https://phabricator.wikimedia.org/T303165 (10AlexisJazz) [11:07:55] 10Release-Engineering-Team (Next), 10Release, 10Train Deployments: 1.38.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T300201 (10AlexisJazz) API broken. T303165 If I got it wrong task please copy to the right deployment blockers thingy. [11:12:40] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed (Feb 2022) - https://phabricator.wikimedia.org/T302699 (10AlexisJazz) >>! In T302699#7756154, @dom_walden wrote: >>>! In T302699#7740763, @dom_walden wrote: >> ` >> AH0028... [11:14:02] 10Beta-Cluster-Infrastructure, 10MediaWiki-API, 10Traffic, 10Beta-Cluster-reproducible: api.php not working (503, Backend fetch failed) - https://phabricator.wikimedia.org/T303165 (10Majavah) [11:14:10] 10Beta-Cluster-Infrastructure, 10Analytics, 10Beta-Cluster-reproducible: 502, connect failed for intake-analytics.wikimedia.beta.wmflabs.org (Mar 2022) - https://phabricator.wikimedia.org/T303160 (10Majavah) [11:14:16] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-extensions-Nuke: Special:Nuke on betacommons returned Secure Connection Failed/502 Next Hop Connection Failed, completed anyway - https://phabricator.wikimedia.org/T302544 (10Majavah) [11:14:19] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed (Feb 2022) - https://phabricator.wikimedia.org/T302699 (10Majavah) [11:17:37] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed (Feb 2022) - https://phabricator.wikimedia.org/T302699 (10AlexisJazz) @Majavah are you sure T303165 is a dupe? That task is about api.php (and nothing else!) **consistentl... [11:19:55] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed (Feb 2022) - https://phabricator.wikimedia.org/T302699 (10Majavah) >>! In T302699#7756393, @AlexisJazz wrote: > @Majavah are you sure T303165 is a dupe? That task is about... [11:35:21] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed (Feb 2022) - https://phabricator.wikimedia.org/T302699 (10AlexisJazz) >>! In T302699#7756404, @Majavah wrote: >>>! In T302699#7756393, @AlexisJazz wrote: >> @Majavah are y... [11:41:52] 10Beta-Cluster-Infrastructure, 10SRE, 10Traffic, 10Beta-Cluster-reproducible: Beta cluster down: Error: 502, Next Hop Connection Failed (Feb 2022) - https://phabricator.wikimedia.org/T302699 (10Vgutierrez) in this case a 502 is emitted by ats-backend cause it isn't able to reach its backend server. The 503... [12:56:09] 10Release-Engineering-Team, 10Growth-Team, 10PageCuration, 10Patch-For-Review, and 2 others: quibble-vendor-mysql-php72-noselenium-docker fails for a noop PageTriage patch - https://phabricator.wikimedia.org/T303092 (10kostajh) I can't reproduce the failure locally, of course. >>! In T303092#7754611, @Dai... [13:02:40] 10Release-Engineering-Team, 10Growth-Team, 10PageCuration, 10Patch-For-Review, and 2 others: quibble-vendor-mysql-php72-noselenium-docker fails for a noop PageTriage patch - https://phabricator.wikimedia.org/T303092 (10kostajh) Tagging @aaron and @Krinkle in case they have ideas about the WANObjectCache fa... [13:20:21] 10Release-Engineering-Team (Next), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.24 deployment blockers - https://phabricator.wikimedia.org/T300200 (10kostajh) Seems like {T303175} was caused during the rollout of wmf.24. [13:26:11] 10Release-Engineering-Team, 10Growth-Team, 10PageCuration, 10Patch-For-Review, and 2 others: quibble-vendor-mysql-php72-noselenium-docker fails for a noop PageTriage patch - https://phabricator.wikimedia.org/T303092 (10Krinkle) [14:18:59] (03CR) 10Hashar: [C: 03+2] "Somehow I missed that one. Jobs updated" [integration/config] - 10https://gerrit.wikimedia.org/r/757393 (https://phabricator.wikimedia.org/T299491) (owner: 10Awight) [14:21:05] (03Merged) 10jenkins-bot: Switch QUnit tests to use the apache backend [integration/config] - 10https://gerrit.wikimedia.org/r/757393 (https://phabricator.wikimedia.org/T299491) (owner: 10Awight) [14:34:18] 10Continuous-Integration-Config, 10Quibble, 10MW-1.38-notes (1.38.0-wmf.21; 2022-02-07), 10Patch-For-Review: Switch all Quibble Selenium and api-testing jobs to use apache - https://phabricator.wikimedia.org/T285649 (10hashar) [14:44:08] 10Release-Engineering-Team, 10Growth-Team, 10PageCuration, 10Patch-For-Review, and 2 others: quibble-vendor-mysql-php72-noselenium-docker fails for a noop PageTriage patch - https://phabricator.wikimedia.org/T303092 (10Krinkle) I haven't found a bug or solved anything, but I have confirmed that the `getMul... [14:59:12] 10Continuous-Integration-Config: Create `check coverage` pipeline for library repos - https://phabricator.wikimedia.org/T302725 (10hashar) [14:59:14] 10Continuous-Integration-Config, 10Librarization, 10Test-Coverage: Run a phpunit coverage job pre-merge for libraries - https://phabricator.wikimedia.org/T147093 (10hashar) [15:34:17] o/ gitlab question: when trying to import a project from github I'm given the options to import into my userspace or inside the group people/wmf-team-search-platform (which I think would be a mistake) but I'd like to import to repos/search-platform [15:48:18] ah I need to be a direct owner of the repos/search-platform to have it in the dropdown menu, looks like it can't infer possible targets if they are only reachable via "me -> people/wmf-team-search-platform -> repos/target" [16:23:39] dcausse: yeah, i think that's correct. [16:24:08] see also the blog post i put up friday: https://phabricator.wikimedia.org/phame/post/view/273/gitlab_rethinking_how_we_handle_access_control/ [17:05:14] brennen: oh, good to know, thanks for the link! [18:19:07] (03CR) 10Dduvall: [C: 04-1] "Sorry to come in late on this. I've left a nit re: the error message but it looks good to merge once that's addressed." [tools/scap] - 10https://gerrit.wikimedia.org/r/765649 (https://phabricator.wikimedia.org/T295494) (owner: 10Jeena Huneidi) [20:01:26] what defines whether I have the permission to theoretically change the visibility of task? I can't really spot the pattern. [20:05:05] Project beta-update-databases-eqiad build #57103: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/57103/ [20:05:27] the projects you are a member of, and the task subtype which defines which edit forms can be used [20:10:39] ah ok, a pattern shows up on the second look, I have the permission when someone use the normal 'new task' edit form but not when someone uses the 'request a software feature' or the 'report a software bug' form (for old task that starts to differ) [20:30:52] i'm seeing tests failing with: [20:30:54] 21:09:51 fatal: unable to look up contint2001.wikimedia.org (port 9418) (Temporary failure in name resolution) [20:31:01] is it just me? [20:33:04] oh, hmm, looks like it works now and just failed once. i misread and thought it failed twice in a row [20:33:11] andrewbogott: ^ [20:33:23] MatmaRex: I might have a guess to the cause [20:33:42] MatmaRex: yes, I fat-fingered my way to a 2-minute dns outage, should be fixed now (although I'm still working in the area) [20:34:11] right, no worries, sorry [20:34:23] i wouldn't complain about a momentary one-off failure [20:34:45] but i misread the gerrit comments and thought that it has happened again after rerunning the tests [20:36:02] 10GitLab (Project Migration), 10Release-Engineering-Team (Done by Feb 23 🧟): Create new GitLab project group for Wikimedia Italia - https://phabricator.wikimedia.org/T301791 (10valerio.bozzolan) Just a question. I see that it is possible for individuals to create private repositories, but in this group it is n... [20:52:49] (03PS1) 10Accraze: inference-services: draftquality cleanup [integration/config] - 10https://gerrit.wikimedia.org/r/768785 (https://phabricator.wikimedia.org/T294419) [20:59:54] 10GitLab (Project Migration), 10Release-Engineering-Team (Done by Feb 23 🧟): Create new GitLab project group for Wikimedia Italia - https://phabricator.wikimedia.org/T301791 (10brennen) As on our Gerrit instance, we don't support private repositories on GitLab currently. GitLab Issues are also disabled here,... [21:05:02] Project beta-update-databases-eqiad build #57104: 15ABORTED in 45 min: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/57104/ [22:24:55] Project mediawiki-core-doxygen-docker build #32418: 04FAILURE in 20 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/32418/ [23:20:32] Yippee, build fixed! [23:20:32] Project mediawiki-core-doxygen-docker build #32419: 09FIXED in 16 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/32419/ [23:45:06] hashar: I'm drafting a doxygen update to fix T295111, which I confimed locally is fixed in 1.9.x doxygen. [23:45:06] T295111: The Doxygen documentation for WANObjectCache::getWithSetCallback has a formatting error - https://phabricator.wikimedia.org/T295111 [23:46:01] I see that we're currently using a backported package for Buster. Do we need to keep that, or is it okay to use upstream Bullseye directly? That seems to come with 1.9.1 [23:46:16] I assume it's okay so I'll start drafting it, but I'll cc you on the patch. It's not urgent :) [23:48:13] 10GitLab (Project Migration), 10Release-Engineering-Team (Done by Feb 23 🧟): Create new GitLab project group for Wikimedia Italia - https://phabricator.wikimedia.org/T301791 (10valerio.bozzolan) Understood. Then if the staff needs to create private Tasks, do you suggest to request a Phabricator Space instead?...