[06:39:44] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: gerrit: Add Envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11752448 (10ABran-WMF) [06:41:22] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: gerrit: Add Envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11752450 (10ABran-WMF) 05In progress→03Resolved >>! In T420909#11747537, @ABran-WMF wrote: >>>! In T420909#11742909, @ABra... [07:26:16] 10Gerrit, 06Release-Engineering-Team, 14Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 06collaboration-services, and 2 others: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when interacting with repos - https://phabricator.wikimedia.org/T246763#11752493 (10ABran-WMF) 05Ope... [07:46:24] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07ci-test-error (WMF-deployed Build Failure): Tests failing with "23:12:20 stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/REPO: GnuTLS recv error (-54): Erro... - https://phabricator.wikimedia.org/T420865#11752523 [07:51:11] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11752537 (10ItsNyoty) > But.. Facebook? Probably their Meta AI? [07:53:44] 10Phabricator: Move "Task search" to the top of the default phabricator sidebar nav - https://phabricator.wikimedia.org/T420801#11752540 (10Aklapper) I don't think we should destroy the global quick search in the upper right corner. I'd rather like to introduce a more limited default scope than now. See T397558. [08:21:37] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic, 13Patch-For-Review: ATS: align ATS and Gerrit Apache timeouts to reenable connection re-use - https://phabricator.wikimedia.org/T417998#11752564 (10hashar) The connection-reuse was enabled as part of introducing Envoy between... [09:14:42] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07ci-test-error (WMF-deployed Build Failure): Tests failing with "23:12:20 stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/REPO: GnuTLS recv error (-54): Erro... - https://phabricator.wikimedia.org/T420865#11752665 [09:31:30] (03update) 10hoo: Remove (broken) develop from default XDEBUG_MODE [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/643 [10:40:23] (03PS8) 10Zfilipin: Add devcontainer.json [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250501 (https://phabricator.wikimedia.org/T421236) [10:41:18] (03CR) 10Zfilipin: "Done in PS8." [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250501 (https://phabricator.wikimedia.org/T421236) (owner: 10Zfilipin) [10:41:45] (03PS9) 10Zfilipin: Add devcontainer.json [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250501 (https://phabricator.wikimedia.org/T421236) [10:42:06] (03CR) 10Zfilipin: "PS9 is a rebase." [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250501 (https://phabricator.wikimedia.org/T421236) (owner: 10Zfilipin) [10:46:50] (03CR) 10Hashar: [C:03+2] "@zfilipin@wikimedia.org confirmed the Sphinx documentation can now be generated!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250501 (https://phabricator.wikimedia.org/T421236) (owner: 10Zfilipin) [10:54:47] 06Release-Engineering-Team, 06Infrastructure-Foundations: Add tox-uv support to the tox-v{3,4} Docker images - https://phabricator.wikimedia.org/T421348 (10elukey) 03NEW [10:58:26] (03CR) 10CI reject: [V:04-1] Add devcontainer.json [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250501 (https://phabricator.wikimedia.org/T421236) (owner: 10Zfilipin) [11:03:37] (03Merged) 10jenkins-bot: Add devcontainer.json [integration/quibble] - 10https://gerrit.wikimedia.org/r/1250501 (https://phabricator.wikimedia.org/T421236) (owner: 10Zfilipin) [11:06:55] 06Release-Engineering-Team, 06Infrastructure-Foundations: Add tox-uv support to the tox-v{3,4} Docker images - https://phabricator.wikimedia.org/T421348#11753040 (10elukey) My understanding was that tox-uv needed to be selected in tox.ini, but after a chart with Riccardo I realized I was wrong, it is picked up... [11:39:50] 10GitLab, 06Release-Engineering-Team: Slack Automation with Gerrit and Gitlab - https://phabricator.wikimedia.org/T421259#11753133 (10DSantamaria) Hey, @dancy I have the email : plusone@wikimedia.org [11:43:21] 06Release-Engineering-Team, 06Infrastructure-Foundations: Add tox-uv support to the tox-v{3,4} Docker images - https://phabricator.wikimedia.org/T421348#11753136 (10Volans) I made a quick try to do an opt-in approach installing tox-uv via tox.ini, but it's not working easily because we manage the python versio... [12:37:40] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Phabricator does not render archived milestone tags as archived - https://phabricator.wikimedia.org/T420605#11753388 (10A_smart_kitten) I guess there may be a question of intended behaviour here. Using #temporary_accounts_major as a... [13:02:25] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Phabricator does not render archived milestone tags as archived - https://phabricator.wikimedia.org/T420605#11753544 (10Aklapper) > would an end-user infer that the entire Temporary accounts project had also been archived, as opposed... [13:19:33] (03CR) 10Jforrester: [C:03+2] build: Updating flatted to 3.4.2 [performance/WikimediaDebug] - 10https://gerrit.wikimedia.org/r/1255934 (owner: 10Libraryupgrader) [13:19:39] (03CR) 10Jforrester: [C:03+2] build: Updating flatted to 3.4.2 [performance/fresnel] - 10https://gerrit.wikimedia.org/r/1258205 (owner: 10Libraryupgrader) [13:20:51] (03CR) 10Jforrester: [C:03+2] build: Updating npm dependencies [integration/docroot] - 10https://gerrit.wikimedia.org/r/1255943 (owner: 10Libraryupgrader) [14:09:00] 06Project-Admins: Create project tag for ReaderExperiments-MobilePagePreviews - https://phabricator.wikimedia.org/T421381 (10matthiasmullie) 03NEW [14:09:03] 06Project-Admins: Create project tag for ReaderExperiments-ShareHighlight - https://phabricator.wikimedia.org/T421382 (10matthiasmullie) 03NEW [14:21:18] 06Project-Admins: Create project tag for ReaderExperiments-ShareHighlight - https://phabricator.wikimedia.org/T421382#11753970 (10A_smart_kitten) q: for this task and/or {T421381}, should the created project(s) be added to {H475}? [14:33:29] 10Beta-Cluster-Infrastructure, 06Content-Transform-Team: Maps don't display on the beta cluster; tile server offline on deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420299#11754081 (10Jgiannelos) We checked wth CTT team and it doesn't look like anyone kno... [14:51:15] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11754167 (10bd808) IP range blocks are cooked. The [[https://en.wikipedia.org/wiki/Proxy_server#Residential_proxy_(RESIP)|residential proxy]] users have found us. In the last 14 hours (s... [15:07:18] 06Release-Engineering-Team (Doing 😎), 10Catalyst (PatchDemo), 07Essential-Work: Patchdemo timing out while stream env creation logs - https://phabricator.wikimedia.org/T421394 (10jnuche) 03NEW [15:08:18] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Patchdemo timing out while streaming env creation logs - https://phabricator.wikimedia.org/T421394#11754269 (10jnuche) [15:08:51] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Patchdemo timing out while streaming env creation logs - https://phabricator.wikimedia.org/T421394#11754273 (10jnuche) [15:11:11] 10Gerrit, 06Release-Engineering-Team, 14Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 06collaboration-services, and 2 others: Jenkins job failing intermittently due to Gerrit HTTP 502 errors when interacting with repos - https://phabricator.wikimedia.org/T246763#11754283 (10dancy) Starting M... [15:18:16] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07ci-test-error (WMF-deployed Build Failure): Tests failing with "23:12:20 stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/REPO: GnuTLS recv error (-54): Erro... - https://phabricator.wikimedia.org/T420865#11754339 [15:31:48] 10Phabricator-Bot-Requests, 06Product Safety and Integrity, 06Security-Team: Create a Mailing list account to replace Slack_connector_for_TSP_team_triage - https://phabricator.wikimedia.org/T421406 (10Rsilvola) 03NEW [15:35:49] 06Release-Engineering-Team (Doing 😎), 10Catalyst (Luka Ijo Pimeja Jan), 07Essential-Work: Patchdemo timing out while streaming env creation logs - https://phabricator.wikimedia.org/T421394#11754531 (10jnuche) [15:36:02] Project mediawiki-core-phpmetrics build #700: 04FAILURE in 2 min 1 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-phpmetrics/700/ [15:36:35] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 07ci-test-error (WMF-deployed Build Failure): Tests failing with "23:12:20 stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/REPO: GnuTLS recv error (-54): Erro... - https://phabricator.wikimedia.org/T420865#11754533 [15:49:01] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.46.0-wmf.21 deployment blockers - https://phabricator.wikimedia.org/T420479#11754598 (10hashar) 05Open→03Resolved I have filed a few issues while triaging but nothing worth blocking the train or the... [16:08:17] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07ci-test-error (WMF-deployed Build Failure): Tests failing with "23:12:20 stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/REPO: G... - https://phabricator.wikimedia.org/T420865#11754694 [16:40:48] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, 10Browser Test Platform, 10Cloud-VPS (Quota-requests): New flavor for the integration project with more vCPU and ephemeral disk space - https://phabricator.wikimedia.org/T421242#11754852 (10Andrew) +1 this seems fine; wo... [17:04:09] 10GitLab, 06Release-Engineering-Team: Slack Automation with Gerrit and Gitlab - https://phabricator.wikimedia.org/T421259#11754917 (10dancy) 05Open→03Resolved a:03dancy I communicated the API token to @DSantamaria. I think we're done. [17:12:28] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 07Jenkins, 10Browser Test Platform, 10Cloud-VPS (Quota-requests): New flavor for the integration project with more vCPU and ephemeral disk space - https://phabricator.wikimedia.org/T421242#11754940 (10hashar) The 8cores one (`g4.co... [18:00:37] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07ci-test-error (WMF-deployed Build Failure): Tests failing with "23:12:20 stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/REPO: G... - https://phabricator.wikimedia.org/T420865#11755081 [18:10:17] 10Beta-Cluster-Infrastructure, 06Content-Transform-Team, 13Patch-For-Review: Maps don't display on the beta cluster; tile server offline on deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420299#11755192 (10bd808) 05Open→03In progress a:03Arlolra [18:31:40] Project mediawiki-core-doxygen build #18974: 04FAILURE in 2 min 1 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18974/ [18:39:01] 10Beta-Cluster-Infrastructure, 06Content-Transform-Team (Work In Progress), 07Essential-Work, 13Patch-For-Review: Maps don't display on the beta cluster; tile server offline on deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420299#11755297 (10ABreault-WM... [19:02:15] beta cluster seems down? I just deployed [Use prod to serve maps in labs (1261515)](https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1261515) but that *seems* like it should be unrelated? [19:07:01] (A_smart_kitten says this is mostly likely "just" T420833) [19:07:02] T420833: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833 [19:21:21] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11755541 (10bd808) Throw a big pile of wide blocks and hope it works for a bit... `lang=shell-session root@deployment-mediawiki14:~# ./big-ban-hammer.sh 10000 +1 | sort -n -k 4 - 78.... [19:31:09] Yippee, build fixed! [19:31:10] Project mediawiki-core-doxygen build #18975: 09FIXED in 13 min: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen/18975/ [20:47:44] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07ci-test-error (WMF-deployed Build Failure): Tests failing with "23:12:20 stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/REPO: G... - https://phabricator.wikimedia.org/T420865#11756032 [20:51:26] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07ci-test-error (WMF-deployed Build Failure): Tests failing with "23:12:20 stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/REPO: G... - https://phabricator.wikimedia.org/T420865#11756051 [20:51:39] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic: gerrit: Add Envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11756053 (10hashar) [20:51:41] 10Continuous-Integration-Infrastructure, 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 07ci-test-error (WMF-deployed Build Failure): Tests failing with "23:12:20 stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/REPO: ... - https://phabricator.wikimedia.org/T420865#11756056 [20:53:13] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic: gerrit: Add Envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11756057 (10DLynch) 05Resolved→03Open Reopening since the merged issue is actively ongoing. [21:09:19] !log patchdemo: deploy to production for https://gitlab.wikimedia.org/repos/test-platform/catalyst/patchdemo/-/merge_requests/312 [21:09:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:11:23] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic: gerrit: Add Envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11756101 (10hashar) We had reports of 502 issues on T420865 (which I have marked as a dupe of this one). After Gerrit was moved to Envoy I did notice A... [21:19:58] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic: gerrit: Add Envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11756114 (10hashar) Envoy configuration for Gerrit is in Puppet at `hieradata/role/common/gerrit.yaml`. It has configuration bits for ATS/CDN: ` lang=y... [21:21:01] 10Beta-Cluster-Infrastructure, 06Content-Transform-Team (Work In Progress), 07Essential-Work: Maps don't display on the beta cluster; tile server offline on deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420299#11756116 (10bd808) 05In progress→03Re... [21:25:22] !log Unblock 89.240.0.0/15 (T421364) [21:25:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:28:42] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-cirrussearch12 is down - https://phabricator.wikimedia.org/T420642#11756139 (10bd808) 05Open→03Invalid [21:40:43] 10Beta-Cluster-Infrastructure, 06Content-Transform-Team (Work In Progress), 07Essential-Work: Maps don't display on the beta cluster; tile server offline on deployment-maps-master02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T420299#11756165 (10matmarex) Thanks y'all. I w... [21:40:45] (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331) [21:41:15] (03update) 10dduvall: policy: Use Rego policies instead of Go templates [repos/releng/jwt-authorizer] - 10https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/22 (https://phabricator.wikimedia.org/T346331) [21:46:46] 10Beta-Cluster-Infrastructure, 05Account-Vanishing: Define how vanishing requests are processed on Wikimedia beta cluster - https://phabricator.wikimedia.org/T383514#11756175 (10bd808) >>! In T383514#10533920, @whym wrote: >> Where would you expect to find the documentation you are hoping to see added? > > My... [21:53:37] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11756190 (10bd808) `81.0.0.0/8` caught some folks who PM'd me. I removed that block since it was part of the "close the gate after the horses left" wild blocks anyway. https://gerrit.wik... [21:58:14] 10Phabricator (Search), 06Release-Engineering-Team (Priority Backlog 📥): Have a more scope-restricted default global (primary) search - https://phabricator.wikimedia.org/T397558#11756212 (10A_smart_kitten) >>! In T397558#10938122, @A_smart_kitten wrote: > Copying over what I said in T397392 regarding this sort... [22:23:31] 10Beta-Cluster-Infrastructure: Beta cluster is slow as sludge / serves 503 and 504 - https://phabricator.wikimedia.org/T420833#11756262 (10bd808) Another unblock request: `lang=shell-session root@deployment-mediawiki14:~# ./subtractNetworks.py 82.0.0.0/8 82.64.0.0/14 abuse_networks: blocked_nets: networks:... [22:44:38] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic: gerrit: Add Envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11756344 (10Dzahn) Looking directly at the file system, to bypass any possible issues with Hiera or Puppet, I see this: envoy: ` clusters.d/00-cluster... [23:00:23] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic: gerrit: Add Envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11756356 (10Dzahn) If that hypothesis was true the issues started with https://gerrit.wikimedia.org/r/c/operations/puppet/+/1241048/2/modules/gerrit/tem... [23:02:53] 10Phabricator-Bot-Requests, 06Product Safety and Integrity, 06Security-Team: Create a Mailing list account to replace Slack_connector_for_TSP_team_triage - https://phabricator.wikimedia.org/T421406#11756362 (10Aklapper) > Please also disable all email notification types in Settings → Email → Email Preference... [23:22:29] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 06Traffic: gerrit: Add Envoy in Gerrit's stack - https://phabricator.wikimedia.org/T420909#11756424 (10hashar) @Dzahn the Jetty timeout is independent to the problem. It is for the {nav Apache > mod_proxy > Jetty} chain. We had that issue seve... [23:49:55] 10Continuous-Integration-Infrastructure (Zuul upgrade), 06collaboration-services: puppetize setup of new zuul VMs - https://phabricator.wikimedia.org/T395938#11756448 (10Dzahn) Debugging the following issue we see on any zuul-related service that starts as the second service after the first one has connected t... [23:53:14] 10Beta-Cluster-Infrastructure: Clean up broken legacy maps stack in Beta Cluster - https://phabricator.wikimedia.org/T421449 (10bd808) 03NEW [23:54:42] 10Beta-Cluster-Infrastructure: Clean up broken legacy maps stack in Beta Cluster - https://phabricator.wikimedia.org/T421449#11756477 (10bd808) [23:56:03] 10Beta-Cluster-Infrastructure: Clean up broken legacy maps stack in Beta Cluster - https://phabricator.wikimedia.org/T421449#11756484 (10bd808) [23:56:08] 10Beta-Cluster-Infrastructure, 06Release-Engineering-Team (Priority Backlog 📥), 10Cloud-VPS (Debian Bullseye Deprecation), 07Epic: Migrate deployment-prep away from Debian Bullseye to Bookworm/Trixie - https://phabricator.wikimedia.org/T401839#11756483 (10bd808)