[00:04:21] (03CR) 10Subramanya Sastry: "Thanks ... good point about my use of "for now" :-) ... only if we can arrive at a good strategy for false positives without impacting fal" [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/1116909 (https://phabricator.wikimedia.org/T383166) (owner: 10Subramanya Sastry) [00:05:05] subbu: There is a deployment-parsoid14.deployment-prep.eqiad1.wikimedia.cloud instance in deployment-prep. I have a hunch that this is a vestigial server as I don't see anything in production using the same `::role::parsoid` configuration. Does that seem right to you? [00:05:48] hmm ... [00:06:03] https://openstack-browser.toolforge.org/server/deployment-parsoid14.deployment-prep.eqiad1.wikimedia.cloud [00:06:23] I wonder if it was the old parsoid/js server for beta cluster. [00:06:47] The instance itself was built by S.outhparkfan when they were being very helpful in updating operating systems. [00:07:03] Created: 2024-07-20T13:51:10Z ... so probably not nodejs [00:07:30] subbu: the old nodejs stuff would be my guess. there is a parsoid.svc.deployment-prep.eqiad1.wikimedia.cloud service name pointed at the instance [00:07:57] so, your theory is that the old nodejs vm was recreated in 2024 when OS was upgraded? [00:08:20] you can shut it down and see if any beta cluster services break ... in which case you can turn it on again? [00:08:53] I have no memory of spinning up any VMs for parsoid-php because we wouldn't need that. [00:10:12] :nod: a shutdown and look for breakage was my next idea if you didn't know for sure :) [00:11:23] T361386 was why the instance was built [00:11:23] T361386: Remove or replace deployment-parsoid12.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361386 [00:19:33] ya .. my hunch is that it is indeed the parsoid/js baggage getting carried around :) [00:19:49] * subbu siggning off now [00:21:57] dancy: is the traindev01.deployment-prep.eqiad1.wikimedia.cloud instance still useful to you? [00:22:38] Yes. [00:23:15] cool. I will add it to the inventory I'm updating then. [00:40:02] 10Beta-Cluster-Infrastructure: Determine usefulness of questionable hosts found in 2025-02-06 audit - https://phabricator.wikimedia.org/T385849 (10bd808) 03NEW [00:40:25] 10Beta-Cluster-Infrastructure: Determine usefulness of questionable hosts found in 2025-02-06 audit - https://phabricator.wikimedia.org/T385849#10530914 (10bd808) 05Openβ†’03In progress p:05Triageβ†’03Medium a:03bd808 [00:42:36] !log Shutoff deployment-parsoid14 to see if anything breaks/anyone yells (T385849) [00:42:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:42:39] T385849: Determine usefulness of questionable hosts found in 2025-02-06 audit - https://phabricator.wikimedia.org/T385849 [00:47:37] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Wikimedia-Phabricator-Extensions: Dead code and non-existing classes mentioned in WMFSecurityPolicy.php - https://phabricator.wikimedia.org/T385850 (10Aklapper) 03NEW p:05Triageβ†’03Low [00:50:09] 10Beta-Cluster-Infrastructure: Determine usefulness of questionable hosts found in 2025-02-06 audit - https://phabricator.wikimedia.org/T385849#10530935 (10bd808) >>! In T385849#10530920, @Stashbot wrote: > {nav icon=file, name=Mentioned in SAL (#wikimedia-releng), href=https://sal.toolforge.org/log/Ue3b3ZQB8tZ8... [00:57:12] Project beta-scap-sync-world build #192737: 04FAILURE in 12 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192737/ [01:02:37] Project beta-scap-sync-world build #192738: 04STILL FAILING in 2 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192738/ [01:06:36] Project beta-scap-sync-world build #192739: 04STILL FAILING in 1 min 42 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192739/ [01:15:50] The scap failures are because I shut off deployment-parsoid14 for T385849 [01:15:51] T385849: Determine usefulness of questionable hosts found in 2025-02-06 audit - https://phabricator.wikimedia.org/T385849 [01:16:31] I should be able to take it out of the scap group via hiera I think... [01:16:39] Project beta-scap-sync-world build #192740: 04STILL FAILING in 1 min 45 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192740/ [01:26:42] Project beta-scap-sync-world build #192741: 04STILL FAILING in 1 min 48 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192741/ [01:34:44] !log Cherry-picked https://gerrit.wikimedia.org/r/c/operations/puppet/+/1117997 to deployment-puppetmaster (T385849) [01:34:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:34:46] T385849: Determine usefulness of questionable hosts found in 2025-02-06 audit - https://phabricator.wikimedia.org/T385849 [01:35:05] that should fix sync-world on its next run [01:36:39] Yippee, build fixed! [01:36:39] Project beta-scap-sync-world build #192742: 09FIXED in 1 min 47 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/192742/ [06:14:34] Project beta-code-update-eqiad build #534073: 04FAILURE in 1 min 33 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/534073/ [06:24:58] Yippee, build fixed! [06:24:58] Project beta-code-update-eqiad build #534074: 09FIXED in 1 min 57 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/534074/ [06:57:34] Project Wikibase-phpmetrics build #238: 04FAILURE in 33 sec: https://integration.wikimedia.org/ci/job/Wikibase-phpmetrics/238/ [08:13:30] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Wikimedia-Phabricator-Extensions: Dead code and non-existing classes mentioned in WMFSecurityPolicy.php - https://phabricator.wikimedia.org/T385850#10531256 (10Aklapper) 05Openβ†’03Stalled a:03Aklapper Pushed https://gitlab.wikimedia.org/r... [10:51:04] 10GitLab (Pipeline Services Migration🐀), 06collaboration-services, 10Wikidata, 10Wikidata Query UI, and 3 others: move query.wikidata.org to kubernetes - https://phabricator.wikimedia.org/T350793#10531475 (10Jelto) >>! In T350793#10528533, @WMDE-leszek wrote: > Thanks @Jelto for making the move, and addres... [11:45:32] 10Phabricator, 10Wikimedia-Phabricator-Extensions: SecurityPolicyEnforcerAction: !empty($forced_policies) is always falsy - https://phabricator.wikimedia.org/T385872 (10Aklapper) 03NEW [13:58:32] (03approved) 10jnuche: Add pyproject.toml [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/660 (owner: 10dancy) [14:39:53] (03open) 10gkyziridis: Draft: uv_testing: uv package manager implementation in blubber. [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/119 (https://phabricator.wikimedia.org/T384584) [15:26:35] (03update) 10gkyziridis: Draft: uv_testing: uv package manager implementation in blubber. [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/119 (https://phabricator.wikimedia.org/T384584) [15:43:19] (03update) 10gkyziridis: Draft: uv_testing: uv package manager implementation in blubber. [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/119 (https://phabricator.wikimedia.org/T384584) [15:57:57] 10Beta-Cluster-Infrastructure, 07OKR-Work, 13Patch-For-Review: Install PHP 8.1 on deployment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752#10532057 (10bd808) >>! In T378752#10531568, @gerritbot wrote: > Change #1118095 had a related patch set uploaded (by ClΓ©ment Goubert; auth... [16:00:39] (03update) 10dancy: Add pyproject.toml [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/660 [16:06:44] 10Beta-Cluster-Infrastructure, 07OKR-Work, 13Patch-For-Review: Install PHP 8.1 on deployment-prep bullseye MediaWiki instances - https://phabricator.wikimedia.org/T378752#10532088 (10Clement_Goubert) >>! In T378752#10532057, @bd808 wrote: >>>! In T378752#10531568, @gerritbot wrote: >> Change #1118095 had a r... [16:06:52] (03update) 10dancy: Add pyproject.toml [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/660 [16:14:19] (03open) 10dancy: fake_backport.py: Require at least one change number/url [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/661 [16:14:22] (03update) 10dancy: fake_backport.py: Require at least one change number/url [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/661 [16:15:03] maintenance-disconnect-full-disks build 673963 integration-agent-docker-1047 (/: 26%, /srv: 96%, /var/lib/docker: 37%): OFFLINE due to disk space [16:18:07] (03update) 10dancy: fake_backport.py: Require at least one change number/url [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/661 [16:20:03] maintenance-disconnect-full-disks build 673964 integration-agent-docker-1047 (/: 26%, /srv: 41%, /var/lib/docker: 36%): RECOVERY disk space OK [16:20:16] (03merge) 10dancy: fake_backport.py: Require at least one change number/url [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/661 [16:26:04] (03update) 10dancy: Add pyproject.toml [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/660 [16:28:09] (03merge) 10dancy: Add pyproject.toml [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/660 [16:30:58] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap (SpiderPig πŸ•ΈοΈ), 06collaboration-services: Scap SpiderPig: Routing for the web frontend - https://phabricator.wikimedia.org/T383946#10532165 (10Dzahn) I was phrasing it like this because now it's about configuring services on deployment servers. And sin... [16:34:54] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap (SpiderPig πŸ•ΈοΈ), 06collaboration-services: Scap SpiderPig: Routing for the web frontend - https://phabricator.wikimedia.org/T383946#10532177 (10dancy) Thanks @Dzahn. I understand now. [17:02:59] (03update) 10gkyziridis: Draft: uv_testing: uv package manager implementation in blubber. [repos/releng/blubber] - 10https://gitlab.wikimedia.org/repos/releng/blubber/-/merge_requests/119 (https://phabricator.wikimedia.org/T384584) [19:03:01] 10Release-Engineering-Team (Doing 😎), 05Release, 05Train Deployments: 1.44.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T382366#10532535 (10thcipriani) [[https://www.mediawiki.org/wiki/MediaWiki_1.44/wmf.15|MediaWiki_1.44/wmf.15]] - 272 Patches β–ˆβ–β–‚β–‚β–‚ -- in 70 repos -- by 78 authors -... [19:03:47] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Doing 😎), 07Regression: Show related Gitlab patches on Phabricator tasks - https://phabricator.wikimedia.org/T385480#10532537 (10taavi) >>! In T385480#10532529, @thcipriani wrote: > But where does the new token go? (NOTE: I have not done t... [19:14:48] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Doing 😎), 07Regression: Show related Gitlab patches on Phabricator tasks - https://phabricator.wikimedia.org/T385480#10532581 (10brennen) > There's profile::phabricator::main::gitlab_api_key in private Hiera which seems likely. Yeah, that... [19:59:42] 06Release-Engineering-Team, 10Scap: scap deploy-promote depends on Toolforge service - https://phabricator.wikimedia.org/T333924#10532662 (10dancy) 05Openβ†’03In progress p:05Triageβ†’03Low a:03dancy [20:07:07] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Doing 😎), 07Regression: Show related Gitlab patches on Phabricator tasks - https://phabricator.wikimedia.org/T385480#10532672 (10thcipriani) Rotated the token. The new on is in: ` thcipriani@phab1004:~$ ls -lhA 2025-02-07-T385480.token -r... [20:11:20] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Doing 😎), 07Regression: Show related Gitlab patches on Phabricator tasks - https://phabricator.wikimedia.org/T385480#10532678 (10taavi) >>! In T385480#10532672, @thcipriani wrote: > Just need an SRE to update that private heira value with... [20:13:15] (03open) 10dancy: Handle inaccessible train-blockers.toolforge.org [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/662 (https://phabricator.wikimedia.org/T333924) [20:13:16] (03update) 10dancy: Handle inaccessible train-blockers.toolforge.org [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/662 (https://phabricator.wikimedia.org/T333924) [20:18:01] (03update) 10dancy: Handle inaccessible train-blockers.toolforge.org [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/662 (https://phabricator.wikimedia.org/T333924) [20:18:37] (03update) 10dancy: Handle inaccessible train-blockers.toolforge.org [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/662 (https://phabricator.wikimedia.org/T333924) [20:53:51] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Doing 😎), 07Regression: Show related Gitlab patches on Phabricator tasks - https://phabricator.wikimedia.org/T385480#10532751 (10thcipriani) >>! In T385480#10532678, @taavi wrote: >>>! In T385480#10532672, @thcipriani wrote: >> Just need a... [21:31:12] 10Beta-Cluster-Infrastructure: Puppet broken on deployment-restbase05 - https://phabricator.wikimedia.org/T385902#10532810 (10bd808) This looks to be fallout from https://gerrit.wikimedia.org/r/c/operations/puppet/+/1117997. When `profile::restbase::parsoid_uri` is not found in a hiera lookup Puppet tries to con... [21:33:55] !log Added `profile::restbase::parsoid_uri: https://phabricator.wikimedia.org/T385902` to deployment-restbase prefix puppet (T385902) [21:33:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:33:57] T385902: Puppet broken on deployment-restbase05 - https://phabricator.wikimedia.org/T385902 [21:34:58] 10Beta-Cluster-Infrastructure: Puppet broken on deployment-restbase05 - https://phabricator.wikimedia.org/T385902#10532826 (10bd808) 05Openβ†’03Resolved a:03bd808 [21:37:07] 10Beta-Cluster-Infrastructure: Puppet broken on deployment-ms-fe04 - https://phabricator.wikimedia.org/T385912 (10bd808) 03NEW [21:46:51] (03open) 10dancy: setup.py: Use minimal positive setup [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/663 [21:46:54] (03update) 10dancy: setup.py: Use minimal positive setup [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/663 [21:48:51] (03merge) 10dancy: setup.py: Use minimal positive setup [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/663 [22:00:37] 10Beta-Cluster-Infrastructure: Puppet broken on deployment-ms-fe04 - https://phabricator.wikimedia.org/T385912#10532897 (10bd808) `lang=puppet $present_pools = $pools.keys() $services = wmflib::service::fetch(true).filter |$lvs_name, $svc| {$lvs_name in $present_pools} include profile::lvs::configura... [22:13:05] 10Beta-Cluster-Infrastructure, 07Documentation: Document post-restart activities needed across deployment-prep services - https://phabricator.wikimedia.org/T385822#10532933 (10bd808) [22:13:07] 10Beta-Cluster-Infrastructure: deployment-prep is broken following 2025-02-06 WMCS reboots - https://phabricator.wikimedia.org/T385803#10532934 (10bd808) [22:13:12] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Doing 😎), 07Regression: Show related Gitlab patches on Phabricator tasks - https://phabricator.wikimedia.org/T385480#10532935 (10brennen) `{{done}}` - thanks all. I'll spend some time documenting various GitLab tokens and maybe set up a ca... [22:13:30] 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team (Doing 😎), 07Regression: Show related Gitlab patches on Phabricator tasks - https://phabricator.wikimedia.org/T385480#10532937 (10brennen) 05Openβ†’03Resolved [22:13:32] 10Beta-Cluster-Infrastructure: deployment-prep is broken following 2025-02-06 WMCS reboots - https://phabricator.wikimedia.org/T385803#10532939 (10bd808) [22:13:35] 10Beta-Cluster-Infrastructure, 13Patch-For-Review: Determine usefulness of questionable hosts found in 2025-02-06 audit - https://phabricator.wikimedia.org/T385849#10532938 (10bd808) [22:14:17] !log phab/phorge: replaced mr-widget token in deployed config (T385480) [22:14:19] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:14:19] T385480: Show related Gitlab patches on Phabricator tasks - https://phabricator.wikimedia.org/T385480 [22:19:45] yay mr. widget [22:20:32] (I'd forgotten it was named it "Mr. Widget" until today when I realized how unsearchable that name is :D) [22:20:47] (but still a fan of the name) [22:30:36] 10Continuous-Integration-Infrastructure, 10Quibble: Add support for multi-DB testing to MediaWiki - https://phabricator.wikimedia.org/T379925#10533025 (10aaron) >>! In T379925#10370224, @kostajh wrote: >>>! In T379925#10362231, @hashar wrote: >> For T342763, @Urbanecm proposed a patch to Quibble to have it spi... [22:51:40] 10Beta-Cluster-Infrastructure: Puppet broken on deployment-ms-fe04 - https://phabricator.wikimedia.org/T385912#10533070 (10bd808) The code paths in Puppet are so dark and twisty that I'm afraid I will bump into a grue. [22:57:53] i had likewise completely forgotten mr. widget somehow [22:58:09] 10Beta-Cluster-Infrastructure: Puppet broken on deployment-ms-fe04 - https://phabricator.wikimedia.org/T385912#10533074 (10bd808) Bodging these hiera settings into deployment-ms-fe prefix puppet got the catalog to compile: `lang=yaml profile::lvs::realserver::pools: swift: services: - dummy ` `lang=yam... [22:59:03] 10Beta-Cluster-Infrastructure: Puppet broken on deployment-ms-fe04 - https://phabricator.wikimedia.org/T385912#10533078 (10bd808) 05Openβ†’03Resolved a:03bd808 [23:00:48] 10Beta-Cluster-Infrastructure: Puppet broken on deployment-maps-master02 - https://phabricator.wikimedia.org/T385921 (10bd808) 03NEW