[00:15:46] 10Continuous-Integration-Infrastructure (Zuul upgrade): Update Magnum cluster config to use higher iops volume type and flavor - https://phabricator.wikimedia.org/T410290#12273377 (10bd808) 05Open→03Resolved a:03dduvall [00:18:12] 06Release-Engineering-Team (Doing 😎), 10MW-on-K8s, 10ServiceOps-Mediawiki, 13Patch-For-Review: Update deployment documentation to factor in pretrain deployment windows - https://phabricator.wikimedia.org/T436187#12273380 (10bd808) 05In progress→03Resolved a:03bd808 [03:30:45] (03open) 10phedenskog: Measure core feedback time per patch run [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/55 [04:09:02] 10MediaWiki-Releasing, 10MediaWiki-extensions-CodeMirror: Bundle Extension:CodeMirror with MediaWiki core - https://phabricator.wikimedia.org/T391926#12273838 (10MusikAnimal) [07:28:19] (03CR) 10Hashar: "Indeed and I am quite happy to have discovered they were broken, albeit indirectly 😄 Thank you for the review/approval!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1332724 (https://phabricator.wikimedia.org/T436226) (owner: 10Hashar) [07:35:10] (03PS2) 10Hashar: tests: migrate setUp mocking to per tests decorator [integration/quibble] - 10https://gerrit.wikimedia.org/r/1331540 (https://phabricator.wikimedia.org/T436226) [07:35:11] (03PS2) 10Hashar: tests: rename classes to match pytest default (Test*) [integration/quibble] - 10https://gerrit.wikimedia.org/r/1331541 (https://phabricator.wikimedia.org/T436226) [07:35:11] (03PS2) 10Hashar: tests: drop unittest.TestCase when it is not used [integration/quibble] - 10https://gerrit.wikimedia.org/r/1331547 (https://phabricator.wikimedia.org/T436226) [07:35:11] (03PS3) 10Hashar: tests: replace TestCase helpers with raw assertions [integration/quibble] - 10https://gerrit.wikimedia.org/r/1331654 (https://phabricator.wikimedia.org/T436226) [07:45:39] (03PS1) 10Hashar: tests: migrate assertRaisesRegex to pytest.raises [integration/quibble] - 10https://gerrit.wikimedia.org/r/1333044 (https://phabricator.wikimedia.org/T436226) [08:00:08] (03CR) 10CI reject: [V:04-1] tests: migrate assertRaisesRegex to pytest.raises [integration/quibble] - 10https://gerrit.wikimedia.org/r/1333044 (https://phabricator.wikimedia.org/T436226) (owner: 10Hashar) [08:12:25] (03merge) 10phedenskog: Measure core feedback time per patch run [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/55 [08:22:48] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 06DC-Ops, 10ops-eqiad, 06SRE: Check and fix BIOS on new Supermicro cloudvirt* systems - https://phabricator.wikimedia.org/T435537#12274272 (10elukey) As FYI for Dells we set: ` 'EnergyPerformanceBias': 'BalancedPerformance', ` But afa... [08:24:09] (03open) 10phedenskog: Follow the castor 0.4.5 section naming in the dashboard queries [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/56 [08:43:00] 10Phabricator, 06Release-Engineering-Team (Doing 😎): Obsolete Sprint extension settings in projects.fields database config value - https://phabricator.wikimedia.org/T429866#12274325 (10Aklapper) 05Open→03Resolved ` aklapper@phab1005:/srv/deployment/phabricator/deployment/phabricator$ sudo ./bin/config... [08:52:17] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Add mouse-over popup tooltips for buttons when viewing images in Phab - https://phabricator.wikimedia.org/T281793#12274343 (10Aklapper) 05Open→03Stalled [08:52:18] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Missing space between two Diffusion strings / messages - https://phabricator.wikimedia.org/T304205#12274342 (10Aklapper) 05Open→03Stalled [08:52:20] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Don't offer "Show Hidden Columns" when there are no hidden columns (Manage Board) - https://phabricator.wikimedia.org/T90779#12274345 (10Aklapper) 05Open→03Stalled [08:52:21] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07CSS, 07Upstream: On small screens, add a button to remove an action from a list of actions - https://phabricator.wikimedia.org/T224276#12274344 (10Aklapper) 05Open→03Stalled [08:54:18] 10Diffusion, 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Attempting to view a non-existent file through Diffusion causes an unhandled CommandException - https://phabricator.wikimedia.org/T435627#12274354 (10Aklapper) 05Open→03Stalled [09:20:04] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 2 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [09:20:30] 10Beta-Cluster-Infrastructure: Found non-revoked Puppet certificates for 2 deleted instances on deployment-puppetserver-1 - https://phabricator.wikimedia.org/T436619 (10wmcs-alerts) 03NEW [09:22:34] 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: install a service on phab1005 - https://phabricator.wikimedia.org/T377889#12274431 (10Jelto) >>! In T377889#12242438, @Arnoldokoth wrote: > @Aklapper I was hoping to use this to track decommissioning `phab1004`. I think once we've done... [10:13:42] 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: install a service on phab1005 - https://phabricator.wikimedia.org/T377889#12274661 (10Aklapper) For the sake of finding tickets again I'd also prefer a separate ticket for phab1004 decommission. This ticket is called "install a service... [10:43:04] 10Continuous-Integration-Infrastructure, 10Castor, 10Cloud-VPS (Quota-requests), 06Test Platform (Arusha 30): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12274757 (10fnegri) > tofu-infra defines no Cinder volume types or QoS... [10:54:14] (03merge) 10phedenskog: Follow the castor 0.4.5 section naming in the dashboard queries [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/56 [12:21:09] 10Phabricator, 06Infrastructure-Foundations, 10SRE-tools, 13Patch-For-Review: offboard-user: Replace deprecated (frozen) Phabricator Conduit API calls with their stable equivalents - https://phabricator.wikimedia.org/T420324#12275267 (10Aklapper) @MoritzMuehlenhoff Would you know who could review/test/merg... [12:32:58] (03PS1) 10Phedenskog: jjb: codehealth: skip browser binary downloads during npm ci [integration/config] - 10https://gerrit.wikimedia.org/r/1333188 (https://phabricator.wikimedia.org/T427471) [12:36:44] (03PS1) 10Phedenskog: jjb: codehealth: garbage collect the npm cache before the castor save [integration/config] - 10https://gerrit.wikimedia.org/r/1333189 (https://phabricator.wikimedia.org/T427471) [12:37:22] 10Continuous-Integration-Infrastructure, 10Castor, 13Patch-For-Review, 06Test Platform (Arusha 30): Speedup mwext-codehealth-master-non-voting Castor job - https://phabricator.wikimedia.org/T427471#12275334 (10Peter) [12:37:44] 10Continuous-Integration-Infrastructure, 10Castor, 13Patch-For-Review, 06Test Platform (Arusha 30): Speedup mwext-codehealth-master-non-voting Castor job - https://phabricator.wikimedia.org/T427471#12275335 (10Peter) a:03Peter [12:37:55] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12275337 (10ABran-WMF) Thanks to @brouberol (see {T435292} for more details), DPE will use https://gitlab.discovery.wmnet:8443 as a Gitlab endpoint. Once... [12:39:20] 10GitLab, 06Release-Engineering-Team (Radar), 06Collaboration-Services, 13Patch-For-Review: gitlab behind CDN - https://phabricator.wikimedia.org/T425441#12275339 (10ABran-WMF) [13:40:41] (03PS2) 10Hashar: tests: migrate assertRaisesRegex to pytest.raises [integration/quibble] - 10https://gerrit.wikimedia.org/r/1333044 (https://phabricator.wikimedia.org/T436226) [13:40:41] (03PS8) 10Hashar: tests: migrate assertLogs to pytest caplog [integration/quibble] - 10https://gerrit.wikimedia.org/r/1330357 (https://phabricator.wikimedia.org/T436226) [13:42:40] (03CR) 10Hashar: "Any order works really :-] I am deploying this now and I50bf35981e64afbc5e6f0ad0b97dffa3ec2180dc should be processed by CI since it has +" [integration/config] - 10https://gerrit.wikimedia.org/r/1329282 (https://phabricator.wikimedia.org/T405008) (owner: 10Mszwarc) [13:42:45] (03CR) 10Hashar: [C:03+2] Zuul: [mediawiki/extensions/CheckUser] Publish JS docs post-merge [integration/config] - 10https://gerrit.wikimedia.org/r/1329282 (https://phabricator.wikimedia.org/T405008) (owner: 10Mszwarc) [13:44:53] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/CheckUser] Publish JS docs post-merge [integration/config] - 10https://gerrit.wikimedia.org/r/1329282 (https://phabricator.wikimedia.org/T405008) (owner: 10Mszwarc) [14:11:40] (03PS2) 10Mszwarc: opensource.yaml: Add CheckUser [integration/docroot] - 10https://gerrit.wikimedia.org/r/1333214 [14:22:07] (03PS3) 10Hashar: opensource.yaml: Add CheckUser [integration/docroot] - 10https://gerrit.wikimedia.org/r/1333214 (owner: 10Mszwarc) [14:22:45] (03CR) 10Hashar: [C:03+2] "Thank! I forgot about adding the link to doc.wm.o !" [integration/docroot] - 10https://gerrit.wikimedia.org/r/1333214 (owner: 10Mszwarc) [14:23:27] (03Merged) 10jenkins-bot: opensource.yaml: Add CheckUser [integration/docroot] - 10https://gerrit.wikimedia.org/r/1333214 (owner: 10Mszwarc) [14:24:15] (03CR) 10Hashar: [C:03+2] "Deployed!" [integration/docroot] - 10https://gerrit.wikimedia.org/r/1333214 (owner: 10Mszwarc) [14:28:59] 10Phabricator, 06Release-Engineering-Team, 06Collaboration-Services: Move aphlict (Phabricator notification server) to aux-k8s - https://phabricator.wikimedia.org/T436657 (10ABran-WMF) 03NEW [14:29:14] 10Phabricator, 06Release-Engineering-Team, 06Collaboration-Services: Move aphlict to aux-k8s - https://phabricator.wikimedia.org/T436657#12275872 (10ABran-WMF) [14:31:57] (03PS2) 10Hashar: jjb: codehealth: skip browser binary downloads during npm ci [integration/config] - 10https://gerrit.wikimedia.org/r/1333188 (https://phabricator.wikimedia.org/T427471) (owner: 10Phedenskog) [14:32:39] 10Phabricator, 06Release-Engineering-Team, 06Collaboration-Services: Move aphlict to k8s - https://phabricator.wikimedia.org/T436657#12275903 (10ABran-WMF) [14:33:06] (03CR) 10Hashar: [C:03+2] "OF COURSE! What a great hack!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" [integration/config] - 10https://gerrit.wikimedia.org/r/1333188 (https://phabricator.wikimedia.org/T427471) (owner: 10Phedenskog) [14:34:55] (03Merged) 10jenkins-bot: jjb: codehealth: skip browser binary downloads during npm ci [integration/config] - 10https://gerrit.wikimedia.org/r/1333188 (https://phabricator.wikimedia.org/T427471) (owner: 10Phedenskog) [14:36:37] !log integration: deleted Cypress from codehealth job after the job learn to instruct Cypress & Puppeter to no more download binary blobs. `rm -fR /srv/castor/castor-mw-ext-and-skins/master/mwext-codehealth-master-non-voting/Cypress` # T427471 [14:36:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:36:40] T427471: Speedup mwext-codehealth-master-non-voting Castor job - https://phabricator.wikimedia.org/T427471 [14:36:47] phedenskog: that is a great finding :] Well done! [15:04:00] 06Release-Engineering-Team, 06Collaboration-Services, 13Patch-For-Review: install a service on phab1005 - https://phabricator.wikimedia.org/T377889#12276052 (10Arnoldokoth) [15:12:06] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:57:41] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Update to Phorge/Arcanist upstream 2026.xx - https://phabricator.wikimedia.org/T428547#12276428 (10Aklapper) [15:57:42] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Add mouse-over popup tooltips for buttons when viewing images in Phab - https://phabricator.wikimedia.org/T281793#12276427 (10Aklapper) [16:24:18] (03open) 10dancy: Release 4.287.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1303 [16:24:19] (03update) 10dancy: Release 4.287.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1303 [16:27:20] (03merge) 10dancy: Release 4.287.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/1303 [16:47:14] 10Beta-Cluster-Infrastructure, 07Puppet: Unduplicate beta cluster hiera keys set both in Horizon and in ops/puppet - https://phabricator.wikimedia.org/T277680#12276764 (10taavi) 05Open→03Resolved a:03taavi [16:57:12] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T430837#12276802 (10dancy) [17:09:52] 10Beta-Cluster-Infrastructure: Found non-revoked Puppet certificates for 2 deleted instances on deployment-puppetserver-1 - https://phabricator.wikimedia.org/T436619#12276870 (10Southparkfan) a:03Southparkfan [17:55:04] RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [17:57:58] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace deployment-docker-citoid02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436466#12277168 (10Southparkfan) Note: this instance has been deleted, `deployment-docker-citoid03` exists. Status unknown. [17:58:02] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace deployment-docker-cxserver02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436467#12277169 (10Southparkfan) Note: this instance has been deleted, `deployment-docker-cxserver03` exists. Status unknown. [18:01:05] 10Beta-Cluster-Infrastructure: Found non-revoked Puppet certificates for 2 deleted instances on deployment-puppetserver-1 - https://phabricator.wikimedia.org/T436619#12277188 (10Southparkfan) Certificates have been [[ https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates#Com... [18:48:20] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace deployment-webperf2[12] (Bullseye deprecation) - https://phabricator.wikimedia.org/T436464#12277371 (10Southparkfan) On `deployment-webperf22`, Arc Lamp has been broken for a long, long time: `bash Sep 01 18:21:52 deployment-webp... [19:10:29] 10Phabricator, 06Release-Engineering-Team (Doing 😎): rewrite "Feature request" maniphest form to follow the what/why pattern - https://phabricator.wikimedia.org/T387072#12277514 (10Aklapper) 05Open→03Resolved a:03Aklapper I changed Form 102 from ` **Feature summary** (what you would like to be able t... [19:24:40] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace deployment-webperf2[12] (Bullseye deprecation) - https://phabricator.wikimedia.org/T436464#12277567 (10Southparkfan) Given the fact Arc Lamp has been broken for ages anyways, I'm going to follow an adjusted procedure for migratin... [19:25:42] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.47.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T430836#12277572 (10seanleong-WMDE) [19:28:20] 10Phabricator: Unknown column 'projectGroupName.indexedObjectName' in 'HAVING' - https://phabricator.wikimedia.org/T427694#12277576 (10Aklapper) →14Duplicate dup:03T428155 [19:28:21] 10Phabricator: EXCEPTION: (AphrontSchemaQueryException) #1054: Unknown column 'projectGroupName.indexedObjectName' in 'HAVING' - https://phabricator.wikimedia.org/T428155#12277578 (10Aklapper) [19:52:06] 10GitLab (Infrastructure): Make wmf-debci a CI/CD component so it shows on GitLab's catalog - https://phabricator.wikimedia.org/T436706 (10andrea.denisse) 03NEW [19:53:09] 10GitLab (Infrastructure): Make wmf-debci a CI/CD component so it shows on GitLab's catalog - https://phabricator.wikimedia.org/T436706#12277653 (10andrea.denisse) [19:57:03] FIRING: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [19:59:04] FIRING: InstanceDown: Project deployment-prep instance deployment-webperf32 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:59:17] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-webperf32 is down - https://phabricator.wikimedia.org/T436707 (10wmcs-alerts) 03NEW [20:04:04] RESOLVED: InstanceDown: Project deployment-prep instance deployment-webperf32 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [20:07:03] RESOLVED: PuppetStaleCertificates: Found non-revoked Puppet certificates for 1 deleted instances on deployment-puppetserver-1 - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetStaleCertificates - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetStaleCertificates [20:58:02] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Phab Scap template vs actual value out of sync for numerous values - https://phabricator.wikimedia.org/T398460#12277920 (10Aklapper) p:05Triage→03Low [20:58:32] 10Phabricator, 07CSS: Allow :target to also trigger various hover effects (show hidden comments, expand Stashbot comments) - https://phabricator.wikimedia.org/T425284#12277922 (10Aklapper) p:05Triage→03Low [20:58:37] 10Phabricator, 07CSS, 07Mobile: Show the chevron to display hidden comments on mobile, somehow - https://phabricator.wikimedia.org/T425285#12277924 (10Aklapper) p:05Triage→03Low [20:58:49] 10GitLab (Integrations), 10Phabricator, 06Release-Engineering-Team, 10Wikimedia-Phabricator-Extensions, 13Patch-For-Review: Phorge GitLab widget generates writes to cache table for task views - https://phabricator.wikimedia.org/T401160#12277928 (10Aklapper) p:05Triage→03Medium [21:02:03] FIRING: InstanceDown: Project deployment-prep instance deployment-webperf22 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [21:02:09] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-webperf22 is down - https://phabricator.wikimedia.org/T436716 (10wmcs-alerts) 03NEW [21:04:05] 10Diffusion, 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): "Trying to access array offset on value of type null" in server pull logs for Phabricator Diffusion repository - https://phabricator.wikimedia.org/T406528#12277964 (10Aklapper) 05Stalled→03Declined Declining as we are workin... [21:06:17] 10Beta-Cluster-Infrastructure: Project deployment-prep instance deployment-webperf22 is down - https://phabricator.wikimedia.org/T436716#12277975 (10Southparkfan) 05Open→03Invalid Decommissioned in T436464 [21:07:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-webperf22 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [21:20:41] 06Release-Engineering-Team, 13Patch-For-Review: Configure logstash error checks for automated supervision of Pretrain deployments - https://phabricator.wikimedia.org/T435419#12278014 (10dancy) >>! In T435419#12267538, @bd808 wrote: > I think the work left here is putting the thresholds @dancy computed above in... [21:33:58] Southparkfan: I just landed the upgrade for the cloudvps provider that should fix the crashing bug when hiera settings contain an empty array. [21:36:05] Great work! [21:36:47] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Replace deployment-webperf2[12] (Bullseye deprecation) - https://phabricator.wikimedia.org/T436464#12278093 (10Southparkfan) Initial bootstrap of the Wikimedia Performance site fails: ` Notice: /Stage[main]/Httpd/Ht... [21:37:31] Southparkfan: if I had not written that bug into the provider I would not have needed to fix it, but thanks. :) [21:37:57] Haha :) [21:38:24] I'm on the verge of wrapping up the migration of the webperf hosts [22:02:14] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Replace deployment-webperf2[12] (Bullseye deprecation) - https://phabricator.wikimedia.org/T436464#12278135 (10Southparkfan) Almost done! After applying the `wmf-config/LabsServices.php` changes on `deployment-media... [22:11:52] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Replace deployment-webperf2[12] (Bullseye deprecation) - https://phabricator.wikimedia.org/T436464#12278158 (10Southparkfan) Seems to work fine with `https://webperf.svc.deployment-prep.eqiad1.wikimedia.cloud`. Afte...