[00:17:49] a gitlab admin/toucher would be helpful in "that security channel" [01:24:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:34:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:29:04] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:34:04] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [03:18:04] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [03:28:04] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [03:37:51] (03open) 10denisse: Add repos/sre/librenms [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/187 (https://phabricator.wikimedia.org/T436578) [04:09:04] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:14:04] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:26:21] (03open) 10phedenskog: Keep the percentile tab selected when clicking the legend [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/62 [04:28:35] (03merge) 10phedenskog: Keep the percentile tab selected when clicking the legend [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/62 [04:34:59] 10Continuous-Integration-Infrastructure, 10Castor, 06cloud-services-team, 10Cloud-VPS (Quota-requests), 06Test Platform (Arusha 30): Apply the 4x-iops Cinder volume type to the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T437127#12301434 (10komla) @fnegri, this is a fol... [04:38:30] 10Phabricator, 07JavaScript: "Other assignee" icon disappears briefly after loading a workboard - https://phabricator.wikimedia.org/T396878#12301453 (10Aklapper) p:05Triage→03Low [04:38:32] 10Phabricator, 06Collaboration-Services, 06Data-Persistence: phabricator long running queries - https://phabricator.wikimedia.org/T383068#12301455 (10Aklapper) p:05Triage→03Low [04:38:49] 10Phabricator, 06Release-Engineering-Team (Radar), 06Traffic: Phabricator videos fail in Firefox ("Range" request gets 503 from Varnish) - https://phabricator.wikimedia.org/T397661#12301457 (10Aklapper) p:05Triage→03Low [04:39:00] 10Phabricator, 07CSS: Gerrit patches are easy to miss in Phabricator history stream - https://phabricator.wikimedia.org/T416510#12301459 (10Aklapper) p:05Triage→03Low [04:40:41] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Result paging: (AphrontParameterQueryException) Expected a numeric scalar or null for %Ld conversion. Query: x.id IN (%Ld) - https://phabricator.wikimedia.org/T430016#12301461 (10Aklapper) 05Open→03Stalled a:03Aklapper [04:40:54] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Update to Phorge/Arcanist upstream 2026.xx - https://phabricator.wikimedia.org/T428547#12301466 (10Aklapper) [04:49:04] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:54:04] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [04:54:52] (03PS1) 10Mhurd: jjb: run npm cache verify at most once a day per cache [integration/config] - 10https://gerrit.wikimedia.org/r/1338086 (https://phabricator.wikimedia.org/T437377) [05:19:04] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [05:44:04] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:45:04] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [06:46:20] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Aruotes - https://phabricator.wikimedia.org/T437405 (10Aruotes) 03NEW [06:46:50] 06Project-Admins, 10Reconciliation: Reconciliation project - Subproject to manage reconciliation service - https://phabricator.wikimedia.org/T426541#12301580 (10Aklapper) @Spinster, @Eugene233: Could you please reply to the last comment? Thanks in advance! [06:55:04] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [07:02:42] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Aruotes - https://phabricator.wikimedia.org/T437405#12301618 (10Aklapper) Hi Aruotes, which project(s) in GitLab do you plan to work on? [07:45:12] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for Aruotes - https://phabricator.wikimedia.org/T437405#12301724 (10Aruotes) Yes Sir, I want to work over the task having id:*T424621 which is related to wikidata* whose main repo is on gitlab because of that i need... [07:52:36] (03PS2) 10Phedenskog: jjb: Overlap npm install with the gated browser tests [integration/config] - 10https://gerrit.wikimedia.org/r/1337439 (https://phabricator.wikimedia.org/T436376) [08:07:41] 06Project-Admins: Create project tag for standard thumbnail sizes bugs - https://phabricator.wikimedia.org/T431143#12301772 (10Aklapper) 05Stalled→03Declined No followup comments; declining for now. [08:28:11] 06Project-Admins, 10Reconciliation: Reconciliation project - Subproject to manage reconciliation service - https://phabricator.wikimedia.org/T426541#12301826 (10Eugene233) @Aklapper Thank you for the ping. @Pintoch I am ok with the sub-project tagging if it becomes useful at some point. So please feel free to... [08:51:33] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Phab Scap template vs actual value out of sync for numerous values - https://phabricator.wikimedia.org/T398460#12301882 (10Aklapper) * Set `storage.mysql-engine.max-size` DB value in config file in https://gitlab.wikimedia.org/repos/phabricator/de... [08:58:52] 10Phabricator (Search), 10Wikimedia-Phabricator-Extensions: People search for MediaWiki User lacks results (missing entries in phabricator_user.user_customfieldstringindex) - https://phabricator.wikimedia.org/T349439#12301902 (10Aklapper) →14Duplicate dup:03T365128 [08:58:53] 10Phabricator (Upstream), 06Release-Engineering-Team (Doing 😎), 07Upstream: Active user accounts don't show up in fulltext search when setting "Document Status" to "open" (and email verification is required) - https://phabricator.wikimedia.org/T365128#12301904 (10Aklapper) [09:37:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:39:45] (03CR) 10Phedenskog: [C:03+2] "This has been tested in Jenkin and no failures. Let's merge and let it run for a day so we can collect performance metrics." [integration/config] - 10https://gerrit.wikimedia.org/r/1337439 (https://phabricator.wikimedia.org/T436376) (owner: 10Phedenskog) [09:41:28] (03Merged) 10jenkins-bot: jjb: Overlap npm install with the gated browser tests [integration/config] - 10https://gerrit.wikimedia.org/r/1337439 (https://phabricator.wikimedia.org/T436376) (owner: 10Phedenskog) [09:42:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [09:42:16] !log jjb: update quibble-with-gated-extensions-selenium-php83 to run npm install ahead of the browser tests T436376 [09:42:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [09:42:18] T436376: Investigate the potential win running multiple npm installs at the same time - https://phabricator.wikimedia.org/T436376 [09:48:35] 10Continuous-Integration-Infrastructure, 10Castor, 06cloud-services-team, 10Cloud-VPS (Quota-requests), 06Test Platform (Arusha 30): Apply the 4x-iops Cinder volume type to the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T437127#12302059 (10fnegri) > Does retyping a liv... [09:51:29] 10Continuous-Integration-Infrastructure, 10Castor, 10Cloud-VPS (Quota-requests), 06Test Platform (Arusha 30): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12302096 (10fnegri) As noted in {T437127}, I pasted the wrong values in... [10:22:18] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [Abdullah1099] - https://phabricator.wikimedia.org/T437423 (10Abdullah1099) 03NEW [10:27:14] 10Phabricator, 06Release-Engineering-Team, 06Collaboration-Services: Move aphlict to k8s - https://phabricator.wikimedia.org/T436657#12302234 (10ABran-WMF) [10:39:25] (03PS1) 10Hslater: Zuul: [BlueSpiceProDistributionConnector] Add OAuth [integration/config] - 10https://gerrit.wikimedia.org/r/1338144 [10:53:59] (03PS1) 10Hslater: Zuul: Add StandardDialogs as transitive dependency [integration/config] - 10https://gerrit.wikimedia.org/r/1338146 [11:05:50] (03PS1) 10Hslater: Zuul: [IssueTrackerLinks] Add OOJSPlus [integration/config] - 10https://gerrit.wikimedia.org/r/1338148 [11:15:16] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for [Abdullah1099] - https://phabricator.wikimedia.org/T437423#12302406 (10Aklapper) 05Open→03Invalid > Developer account / GitLab username:Abdullah1099 No such account exists in GitLab. [11:15:47] 10Phabricator, 06Collaboration-Services: phab1006 Setup - https://phabricator.wikimedia.org/T437421#12302410 (10Aklapper) [11:50:40] (03PS3) 10Phedenskog: jjb: remove dupe --reporting-url from Quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1337963 (https://phabricator.wikimedia.org/T323750) (owner: 10Hashar) [11:50:40] (03PS4) 10Phedenskog: jjb: Run only database tests in the sqlite and postgres core jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1337437 (https://phabricator.wikimedia.org/T437134) [12:33:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:38:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:57:50] 10Continuous-Integration-Config, 06Release-Engineering-Team, 10BlueSpice: BlueSpice/HalloWelt force merging to branches and messing up mediawiki/extensions.git - https://phabricator.wikimedia.org/T431845#12302941 (10Osnard) I went through the repos and changed the settings. Unfortunately, I did not find all... [13:59:04] (03open) 10phedenskog: Add p75 and p95 to the feedback time per patch run tables for core [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/63 [14:01:29] (03merge) 10phedenskog: Add p75 and p95 to the feedback time per patch run tables for core [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/63 [14:07:40] 06Project-Admins, 10Produnto: Phabricator project for bug report, management and moderation of Produnto packages - https://phabricator.wikimedia.org/T437182#12303356 (10taavi) > So I propose a new Phabricator project "Produnto-Packages" for such tasks (which is similar to VPS-Projects and Tools). Alternatively... [14:45:06] (03open) 10dpogorzelski: chore: Add liftwing-studio repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/188 [14:49:47] (03merge) 10dancy: Add repos/sre/klaxon [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/186 (https://phabricator.wikimedia.org/T437001) (owner: 10denisse) [14:51:45] (03update) 10dpogorzelski: chore: Add liftwing-studio repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/188 [14:51:54] (03approved) 10dancy: Add repos/sre/librenms [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/187 (https://phabricator.wikimedia.org/T436578) (owner: 10denisse) [14:53:13] (03update) 10dpogorzelski: chore: Add liftwing-studio repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/188 [14:55:24] (03merge) 10dancy: chore: Add liftwing-studio repo [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/188 (owner: 10dpogorzelski) [15:08:58] 06Project-Admins, 10Reconciliation: Reconciliation project - Subproject to manage reconciliation service - https://phabricator.wikimedia.org/T426541#12303796 (10Spinster) Thanks for the notification! I removed myself as a member from the overarching project. Feel free to proceed with creating the subproject. [15:12:08] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance deployment-cp-text09 in project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:21:43] 06Project-Admins, 10Produnto: Phabricator project for bug report, management and moderation of Produnto packages - https://phabricator.wikimedia.org/T437182#12303841 (10Bugreporter) Currently Produnto is not yet deployed so the "community of package developers" does not yet exist. https://gitlab.wikimedia.org/... [16:09:30] !log deployed some anti-scraper mitigations on beta [16:09:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:11:03] 10Beta-Cluster-Infrastructure, 06MediaWiki-Engineering, 10Cloud-VPS (Debian Bullseye Deprecation): Beta Cluster: Switch to PHP 8.5, includes writing Puppet changes to support PHP 8.5 on baremetal - https://phabricator.wikimedia.org/T435393#12304100 (10Southparkfan) a:03Southparkfan [16:23:14] 10Phabricator Translations Extension: Make WMF translations extension pass `bin/i18n validate` - https://phabricator.wikimedia.org/T412651#12304155 (10Pppery) Next up: ` The locale `sk` defines a translation for the key `%d week(s)`, which varies on the plurality or gender of parameter 1, however that parameter... [16:43:20] 06Project-Admins, 10Produnto: Phabricator project for bug report, management and moderation of Produnto packages - https://phabricator.wikimedia.org/T437182#12304314 (10Aklapper) This sounds like a chicken and egg problem. :) If existing packages developers want to use Phabricator to organize work, they are ve... [16:50:15] 10Phabricator, 06Release-Engineering-Team (Priority Backlog 📥): Tasks created via certain Forms do not allow editing the task - https://phabricator.wikimedia.org/T403885#12304336 (10Aklapper) See also {T420132} going this direction. [16:58:49] 06Project-Admins, 10Produnto: Phabricator project for bug report, management and moderation of Produnto packages - https://phabricator.wikimedia.org/T437182#12304382 (10MSantos) This project is entering a pilot phase so we have a lot to learn and I really appreciate these suggestions. @Bugreporter in the next... [17:23:03] FIRING: PuppetAgentNoResources: No Puppet resources found on instance deployment-docker-mathoid02 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [17:23:09] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-mathoid02 on project deployment-prep - https://phabricator.wikimedia.org/T437489 (10wmcs-alerts) 03NEW [18:19:36] 10Phabricator, 06Collaboration-Services: phab1006 Setup - https://phabricator.wikimedia.org/T437421#12304713 (10Dzahn) Yea, let's put this in team meeting to chat about this host and what we do with it next. [18:30:07] 06Release-Engineering-Team, 06Infrastructure-Foundations: Test the Docker Registry's GC with Releng/MediaWiki images - https://phabricator.wikimedia.org/T437297#12304790 (10dancy) > 18 and 98 minutes. Unimpressive. I wonder what it's doing that takes so much time? How many images exist in each bucket? >... [18:50:26] 10Beta-Cluster-Infrastructure, 10Openstack-Magnum, 06tools-infrastructure-team: Rebuild the deployment-wikikube-v127 magnum cluster with the new magnum-cluster-api driver and k8s v1.34 - https://phabricator.wikimedia.org/T437500#12304892 (10Andrew) [19:11:21] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12304969 (10Scott_French) Checking in, it looks like this is still broken in the same way as yesterday T437069#12297979 (InRel... [19:32:19] 06Release-Engineering-Team (Radar), 06Infrastructure-Foundations, 06SRE: apt-get broken in docker-registry.wikimedia.org/bullseye:20260830 - https://phabricator.wikimedia.org/T437069#12305008 (10Scott_French) If I'm understanding the very end of T437069#12290846 correctly, it sounds like the one potential wo... [21:00:57] 06Release-Engineering-Team, 10Scap, 06ServiceOps, 07Epic, 13Patch-For-Review: Proposal: scap deploy-service - https://phabricator.wikimedia.org/T412941#12305349 (10dancy) Sample dry run output: ` (scap-venv) dancy@deploy1003:~/src/scap$ bin/run-dev-scap deploy-service shellbox --dry-run 20:59:26 Deployin... [21:15:01] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation): Replace deployment-docker-mathoid02 (Bullseye deprecation) - https://phabricator.wikimedia.org/T436465#12305393 (10Southparkfan) 05In progress→03Resolved [21:16:05] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-mathoid02 on project deployment-prep - https://phabricator.wikimedia.org/T437489#12305401 (10Southparkfan) Not sure why this is still alerting. Instance has been decommissioned earlier today: T436465. Instructed the Puppetserv... [21:17:27] 06Release-Engineering-Team, 13Patch-For-Review: Configure logstash error checks for automated supervision of Pretrain deployments - https://phabricator.wikimedia.org/T435419#12305410 (10dancy) Fresh figures: ` (scap-venv) dancy@deploy1003:~/src/scap$ bin/run-dev-scap analyze-logstash canaries --wiki testwiki 2... [21:18:33] 10Beta-Cluster-Infrastructure: No Puppet resources found on instance deployment-docker-mathoid02 on project deployment-prep - https://phabricator.wikimedia.org/T437489#12305414 (10Southparkfan) 05Open→03Resolved a:03Southparkfan Apparently, I didn't delete the instance properly. Fixed. [21:23:03] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance deployment-docker-mathoid02 on project deployment-prep - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:40:27] (03PS1) 10MusikAnimal: zuul: add ORES and Wikibase as phan dependency of PersonalDashboard [integration/config] - 10https://gerrit.wikimedia.org/r/1338299 (https://phabricator.wikimedia.org/T436570) [22:03:27] (03open) 10bwang: Draft: Codex 2.7.0 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/146 [23:41:12] (03update) 10denisse: Add repos/sre/librenms [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/187 (https://phabricator.wikimedia.org/T436578) [23:44:03] (03merge) 10dancy: Add repos/sre/librenms [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/187 (https://phabricator.wikimedia.org/T436578) (owner: 10denisse) [23:44:06] (03update) 10dancy: Add repos/sre/librenms [repos/releng/gitlab-trusted-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-trusted-runner/-/merge_requests/187 (https://phabricator.wikimedia.org/T436578) (owner: 10denisse) [23:49:03] FIRING: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [23:54:03] RESOLVED: InstanceDown: Project deployment-prep instance deployment-cache-text08 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown