[05:27:40] 10Phabricator, 10Release-Engineering-Team (Doing 😎), 13Patch-For-Review: Remove "Deadline" task subtype - https://phabricator.wikimedia.org/T335395#10616980 (10Aklapper) Bonus point: Deadlines would also get rendered in the user-assigned tasks list at https://phabricator.wikimedia.org/people/tasks/1/. Wasn't... [06:46:20] 10Gerrit, 10BlueSpice, 07affects-Miraheze: SimpleBlogPage breaks with BlueSpice function requirements - https://phabricator.wikimedia.org/T388359#10617019 (10RhinosF1) [06:59:32] 10Phabricator: Fix personal Herald rule H376 - https://phabricator.wikimedia.org/T388312#10617023 (10Marostegui) 05Open→03Resolved Disabled, it is not needed anymore! Thanks for pointing this out [07:47:35] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Make assignees or setters of a past Due Date on a non-closed task more aware of that task - https://phabricator.wikimedia.org/T293734#10617097 (10Aklapper) Hmm, could also be another notification icon in the top bar (please ignore the duplicated d... [08:09:32] 10Phabricator (Upstream), 07Upstream: Provide an easy way to link to a Phabricator task in a user-friendly way - https://phabricator.wikimedia.org/T388243#10617175 (10Peachey88) [09:04:19] 06Project-Admins: Consider archiving #WikiProject-tools - https://phabricator.wikimedia.org/T374765#10617388 (10Aklapper) 05Open→03Resolved a:03Aklapper No specific related codebase, thus indeed declining. (I've updated the few related tasks.) [10:33:39] 06Release-Engineering-Team, 10Data-Engineering-Roadmap, 06Data-Platform-SRE, 10Dumps-Generation, and 3 others: Orchestrate dumps v1 from an airflow instance - https://phabricator.wikimedia.org/T388378 (10brouberol) 03NEW [10:51:53] 10GitLab (Infrastructure), 06collaboration-services: SystemdUnitFailed (backup-restore.service on gitlab replicas) - https://phabricator.wikimedia.org/T388308#10617779 (10Jelto) 05Open→03Resolved The restore on the replica was moved to a later schedule with the change above. The last restore I trigger... [11:05:03] maintenance-disconnect-full-disks build 682824 integration-agent-docker-1043 (/: 25%, /srv: 96%, /var/lib/docker: 28%): OFFLINE due to disk space [11:10:02] maintenance-disconnect-full-disks build 682825 integration-agent-docker-1043 (/: 25%, /srv: 10%, /var/lib/docker: 25%): RECOVERY disk space OK [11:11:01] I think something’s wrong with the deployment calendar at the moment… it claims that this week, the UTC afternoon window is still at 15:00 UTC+1 (7:00 PDT), and only next week it moves to 14:00 UTC+1 (6:00 PDT) [11:11:21] shouldn’t it be earlier this week already, since daylight confusion time started yesterday? cc thcipriani [11:24:30] welcome to the confusion! [11:25:44] o/ [11:25:58] Lucas_WMDE: I have moved the morning backport earlier by an hour so that it stays at 9am CET [11:26:27] I don't remember whether I have moved the afternoon one, but we can move it as well [11:26:43] Reedy: I’m definitely confused :P [11:27:10] given most deployers (if not all) that can do the monring and afternoon windows are based in CET and subject to the same DST, I think it is fine [11:27:43] makes sense to me… so they should be moved (stick to 9/15 CET) the next two weeks as well? [11:28:55] yes [11:29:37] the script that generates the calendar was written in a rush a while ago and assumes it is run from a machine in PST [11:29:49] and the script has some oddities [11:30:08] which makes it a bit complicated to adjust [11:31:02] I should look again, maybe it is fine to pin them to CET [11:44:16] Time to make it even more confusing by mapping and only listing it in swatch internet time >.> [11:57:31] p858snake|cloud: if it has a timezone name (SIT?) and is in the tz database used by Scribunto/Lua (I am assuming that is the bit that needs to know about the timezones), then I guess it is possible [11:58:05] that was a great marketing coup [12:02:21] 06Release-Engineering-Team, 06serviceops: deploy1003 reports helmfileAdminPendingChanges - https://phabricator.wikimedia.org/T387900#10618187 (10JMeybohm) 05Open→03Resolved a:03JMeybohm I'll resolve this, since I just ran apply against all clusters [12:02:33] dduvall: When you've got some time... Mind chipping in on https://gitlab.wikimedia.org/repos/data-engineering/metrics-platform/-/merge_requests/81 please? (blubber config related changes to hopefully test on multiple PHP versions) [12:04:16] 10Phabricator, 06SRE, 07Documentation: Outdated documentation how to request LDAP group membership - https://phabricator.wikimedia.org/T388307#10618195 (10MoritzMuehlenhoff) p:05Triage→03Medium a:03MoritzMuehlenhoff [12:43:48] 10Release-Engineering-Team (Radar), 10Similarusers, 06Trust and Safety Product Team: Configure SimilarEditors in production with Similarusers credentials - https://phabricator.wikimedia.org/T308670#10618377 (10kostajh) [13:35:48] Hi, does anyone know why I get: [13:35:53] https://www.irccloud.com/pastebin/o4dcpZsJ/ [13:35:57] on https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-php74-selenium/75703/console [13:36:12] the error isn't super helpful in finding what is causing it. [13:36:35] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/MintyDocs/+/1125637 is fixing the removed class aliases [13:41:14] Definitely not just a flaky test? [13:42:05] Or, is it a newly excessively flaky test that maybe wants disabling/a task filing about if there's not one already? [13:43:12] it's not flaky as far as I can tell. Happens on another extension as well ApprovedRevs. [13:47:44] !log gerrit: removed leftover empty directory `/srv/gerrit/plugins/lfs`. Data have been migrated to `/srv/gerrit/plugins/lfs` as part of moving Gerrit data out of `/`. See T333143 [13:47:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [13:47:46] T333143: Move Gerrit data out of root partition - https://phabricator.wikimedia.org/T333143 [14:10:03] oh! [14:10:04] found it [14:19:44] 10Phabricator, 06SRE, 07Documentation: Outdated documentation how to request LDAP group membership - https://phabricator.wikimedia.org/T388307#10618984 (10MoritzMuehlenhoff) [14:32:08] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10ci-test-error (WMF-deployed Build Failure): CI jobs failing with various timeouts (March 2025) - https://phabricator.wikimedia.org/T388416 (10Daimona) 03NEW [14:34:24] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10SonarQube Bot, 07Code-Health, and 2 others: Improve speed of codehealth checks - https://phabricator.wikimedia.org/T351561#10619050 (10SDunlap) [14:37:47] 10Phabricator, 06SRE, 07Documentation: Outdated documentation how to request LDAP group membership - https://phabricator.wikimedia.org/T388307#10619073 (10MoritzMuehlenhoff) [14:39:23] 10Phabricator, 06SRE, 07Documentation: Outdated documentation how to request LDAP group membership - https://phabricator.wikimedia.org/T388307#10619078 (10MoritzMuehlenhoff) [14:39:44] 10Phabricator, 06SRE, 07Documentation: Outdated documentation how to request LDAP group membership - https://phabricator.wikimedia.org/T388307#10619079 (10MoritzMuehlenhoff) 05Open→03Resolved All updated. [15:15:00] (03PS1) 10Jforrester: Zuul: [mediawiki/services/servicelib-node] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1126080 (https://phabricator.wikimedia.org/T388424) [15:15:02] (03PS1) 10Jforrester: jjb: Drop now-unused servicelib-node jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1126081 (https://phabricator.wikimedia.org/T388424) [15:17:31] (03CR) 10Jforrester: [C:03+2] Zuul: [mediawiki/services/servicelib-node] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1126080 (https://phabricator.wikimedia.org/T388424) (owner: 10Jforrester) [15:18:08] (03PS2) 10Jforrester: jjb: Drop now-unused servicelib-node jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1126081 (https://phabricator.wikimedia.org/T388424) [15:19:27] (03Merged) 10jenkins-bot: Zuul: [mediawiki/services/servicelib-node] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1126080 (https://phabricator.wikimedia.org/T388424) (owner: 10Jforrester) [15:19:59] !log Zuul: [mediawiki/services/servicelib-node] Mark as archived, for T388424 [15:20:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:20:02] T388424: Archive the mediawiki/services/servicelib-node repo - https://phabricator.wikimedia.org/T388424 [15:20:53] (03CR) 10Jforrester: [C:03+2] jjb: Drop now-unused servicelib-node jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1126081 (https://phabricator.wikimedia.org/T388424) (owner: 10Jforrester) [15:22:48] (03Merged) 10jenkins-bot: jjb: Drop now-unused servicelib-node jobs [integration/config] - 10https://gerrit.wikimedia.org/r/1126081 (https://phabricator.wikimedia.org/T388424) (owner: 10Jforrester) [15:24:35] (03open) 10jforrester: Stop branching ActiveAbstract for production, no longer used [repos/releng/release] - 10https://gitlab.wikimedia.org/repos/releng/release/-/merge_requests/152 (https://phabricator.wikimedia.org/T382069) [15:26:28] (03PS1) 10Jforrester: Zuul: [mediawiki/extensions/ActiveAbstract] Mark as archived [integration/config] - 10https://gerrit.wikimedia.org/r/1126085 (https://phabricator.wikimedia.org/T382069) [15:37:04] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: SystemdUnitFailed (backup-restore.service on gitlab replicas) - https://phabricator.wikimedia.org/T388308#10619477 (10Jelto) [15:37:17] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: SystemdUnitFailed (backup-restore.service on gitlab replicas) - https://phabricator.wikimedia.org/T388308#10619479 (10Jelto) 05Resolved→03Open [15:38:22] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: SystemdUnitFailed (backup-restore.service on gitlab replicas) - https://phabricator.wikimedia.org/T388308#10619487 (10Jelto) This happened again because the sync job is missing a dependency to the `partial-backup.service`. The change ab... [16:19:02] 10Diffusion, 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10Wikimedia-GitHub: Mirroring of toolforge/spacemedia repo from Diffusion to GitHub is broken - https://phabricator.wikimedia.org/T385126#10619667 (10bd808) * The https://github.com/settings/tokens screen when accessed as the http... [16:26:54] 06Release-Engineering-Team, 06Security-Team, 07SecTeam-Processed, 07Security: Develop a means for preventing developers from accidentally pushing embargoed security patches to gerrit - https://phabricator.wikimedia.org/T388247#10619724 (10sbassett) [16:30:02] maintenance-disconnect-full-disks build 682889 integration-agent-docker-1053 (/: 25%, /srv: 95%, /var/lib/docker: 31%): OFFLINE due to disk space [16:35:02] maintenance-disconnect-full-disks build 682890 integration-agent-docker-1053 (/: 25%, /srv: 48%, /var/lib/docker: 31%): RECOVERY disk space OK [16:35:30] (03open) 10dancy: Release 4.140.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/677 [16:36:46] (03update) 10dancy: kubernetes: support non-deploy helmfile releases [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/676 (https://phabricator.wikimedia.org/T387917) (owner: 10swfrench) [16:37:30] (03close) 10dancy: Release 4.140.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/677 [16:39:41] (03merge) 10dancy: kubernetes: support non-deploy helmfile releases [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/676 (https://phabricator.wikimedia.org/T387917) (owner: 10swfrench) [16:41:35] (03PS1) 10Scott French: Docker: [quibble-bullseye-php81] Use PCRE2 backport from component/php81 [integration/config] - 10https://gerrit.wikimedia.org/r/1125557 (https://phabricator.wikimedia.org/T386006) [16:41:35] (03CR) 10Scott French: "Thanks for taking a look!" [integration/config] - 10https://gerrit.wikimedia.org/r/1125557 (https://phabricator.wikimedia.org/T386006) (owner: 10Scott French) [16:42:58] (03PS2) 10Scott French: Docker: [php81] Use PCRE2 backport from component/php81 [integration/config] - 10https://gerrit.wikimedia.org/r/1125558 (https://phabricator.wikimedia.org/T386006) [16:42:59] (03CR) 10Scott French: "If you could review these as well, that would be greatly appreciated. Thanks!" [integration/config] - 10https://gerrit.wikimedia.org/r/1125558 (https://phabricator.wikimedia.org/T386006) (owner: 10Scott French) [16:43:18] (03open) 10dancy: Release 4.140.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/678 [16:45:30] (03merge) 10dancy: Release 4.140.0 [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/678 [17:20:12] 10Release-Engineering-Team (Radar), 10Scap, 06serviceops: Set scap minimum python version to 3.7 - https://phabricator.wikimedia.org/T302086#10620079 (10dancy) 05Stalled→03Resolved a:03dancy Scap has been 3.7+ since Wed Feb 22 13:20:10 2023. [17:26:42] 10GitLab, 06collaboration-services, 10Stewards-Onboarding-Tool: Update profile::stewards::gitlab_api_token to match new token - https://phabricator.wikimedia.org/T388212#10620094 (10LSobanski) p:05Triage→03Medium [17:27:23] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit1003) - https://phabricator.wikimedia.org/T388235#10620100 (10LSobanski) a:03Jelto [17:27:34] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit1003) - https://phabricator.wikimedia.org/T388235#10620103 (10LSobanski) p:05Triage→03High [17:29:00] 10Diffusion, 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10Wikimedia-GitHub: Mirroring of toolforge/spacemedia repo from Diffusion to GitHub is broken - https://phabricator.wikimedia.org/T385126#10620124 (10thcipriani) I don't think the problem is the token, rather whatever this: > err... [17:46:09] 06Release-Engineering-Team, 06serviceops: deploy1003 reports helmfileAdminPendingChanges - https://phabricator.wikimedia.org/T387900#10620207 (10jijiki) This might be related to T381417? [17:57:30] 10Gerrit, 06Release-Engineering-Team, 06collaboration-services, 10Sustainability (Incident Followup): Move Gerrit data out of root partition - https://phabricator.wikimedia.org/T333143#10620267 (10Dzahn) >>! In T333143#10618795, @Stashbot wrote: > gerrit: removed leftover empty directory `/srv/gerrit/plugi... [18:24:03] (03approved) 10dduvall: Expose grafana via Ingress [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/432 (https://phabricator.wikimedia.org/T363919) (owner: 10dancy) [18:24:33] (03merge) 10dancy: Expose grafana via Ingress [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/432 (https://phabricator.wikimedia.org/T363919) [18:26:38] 10GitLab, 06collaboration-services, 10Stewards-Onboarding-Tool: Update profile::stewards::gitlab_api_token to match new token - https://phabricator.wikimedia.org/T388212#10620361 (10Dzahn) 05Open→03Resolved I have updated the token in the private puppet repo and ran puppet on stewards1001. It has bee... [18:53:59] (03open) 10dancy: Expose grafana via Ingress (followup) [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/434 (https://phabricator.wikimedia.org/T363919) [18:54:01] (03update) 10dancy: Expose grafana via Ingress (followup) [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/434 (https://phabricator.wikimedia.org/T363919) [18:55:10] (03merge) 10dancy: Expose grafana via Ingress (followup) [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/434 (https://phabricator.wikimedia.org/T363919) [19:39:16] (03open) 10dancy: grafana.yaml.tftpl: Configure the default dashboard [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/435 (https://phabricator.wikimedia.org/T363919) [19:39:18] (03update) 10dancy: grafana.yaml.tftpl: Configure the default dashboard [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/435 (https://phabricator.wikimedia.org/T363919) [19:40:18] (03merge) 10dancy: grafana.yaml.tftpl: Configure the default dashboard [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/435 (https://phabricator.wikimedia.org/T363919) [19:41:00] Anyone having problems getting gitlab pipelines to run? Or just me? :P [19:42:59] It's definitely completely not broken... [19:45:13] https://gitlab.wikimedia.org/repos/data-engineering/metrics-platform/-/merge_requests/81/pipelines [19:47:08] (03open) 10dancy: grafana.yaml.tftpl: Configure the default dashboard (fix) [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/436 (https://phabricator.wikimedia.org/T363919) [19:47:10] (03update) 10dancy: grafana.yaml.tftpl: Configure the default dashboard (fix) [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/436 (https://phabricator.wikimedia.org/T363919) [19:47:24] 10GitLab (Upstream pit of despair 🕳️): GraphQL error: Field 'troubleshootJobWithAi' doesn't exist on type 'Pipeline' - https://phabricator.wikimedia.org/T388459 (10Reedy) 03NEW [19:47:54] 10GitLab, 07Upstream: GraphQL error: Field 'troubleshootJobWithAi' doesn't exist on type 'Pipeline' - https://phabricator.wikimedia.org/T388459#10620836 (10Reedy) [19:48:07] (03merge) 10dancy: grafana.yaml.tftpl: Configure the default dashboard (fix) [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/436 (https://phabricator.wikimedia.org/T363919) [19:48:41] 10GitLab (Upstream pit of despair 🕳️): GraphQL error: Field 'troubleshootJobWithAi' doesn't exist on type 'Pipeline' - https://phabricator.wikimedia.org/T388459#10620842 (10Reedy) [19:49:18] 10GitLab (Upstream pit of despair 🕳️): GraphQL error: Field 'troubleshootJobWithAi' doesn't exist on type 'Pipeline' - https://phabricator.wikimedia.org/T388459#10620851 (10Reedy) [20:03:15] 10GitLab (Upstream pit of despair 🕳️), 07Upstream: GraphQL error: Field 'troubleshootJobWithAi' doesn't exist on type 'Pipeline' - https://phabricator.wikimedia.org/T388459#10620914 (10bd808) [20:30:07] (03update) 10dancy: Revert: grafana.yaml.tftpl: Configure the default dashboard [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/437 (https://phabricator.wikimedia.org/T363919) [20:30:08] (03open) 10dancy: Revert: grafana.yaml.tftpl: Configure the default dashboard [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/437 (https://phabricator.wikimedia.org/T363919) [20:30:53] (03merge) 10dancy: Revert: grafana.yaml.tftpl: Configure the default dashboard [repos/releng/gitlab-cloud-runner] - 10https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/merge_requests/437 (https://phabricator.wikimedia.org/T363919) [21:33:16] 10GitLab (Upstream pit of despair 🕳️), 07Upstream: GraphQL error: Field 'troubleshootJobWithAi' doesn't exist on type 'Pipeline' - https://phabricator.wikimedia.org/T388459#10621190 (10Reedy) Expand the "failed jobs" part.. Makes the banner appear {F58760279 size=full} [21:50:05] 10Release-Engineering-Team (Priority Backlog 📥), 05Release, 05Train Deployments: 1.44.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T386215#10621296 (10Jdlrobson-WMF) [22:45:26] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Replace or remove deployment-echostore02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361383#10621478 (10bd808) 05Resolved→03Open This instance apparently never got a replacement. The `echostore.svc.deplo... [23:19:58] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Buster Deprecation): Replace or remove deployment-echostore02.deployment-prep.eqiad1.wikimedia.cloud - https://phabricator.wikimedia.org/T361383#10621529 (10bd808) @thcipriani dug around and found some config from older instances that might get somebody starte...