[08:54:10] 10Continuous-Integration-Config: JJB: audit-resources job keeps running - https://phabricator.wikimedia.org/T332749 (10hashar) 05Openβ†’03Resolved a:03hashar Oh **thanks**! We don't necessarily delete Jenkins jobs since most of them are solely triggered by Zuul and once the jobs are unconfigured from Zuul th... [09:11:19] 10Gerrit, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10SRE-OnFire, 10SRE-Sprint-Week-Sustainability-March2023, and 3 others: gerrit1001 running out of space on / - https://phabricator.wikimedia.org/T323262 (10hashar) The incident report is at https://wikitech.wikimedia.org/wiki/Incidents/2022-11-17_G... [09:20:31] 10Gerrit, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10SRE-OnFire, 10SRE-Sprint-Week-Sustainability-March2023, and 3 others: gerrit1001 running out of space on / - https://phabricator.wikimedia.org/T323262 (10jcrespo) > I guess we can drop the SRE-OnFire tag? Hashar: alternatively, this could be clo... [09:56:07] 10Release-Engineering-Team, 10Fresh, 10MediaWiki-Core-Tests, 10Browser-Tests, and 2 others: Fresh problem when running Selenium tests - https://phabricator.wikimedia.org/T313899 (10zeljkofilipin) >>! In T313899#8714259, @kostajh wrote: > I see in https://github.com/webdriverio/webdriverio/issues/8279 some... [10:02:14] 10Release-Engineering-Team, 10Fresh, 10MediaWiki-Core-Tests, 10Browser-Tests, and 2 others: Fresh problem when running Selenium tests - https://phabricator.wikimedia.org/T313899 (10zeljkofilipin) >>! In T313899#8714259, @kostajh wrote: > But the system requirements for webdriver 8 say to use at least Node... [11:26:42] mediawiki-i18n-check-docker is intermittently failing with: 09:21:50 error: could not lock config file /mnt/home/jenkins-deploy/.gitconfig: File exists [12:40:33] maintenance-disconnect-full-disks build 475792 integration-agent-docker-1029 (/: 29%, /srv: 95%, /var/lib/docker: 53%): OFFLINE due to disk space [12:45:35] maintenance-disconnect-full-disks build 475793 integration-agent-docker-1029 (/: 29%, /srv: 68%, /var/lib/docker: 52%): RECOVERY disk space OK [13:11:14] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble: Quibble: Update nodeJS to at least v14.18.0 - https://phabricator.wikimedia.org/T332387 (10kostajh) p:05Triageβ†’03High Blocking CI and merges to Quibble. [14:07:55] 10Continuous-Integration-Infrastructure, 10MW-1.40-notes (1.40.0-wmf.20; 2023-01-23), 10User-zeljkofilipin: Deal with release of npm 7 - https://phabricator.wikimedia.org/T273785 (10hashar) 05Openβ†’03Stalled Marking as {nav Stalled} until `package-lock.json` files have been updated to lockfile version 2:... [14:34:08] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble: Quibble: Update nodeJS to at least v14.18.0 - https://phabricator.wikimedia.org/T332387 (10Jdforrester-WMF) >>! In T332387#8717798, @kostajh wrote: > Blocking CI and merges to Quibble. Your patch(es?) adding a requirement on ultra-bleed... [14:35:03] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble: Quibble: Update nodeJS to at least v14.18.0 - https://phabricator.wikimedia.org/T332387 (10Jdforrester-WMF) I propose to Decline this; the node14 support is EOL, and efforts should go into the node16 migration. [14:38:17] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble: Quibble: Update nodeJS to at least v14.18.0 - https://phabricator.wikimedia.org/T332387 (10kostajh) >>! In T332387#8718098, @Jdforrester-WMF wrote: >>>! In T332387#8717798, @kostajh wrote: >> Blocking CI and merges to Quibble. > > Your... [14:45:52] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble: Quibble: Update nodeJS to at least v14.18.0 - https://phabricator.wikimedia.org/T332387 (10kostajh) a:03kostajh [14:48:53] (03PS1) 10Kosta Harlan: dockerfiles: Update node14 image to 14.21.3 [integration/config] - 10https://gerrit.wikimedia.org/r/902104 (https://phabricator.wikimedia.org/T332387) [14:49:41] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble, 10Patch-For-Review: Quibble: Update nodeJS to at least v14.18.0 - https://phabricator.wikimedia.org/T332387 (10kostajh) >>! In T332387#8718101, @Jdforrester-WMF wrote: > I propose to Decline this; the node14 support is EOL, and efforts... [14:53:25] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble, 10Patch-For-Review: Quibble: Update nodeJS to at least v14.18.0 - https://phabricator.wikimedia.org/T332387 (10Jdforrester-WMF) >>! In T332387#8718105, @kostajh wrote: > In any case, reverting our patches would cause considerable diffi... [14:59:00] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Wikidata-Termbox, 10wdwb-tech, 10User-zeljkofilipin: Migrate termbox to lockfileVersion 2 in package-lock.json - https://phabricator.wikimedia.org/T327652 (10zeljkofilipin) [14:59:18] 10Continuous-Integration-Infrastructure, 10MW-1.40-notes (1.40.0-wmf.20; 2023-01-23): Deal with release of npm 7 - https://phabricator.wikimedia.org/T273785 (10zeljkofilipin) [15:03:19] 10Continuous-Integration-Infrastructure, 10Wikidata, 10Wikidata-Termbox, 10wdwb-tech, and 2 others: Migrate termbox to lockfileVersion 2 in package-lock.json - https://phabricator.wikimedia.org/T327652 (10Jdforrester-WMF) 05Openβ†’03Resolved a:03Michael Done in https://gerrit.wikimedia.org/r/c/wikibase... [15:03:21] 10Continuous-Integration-Infrastructure, 10MW-1.40-notes (1.40.0-wmf.20; 2023-01-23): Deal with release of npm 7 - https://phabricator.wikimedia.org/T273785 (10Jdforrester-WMF) [15:03:55] 10Continuous-Integration-Infrastructure, 10MW-1.40-notes (1.40.0-wmf.20; 2023-01-23): Deal with release of npm 7 - https://phabricator.wikimedia.org/T273785 (10Jdforrester-WMF) 05Stalledβ†’03Resolved a:03Jdforrester-WMF >>! In T273785#8718012, @hashar wrote: > Marking as {nav Stalled} until `package-lock.j... [15:07:41] 10GitLab (Infrastructure), 10serviceops-collab, 10Patch-For-Review: Define future design of GitLab backups - https://phabricator.wikimedia.org/T330172 (10Jclark-ctr) @LSobanski additional ssd drives added to gitlab servers in eqiad [15:20:50] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble, 10Patch-For-Review: Quibble: Update nodeJS to at least v14.18.0 - https://phabricator.wikimedia.org/T332387 (10hashar) NodeJS 14 is end of life in a month April 2023 and is still receiving updates (ex: [[[ https://github.com/nodejs/nod... [15:35:21] 10Gerrit, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10SRE-OnFire, 10SRE-Sprint-Week-Sustainability-March2023, and 3 others: gerrit1001 running out of space on / - https://phabricator.wikimedia.org/T323262 (10hashar) > Hashar: alternatively, this could be closed, as per title scope and the mentioned... [15:36:31] 10Project-Admins: Creation of a new project "All-and-every-Wikibooks" - https://phabricator.wikimedia.org/T330600 (10Aklapper) It sounds like the topic of printable versions is a small subset of any kinds of technical issues on all and every Wikibooks (which this tag would be about) [16:07:51] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10serviceops, 10Release Pipeline (Blubber): Buildkit erroring with "cannot reuse body, request must be retried" upon multi-platform push - https://phabricator.wikimedia.org/T322453 (10dduvall) I'm finally circling back to this, and I ran another test yesterday... [16:09:03] 10Release-Engineering-Team (Radar), 10serviceops-collab, 10serviceops-radar: give releng access to logs to debug buildkit-to-wmf-registry publishing - https://phabricator.wikimedia.org/T322579 (10dduvall) Friendly ping. :) [16:20:48] Project beta-update-databases-eqiad build #66006: 04FAILURE in 46 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/66006/ [16:30:41] hm, that's not supposed to run automatically.. looking [16:36:08] Project beta-update-databases-eqiad build #66007: 04STILL FAILING in 2 min 11 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/66007/ [16:46:02] Yippee, build fixed! [16:46:02] Project beta-update-databases-eqiad build #66008: 09FIXED in 8 min 48 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/66008/ [16:52:06] I created totally new VMs, did not deploy anyting to them.. yet directory under /srv/deployment/ looks like I have the same data already, git log looks the same.. but how is this possible:) [16:53:04] maybe a deploy happened and puppet already added them to scap::targets.. but also git log says it's at 2021 .. eh.... [16:53:21] I did not think I would rsync into /srv/deployment ever... hrmm [16:53:42] which host? [16:54:28] mutante: I think puppet does the first deploy based on the scap::target resource, and the scap::targets hiera key and similar only affect subsequent deploys [17:13:15] taavi: that's great, it looks like everything "just works" where I expected I need deployments [17:14:07] that is.. after I fixed the initial scap deploy done by puppet [17:14:25] by editing a file that contains "tin.eqiad.wmnet" as git_host [17:14:35] but other than that.. yea:) thx [17:29:51] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T330207 (10hashar) When doing a `scap sync-file` half an hour ago, scap complained about two kubernetes hosts not being... [17:41:02] 10Release-Engineering-Team (GitLab V: Event Horizon πŸŒ„), 10serviceops-collab: Isolate buildkitd build container namespaces on trusted runners - https://phabricator.wikimedia.org/T332804 (10dduvall) [18:02:26] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble, 10Patch-For-Review: Quibble: Update nodeJS to at least v14.18.0 - https://phabricator.wikimedia.org/T332387 (10kostajh) >>! In T332387#8718320, @hashar wrote: > NodeJS 14 is end of life in a month April 2023 and is still receiving upda... [18:03:26] 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Quibble, 10Patch-For-Review: Quibble: Update nodeJS to at least v14.18.0 - https://phabricator.wikimedia.org/T332387 (10Sgs) >>! In T332387#8718320, @hashar wrote: > Is it worse it? I can't tell the impacts of upgrading NodeJS {nav 14.17.5 > 14... [18:11:27] 10Release-Engineering-Team, 10Scap, 10Wikimedia-IEG-grant-review, 10serviceops-collab: scap deploy fails for iegreview - https://phabricator.wikimedia.org/T332623 (10Dzahn) [18:27:10] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.41.0-wmf.1 deployment blockers - https://phabricator.wikimedia.org/T330207 (10Clement_Goubert) >>! In T330207#8718757, @hashar wrote: > When doing a `scap sync-file` half an hour ago, sc... [18:56:27] (03CR) 10Herron: Zuul: [operations/grafana-grizzly] Add CI (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/901598 (https://phabricator.wikimedia.org/T331659) (owner: 10Ayounsi) [19:37:14] (03CR) 10Hashar: "I would go with a PipelineLib job which lets you define the go version, go install commands and add any extra Debian package you might nee" [integration/config] - 10https://gerrit.wikimedia.org/r/901598 (https://phabricator.wikimedia.org/T331659) (owner: 10Ayounsi) [20:24:49] 10Phabricator, 10translatewiki.net: Remove trailing whitespace from Phabricator message documentation - https://phabricator.wikimedia.org/T294754 (10brennen) a:05mmodellβ†’03None [20:25:06] 10Phabricator: Phabricator failed to generate thumbnails for some 800-900KB files - https://phabricator.wikimedia.org/T290959 (10brennen) a:05mmodellβ†’03None [20:25:33] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10DataΒ³: Phab feature request: Cycle time for a task entering a column to resolution, with support for wildcards - https://phabricator.wikimedia.org/T148805 (10brennen) a:05mmodellβ†’03None [20:25:43] 10Phabricator: move_project breaks the world when moving a subproject that already has subprojects. - https://phabricator.wikimedia.org/T242254 (10brennen) a:05mmodellβ†’03None [20:32:56] votewiki is the only beta wiki that's not in centralauth, right? [20:34:32] !log cat /srv/mediawiki/php-master/extensions/OATHAuth/sql/mysql/tables-generated.sql | sql centralauth --write # T242031 [20:34:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:34:36] T242031: Allow multiple different 2FA devices - https://phabricator.wikimedia.org/T242031 [20:37:59] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Developer Productivity, 10Upstream, 10User-brennen: "Call to phutil_nonempty_string() expected null or a string, got: int." when attempting to view Subversion repos - https://phabricator.wikimedia.org/T310936 (10brennen) a:03brennen Looks like we... [21:14:15] !log taavi@deployment-mwmaint02:~$ mwscript extensions/OATHAuth/maintenance/UpdateForMultipleDevicesSupport.php metawiki --force # T242031 [21:14:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:14:18] T242031: Allow multiple different 2FA devices - https://phabricator.wikimedia.org/T242031 [22:33:50] 10Phabricator Antivandalism Extension, 10Patch-For-Review, 10User-brennen: PHAVA: Create a CLI and Test suite for phabricator antivandalism - https://phabricator.wikimedia.org/T249741 (10brennen) a:05mmodellβ†’03None [22:34:05] 10Phabricator Antivandalism Extension, 10Release-Engineering-Team, 10Patch-For-Review, 10User-brennen: PHAVA: Create a CLI and Test suite for phabricator antivandalism - https://phabricator.wikimedia.org/T249741 (10brennen) [22:40:25] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Allow tools to use phabricator webhooks - https://phabricator.wikimedia.org/T321790 (10brennen) Just coming across this. I'm noting that we allow webhook traffic from GitLab to toolforge at present. I can't really think why this would be a different... [22:40:48] 10Phabricator, 10Release-Engineering-Team, 10User-brennen: Allow tools to use phabricator webhooks - https://phabricator.wikimedia.org/T321790 (10brennen) @Dzahn curious if you have any thoughts here...