[00:02:17] PROBLEM - SSH on gerrit2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [01:03:13] RECOVERY - SSH on gerrit2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [03:02:22] Project mwcore-phpunit-coverage-master build #1700: 04STILL FAILING in 2 min 21 sec: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1700/ [03:17:54] 10Project-Admins, 10Wikimedia-Small-Projects-User-Group: Create project tag for WikiSP-Projects and WikiSP-Millestone - https://phabricator.wikimedia.org/T293170 (10Galahad) [03:18:17] 10Project-Admins, 10Wikimedia-Small-Projects-User-Group: Create project tags for WikiSP-Projects and WikiSP-Millestone - https://phabricator.wikimedia.org/T293170 (10Galahad) [04:44:06] (03CR) 10Jforrester: [C: 03+2] "Deployed. Seems to work (or at least, the ones that were already working aren't broken; sorry, fresnel)." [integration/config] - 10https://gerrit.wikimedia.org/r/730329 (https://phabricator.wikimedia.org/T293105) (owner: 10Jforrester) [04:45:57] (03Merged) 10jenkins-bot: jjb: Update node{12,14} jobs to use images with npm 7.21.0 [integration/config] - 10https://gerrit.wikimedia.org/r/730329 (https://phabricator.wikimedia.org/T293105) (owner: 10Jforrester) [04:46:18] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Upgrade the pinned version of npm in our CI images from 7.5.2 to at least 7.14 - https://phabricator.wikimedia.org/T293105 (10Jdforrester-WMF) 05Openβ†’03Resolved a:03Jdforrester-WMF [07:14:53] 10Phabricator: Phabricator should not embed videos from Wikimedia Commons when the file is deleted - https://phabricator.wikimedia.org/T293139 (10Aklapper) Thanks, confirming. (I nowadays personally wonder why T116515 got resolved, apart from creating additional followup maintenance costs...) [08:24:40] 10Phabricator: Phabricator should not embed videos from Wikimedia Commons when the file is deleted - https://phabricator.wikimedia.org/T293139 (10Lucas_Werkmeister_WMDE) [08:52:00] (03PS1) 10Hashar: deployment-calendar: run tests via tox for CI [tools/release] - 10https://gerrit.wikimedia.org/r/730428 [08:59:31] (03CR) 10jerkins-bot: [V: 04-1] deployment-calendar: run tests via tox for CI [tools/release] - 10https://gerrit.wikimedia.org/r/730428 (owner: 10Hashar) [09:16:50] (03CR) 10Hashar: "Apparently parser.find_train() returns a datetime object which is not timezone aware, it thus depends on the local timezone and the return" [tools/release] - 10https://gerrit.wikimedia.org/r/730428 (owner: 10Hashar) [09:22:50] 10Deployments, 10Release-Engineering-Team: During holiday: add the automatic train branch as a 'window' in the deployments calendar so people can be aware of it happening - https://phabricator.wikimedia.org/T293101 (10hashar) I got a patch to have CI to run tests https://gerrit.wikimedia.org/r/c/mediawiki/tool... [09:55:41] 10Release-Engineering-Team (Priority Backlog πŸ”₯), 10MediaWiki-Docker, 10dev-images, 10mwcli, 10User-brennen: Create a PHP 8 dev image for use with mediawiki-docker-dev - https://phabricator.wikimedia.org/T283426 (10jeena) [09:58:58] 10Release-Engineering-Team (Priority Backlog πŸ”₯), 10dev-images, 10mwcli: Add php-luasandbox to dev-images used by mwcli - https://phabricator.wikimedia.org/T286678 (10jeena) [10:14:58] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: Scap rollback command - https://phabricator.wikimedia.org/T287045 (10jeena) Need to rethink this since the image tag won't be updated via the deployment-charts repo. [10:21:01] 10Release-Engineering-Team (Priority Backlog πŸ”₯), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: Scap backport change_url command - https://phabricator.wikimedia.org/T287042 (10jeena) [10:21:43] 10Release-Engineering-Team (Priority Backlog πŸ”₯), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: Scap backport change_url command - https://phabricator.wikimedia.org/T287042 (10jeena) [10:23:12] 10Release-Engineering-Team (Priority Backlog πŸ”₯), 10MW-on-K8s, 10Release Pipeline, 10User-brennen: Scap backport change_url command - https://phabricator.wikimedia.org/T287042 (10jeena) [11:10:55] 10Deployments, 10Sustainability (Incident Followup): Local private files on deployment host should be backed up somewhere - https://phabricator.wikimedia.org/T69818 (10jcrespo) a:05jcrespoβ†’03Krinkle Waiting for an answer, even if it is "no". :-) [11:11:10] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10ci-test-error (WMF-deployed Build Failure): TAR_ENTRY_ERROR ENOSPC: no space left on device - https://phabricator.wikimedia.org/T292729 (10kostajh) Seen again on https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php72-docker/1... [11:11:54] hashar: if you're around, could you look at this? it's blocking the backports (assuming the problem keeps reoccurring) [11:12:22] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10ci-test-error (WMF-deployed Build Failure): TAR_ENTRY_ERROR ENOSPC: no space left on device - https://phabricator.wikimedia.org/T292729 (10kostajh) 05Resolvedβ†’03Open [11:34:40] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for scherukuwada - https://phabricator.wikimedia.org/T293087 (10thcipriani) p:05Triageβ†’03Medium Hi @SCherukuwada! I added you to the training on 2021-10-14 at 11UTC. * * * @ArielGlenn will you be able to lead ^ training... [11:36:36] kostajh: yup [11:38:58] https://grafana-labs.wikimedia.org/d/000000059/cloud-vps-project-board?viewPanel=18&orgId=1&var-project=integration&var-server=integration-agent-docker-1011&from=now-3h&to=now [11:39:06] per partition disk usage for the agent [11:50:03] 10Beta-Cluster-Infrastructure, 10AbuseFilter, 10affects-translatewiki.net: AbuseFilter causes Error 1091: Can't DROP INDEX `afl_filter_timestamp`; check that it exists - https://phabricator.wikimedia.org/T291725 (10Nikerabbit) 05Resolvedβ†’03Open We hit the same issue when updating translatewiki.net from 1... [11:50:40] 10Beta-Cluster-Infrastructure, 10AbuseFilter, 10MW-1.37-release, 10affects-translatewiki.net: AbuseFilter causes Error 1091: Can't DROP INDEX `afl_filter_timestamp`; check that it exists - https://phabricator.wikimedia.org/T291725 (10Majavah) [11:51:52] 10Deployments, 10Release-Engineering-Team: During holiday: add the automatic train branch as a 'window' in the deployments calendar so people can be aware of it happening - https://phabricator.wikimedia.org/T293101 (10thcipriani) >>! In T293101#7423018, @hashar wrote: > I got a patch to have CI to run tests ht... [11:54:01] kostajh: looks like we should prune every few hours [11:55:32] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10ci-test-error (WMF-deployed Build Failure): TAR_ENTRY_ERROR ENOSPC: no space left on device - https://phabricator.wikimedia.org/T292729 (10hashar) The build ran on `integration-agent-docker-1011`, I checked the disk usage via https://graf... [11:56:31] !log integration: sudo cumin --force 'name:docker' 'docker system prune --force --volumes' [11:56:32] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:09:55] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for scherukuwada - https://phabricator.wikimedia.org/T293087 (10ArielGlenn) I should be there. But I hope I will not be the only deployer. As long as there's a buddy, I'm in! Ariel [12:12:49] apergos: ^ I can probably be around for the training tomorrow [12:13:01] sweet. we're a team! [12:13:29] I was out of action last week, migraines, didn't even try to log in... but that should be that for another month, that's the typical frequency [12:40:46] 10Project-Admins: Create project tag for Web2Cit - https://phabricator.wikimedia.org/T292323 (10diegodlh) Hi, all! Any news about this? I'd appreciate it if someone could take care of this, so we can start tracking pending task in the project. Also, I'd thank you if someone could reply to my questions above. Th... [12:41:46] (03PS1) 10Zabe: Add A2093064 to the CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/730517 [12:44:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10ci-test-error (WMF-deployed Build Failure): TAR_ENTRY_ERROR ENOSPC: no space left on device - https://phabricator.wikimedia.org/T292729 (10kostajh) > I guess we should run docker system prune --force --volumes regularly, possibly with --a... [13:06:24] 10Phabricator: Growth team blog - https://phabricator.wikimedia.org/T293240 (10kostajh) [13:24:42] 10Phabricator: Create Growth team blog on Phame - https://phabricator.wikimedia.org/T293240 (10Aklapper) [13:25:18] 10Phabricator: Create Growth team blog on Phame - https://phabricator.wikimedia.org/T293240 (10Aklapper) (I myself am clueless how to set up as I cannot find [blog related docs](https://www.mediawiki.org/wiki/Phabricator/Help), so I'll leave this to appreciated coworkers.) [14:05:29] (03PS3) 10Isabelle Hurbain-Palatin: Translate contains VisualEditor components now [integration/config] - 10https://gerrit.wikimedia.org/r/730207 (owner: 10C. Scott Ananian) [14:05:59] 10Beta-Cluster-Infrastructure: upload.wikimedia.beta.wmflabs.org certificate expired - https://phabricator.wikimedia.org/T293070 (10Majavah) [14:10:17] 10Beta-Cluster-Infrastructure: upload.wikimedia.beta.wmflabs.org certificate expired - https://phabricator.wikimedia.org/T293070 (10AlexisJazz) November 13, 2020: T267858 January 12, 2021: T271808 @Vgutierrez @Krenair [14:37:00] 10Phabricator (Upstream), 10Upstream: Assign / claim tasks to yourself with one click - https://phabricator.wikimedia.org/T75705 (10Osamaahmed17) a:05mmodellβ†’03Osamaahmed17 [14:38:33] 10Phabricator (Upstream), 10Upstream: Assign / claim tasks to yourself with one click - https://phabricator.wikimedia.org/T75705 (10Osamaahmed17) a:05Osamaahmed17β†’03None [15:01:48] Project mwcore-phpunit-coverage-master build #1701: 04STILL FAILING in 1 min 45 sec: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1701/ [15:03:19] 10Continuous-Integration-Config: How to set up CI to have "PHP Parsoid" enabled? - https://phabricator.wikimedia.org/T267511 (10Osnard) @hashar Do you have any idea maybe? [15:05:37] (03CR) 10DannyS712: "This change is ready for review." [design/codex] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/730503 (owner: 10DannyS712) [15:10:21] (03PS1) 10Hashar: Support multiple workers in PHP 7.4+ web server [integration/quibble] - 10https://gerrit.wikimedia.org/r/730547 (https://phabricator.wikimedia.org/T259456) [15:10:48] (03CR) 10Hashar: [C: 04-1] "WIP :)" [integration/quibble] - 10https://gerrit.wikimedia.org/r/730547 (https://phabricator.wikimedia.org/T259456) (owner: 10Hashar) [15:11:18] 10Continuous-Integration-Config: How to set up CI to have "PHP Parsoid" enabled? - https://phabricator.wikimedia.org/T267511 (10ihurbain) I learnt about the existence of zuul/parameter_functions.py yesterday, and my understanding is that this is what you'd want to edit (I have https://gerrit.wikimedia.org/r/c/i... [15:14:26] (03CR) 10jerkins-bot: [V: 04-1] Support multiple workers in PHP 7.4+ web server [integration/quibble] - 10https://gerrit.wikimedia.org/r/730547 (https://phabricator.wikimedia.org/T259456) (owner: 10Hashar) [15:16:04] 10Continuous-Integration-Config: How to set up CI to have "PHP Parsoid" enabled? - https://phabricator.wikimedia.org/T267511 (10Osnard) Thank you very much for your response. But unfortunately, we already have VisualEditor declared as a dependency :( https://github.com/wikimedia/integration-config/blob/995a0527... [15:18:21] 10Continuous-Integration-Config: How to set up CI to have "PHP Parsoid" enabled? - https://phabricator.wikimedia.org/T267511 (10ihurbain) Yup, but VE doesn't depend on parsoid, so it wouldn't help, I think. adding "parsoid" as a dependency might help. [15:21:27] (03PS1) 10Robert Vogel: Add `parsoid` as a dependency to BlueSpiceVisualEditorConnector [integration/config] - 10https://gerrit.wikimedia.org/r/730548 [15:21:57] 10Continuous-Integration-Config: How to set up CI to have "PHP Parsoid" enabled? - https://phabricator.wikimedia.org/T267511 (10Osnard) Good idea. I will try [1]. Thanks. [1] https://gerrit.wikimedia.org/r/c/integration/config/+/730548 [15:53:11] 10Project-Admins: (Re) Create project tag for Vue.js - https://phabricator.wikimedia.org/T293272 (10LZaman) [16:09:21] 10Phabricator: Add a Herald rule for User-MediaJS - https://phabricator.wikimedia.org/T286077 (10MediaJS) If I'm subscribed to a task, add #user-mediajs [16:16:52] Hi brennen ! Have you had an opportunity to work on the docker image auto-publisher for dev-images? [16:23:42] 10Phabricator: Add a Herald rule for User-MediaJS - https://phabricator.wikimedia.org/T286077 (10Aklapper) That is very noisy for everyone else, and I'd prefer not doing that without a clear reason / use case. [16:23:58] PROBLEM - SSH on contint2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [16:34:52] 10Project-Admins: (Re) Create project tag for Vue.js - https://phabricator.wikimedia.org/T293272 (10Aklapper) Would this be a milestone of https://phabricator.wikimedia.org/project/subprojects/4662/ like its existing "Vue.js something" milestones, or a separate top-level project? Could someone please provide a d... [16:43:46] 10Release-Engineering-Team, 10FR-Docker: Publish 2 fundraising dev-images to docker repository - https://phabricator.wikimedia.org/T293277 (10Ejegg) [16:43:47] ejegg: i have not. however if you need the current state of dev-images published, i'm happy to do so right now [16:44:00] thanks! I just now made that phab ^^ [16:44:42] right on, doing. will test with a local build first. [16:44:47] cool cool [16:45:22] 10Release-Engineering-Team (Doing), 10FR-Docker, 10dev-images: Publish 2 fundraising dev-images to docker repository - https://phabricator.wikimedia.org/T293277 (10brennen) 05Openβ†’03In progress a:03brennen [16:48:28] !log Updating dev-images docker-pkg files on primary contint for T293277 [16:48:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:48:31] T293277: Publish 2 fundraising dev-images to docker repository - https://phabricator.wikimedia.org/T293277 [16:50:34] 10Release-Engineering-Team, 10dev-images, 10User-brennen: Check for empty continuation lines in dev-images - https://phabricator.wikimedia.org/T293278 (10brennen) [16:57:29] 10Phabricator, 10Release-Engineering-Team (Priority Backlog πŸ”₯), 10DataΒ³: Phab feature request: Cycle time for a task entering a column to resolution, with support for wildcards - https://phabricator.wikimedia.org/T148805 (10mmodell) [16:58:04] 10Release-Engineering-Team (Doing), 10FR-Docker, 10dev-images: Publish 2 fundraising dev-images to docker repository - https://phabricator.wikimedia.org/T293277 (10brennen) 05In progressβ†’03Resolved ` [contint.wikimedia.org] out: == Step 2: publishing == [contint.wikimedia.org] out: Successfully published... [16:58:07] thanks brennen ! [16:59:22] you bet - sorry that's not automated yet. One Of These Daysβ„’. [17:01:53] 10Release-Engineering-Team (Priority Backlog πŸ”₯), 10mwcli, 10GitLab (CI & Job Runners), 10User-brennen: Increase maximum artifacts size for gitlab CI - https://phabricator.wikimedia.org/T292372 (10brennen) [17:05:29] (03CR) 10Catrope: [C: 03+2] Edit Repo Config - rebase if necessary [design/codex] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/730503 (owner: 10DannyS712) [17:05:37] (03CR) 10Catrope: [V: 03+2 C: 03+2] Edit Repo Config - rebase if necessary [design/codex] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/730503 (owner: 10DannyS712) [17:07:16] (03PS2) 10Hashar: Support multiple workers in PHP 7.4+ web server [integration/quibble] - 10https://gerrit.wikimedia.org/r/730547 (https://phabricator.wikimedia.org/T259456) [17:07:30] (03PS3) 10Hashar: Support multiple workers in PHP 7.4+ web server [integration/quibble] - 10https://gerrit.wikimedia.org/r/730547 (https://phabricator.wikimedia.org/T259456) [17:09:01] (03CR) 10Hashar: [C: 04-1] "Sorry for the spam it is a bit messy. I still have an issue when shutting down some backend cause the threading shutdown function fails t" [integration/quibble] - 10https://gerrit.wikimedia.org/r/730547 (https://phabricator.wikimedia.org/T259456) (owner: 10Hashar) [17:11:14] (03CR) 10jerkins-bot: [V: 04-1] Support multiple workers in PHP 7.4+ web server [integration/quibble] - 10https://gerrit.wikimedia.org/r/730547 (https://phabricator.wikimedia.org/T259456) (owner: 10Hashar) [17:21:37] 10Deployments, 10Sustainability (Incident Followup): Local private files on deployment host should be backed up somewhere - https://phabricator.wikimedia.org/T69818 (10Krinkle) @jcrespo OK, could you restore `/srv/mediawiki-staging/private/PrivateSettings.php` from ~4 weeks ago to my home directory? [17:26:51] (03PS4) 10Hashar: Support multiple workers in PHP 7.4+ web server [integration/quibble] - 10https://gerrit.wikimedia.org/r/730547 (https://phabricator.wikimedia.org/T259456) [17:27:12] (03CR) 10Hashar: "python 3.5 back compatibility ;)" [integration/quibble] - 10https://gerrit.wikimedia.org/r/730547 (https://phabricator.wikimedia.org/T259456) (owner: 10Hashar) [17:30:08] hashar: are there jobs where we want to keep using php web server? [17:32:15] (03CR) 10jerkins-bot: [V: 04-1] Support multiple workers in PHP 7.4+ web server [integration/quibble] - 10https://gerrit.wikimedia.org/r/730547 (https://phabricator.wikimedia.org/T259456) (owner: 10Hashar) [17:46:28] 10Project-Admins: (Re) Create project tag for Vue.js - https://phabricator.wikimedia.org/T293272 (10LZaman) Hi @Aklapper No, unfortunately, a milestone would not work because there are other teams who will do Vue migration work, not just Design Systems Team. I did indicate a project type; is component not accur... [17:48:32] 10Project-Admins: (Re) Create project tag for Vue.js - https://phabricator.wikimedia.org/T293272 (10LZaman) [18:10:58] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10ci-test-error (WMF-deployed Build Failure): TAR_ENTRY_ERROR ENOSPC: no space left on device - https://phabricator.wikimedia.org/T292729 (10thcipriani) >>! In T292729#7423777, @hashar wrote: > I guess we should run `docker system prune --f... [18:25:48] RECOVERY - SSH on contint2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [19:01:07] 10Project-Admins: (Re) Create project tag for Vue.js - https://phabricator.wikimedia.org/T293272 (10Jdforrester-WMF) Yeah, converting the general "vue.js" tag into a team project means that there's a hole left for cross-cutting things. That said, I'm not sure how much use the general tags like #javascript or #be... [19:56:53] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T281168 (10dancy) [20:06:19] 10Project-Admins: Archive the User-Zabe project - https://phabricator.wikimedia.org/T293304 (10Zabe) [20:40:58] (03CR) 10Hashar: "The packaging module is an indirect requirements for tests, but is not one for Quibble itself. I will try to find an alternative since I d" [integration/quibble] - 10https://gerrit.wikimedia.org/r/730547 (https://phabricator.wikimedia.org/T259456) (owner: 10Hashar) [20:50:28] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T281168 (10brennen) [21:11:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10Cloud-VPS, 10Wikidata, and 3 others: Wikibase selenium tests timeout, seemingly due to "memory compaction" events on CI VMs - https://phabricator.wikimedia.org/T281122 (10nskaggs) I wanted to follow-up on this ticket to say a few... [21:32:00] (03PS1) 10Hashar: Replace setuptools_scm with setuptools-git-versioning [integration/quibble] - 10https://gerrit.wikimedia.org/r/730646 (https://phabricator.wikimedia.org/T292772) [21:37:01] so hmm python ecosystem is fun but has some issues sometime ;) [22:14:42] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T281168 (10dancy) [23:12:49] !log Killed a couple of deployment-prep update jobs in Jerkins that were hung on "waiting for executor" with 4 open executors. [23:12:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:13:01] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T281168 (10matmarex) [23:24:40] 10Phabricator: Add a Herald rule for User-MediaJS - https://phabricator.wikimedia.org/T286077 (10MediaJS) If authored/assigned If description contains Miraheze Add #user-mediajs Add @MediaJS as subscriber