[02:03:00] (03PS1) 10TrainBranchBot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/708195 [02:03:02] (03CR) 10TrainBranchBot: [C: 03+2] Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/708195 (owner: 10TrainBranchBot) [02:04:21] (03Merged) 10jenkins-bot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/708195 (owner: 10TrainBranchBot) [02:32:50] (03PS1) 10TrainBranchBot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/708198 [02:32:52] (03CR) 10TrainBranchBot: [C: 03+2] Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/708198 (owner: 10TrainBranchBot) [02:33:47] (03Merged) 10jenkins-bot: Update state/train-versions.json [tools/release] - 10https://gerrit.wikimedia.org/r/708198 (owner: 10TrainBranchBot) [04:51:53] 10Release-Engineering-Team (Deployment Training Requests), 10User-Urbanecm_WMF: Deployment training request for TThoabala - https://phabricator.wikimedia.org/T286546 (10ArielGlenn) a:05ArielGlenn→03Urbanecm_WMF I didn't acrtually give this training because my brain was fried from collab sessions + offsite!... [05:08:02] 10Continuous-Integration-Infrastructure, 10Pywikibot, 10Pywikibot-tests, 10Upstream: flake8 hacking does not reportunittest.TestCase issues - https://phabricator.wikimedia.org/T287434 (10Xqt) [05:10:16] 10Continuous-Integration-Infrastructure, 10Pywikibot, 10Pywikibot-tests, 10Upstream: flake8 hacking does not report unittest.TestCase issues - https://phabricator.wikimedia.org/T287434 (10Xqt) [07:20:18] (03CR) 10Jforrester: [C: 03+2] setup.cfg: replace dashes with underscores [integration/quibble] - 10https://gerrit.wikimedia.org/r/707345 (owner: 10Hashar) [07:24:11] James_F: :] [07:24:19] good morning [07:30:21] Morning! [07:38:52] (03Merged) 10jenkins-bot: setup.cfg: replace dashes with underscores [integration/quibble] - 10https://gerrit.wikimedia.org/r/707345 (owner: 10Hashar) [07:57:14] (03CR) 10Hashar: "Given it is doing a single request which can be done anonymously, I think it is fine to go directly with requests.get(), that makes it str" (034 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) (owner: 10Jeena Huneidi) [08:19:27] (03CR) 10Jeena Huneidi: scap backport command (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) (owner: 10Jeena Huneidi) [08:44:59] (03PS19) 10Jeena Huneidi: scap backport command [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) [08:45:36] (03CR) 10jerkins-bot: [V: 04-1] scap backport command [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) (owner: 10Jeena Huneidi) [08:47:25] (03PS20) 10Jeena Huneidi: scap backport command [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) [08:47:58] (03CR) 10jerkins-bot: [V: 04-1] scap backport command [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) (owner: 10Jeena Huneidi) [08:49:27] (03CR) 10Kevin Bazira: [C: 03+1] "LGTM!" [integration/config] - 10https://gerrit.wikimedia.org/r/708175 (https://phabricator.wikimedia.org/T279004) (owner: 10Accraze) [08:49:50] (03PS21) 10Jeena Huneidi: scap backport command [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) [08:50:24] (03CR) 10jerkins-bot: [V: 04-1] scap backport command [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) (owner: 10Jeena Huneidi) [08:51:20] (03PS22) 10Jeena Huneidi: scap backport command [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) [08:59:20] Morning, I want to build a daily job to run quibble for ungated extensions of Wikidata team daily (T285049) [08:59:20] T285049: Run CI tests daily on master for ungated extensions - https://phabricator.wikimedia.org/T285049 [08:59:38] I made this patch and output of jjb seems okay but I'm not sure if it's correct [08:59:45] https://gerrit.wikimedia.org/r/703785 [09:00:11] if I'm missing something super obvious, please point me to some documentation or anything you think is needed [09:08:42] 10Release-Engineering-Team, 10Wikidata-Query-Service: wikidata-query-rdf-maven-release-docker build is too slow and always times out - https://phabricator.wikimedia.org/T287445 (10dcausse) [09:34:35] 10Release-Engineering-Team, 10Wikidata-Query-Service: wikidata-query-rdf-maven-release-docker build is too slow and always times out - https://phabricator.wikimedia.org/T287445 (10dcausse) Might be related to T273086 [10:08:49] 10Release-Engineering-Team (Doing), 10Wikidata, 10Wikidata-Query-Service: wikidata-query-rdf-maven-release-docker build is too slow and always times out - https://phabricator.wikimedia.org/T287445 (10hashar) a:03hashar The two root causes are: - archiva.wikimedia.org has very poor latency/bandwith. I have... [10:36:37] 10Project-Admins: Create project tag for - https://phabricator.wikimedia.org/T287456 (10Spinster) [10:38:27] 10Project-Admins: Create project tag for Reconciliation - https://phabricator.wikimedia.org/T287456 (10Peachey88) [10:41:37] 10Project-Admins: Create project tag for Reconciliation - https://phabricator.wikimedia.org/T287456 (10Spinster) Some further clarification and context: * This project partly overlaps with #openrefine, but reconciliation code and functionalities are also used outside of OpenRefine. It will be convenient to have... [10:43:58] 10Project-Admins: Create project tag for Reconciliation - https://phabricator.wikimedia.org/T287456 (10Spinster) [11:29:19] 10Gerrit, 10Wikidata, 10Wikidata-Campsite: wikidata-dev instances causing git "Internal error during upload-pack" every 5 minutes - https://phabricator.wikimedia.org/T287459 (10hashar) [12:42:36] 10Gerrit, 10Wikidata, 10Wikidata-Campsite: wikidata-dev instances causing git "Internal error during upload-pack" every 5 minutes - https://phabricator.wikimedia.org/T287459 (10Lucas_Werkmeister_WMDE) The wb-reconcile instance uses our [git-updater ansible role](https://github.com/wmde/ansible-roles/blob/mai... [13:03:08] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T281157 (10Pchelolo) [13:07:13] (03PS1) 10Hashar: jjb: save maven cache in maven-release-docker jobs [integration/config] - 10https://gerrit.wikimedia.org/r/708285 (https://phabricator.wikimedia.org/T287445) [13:07:15] (03PS1) 10Hashar: jjb: raise wikidata/query/rdf timeout [integration/config] - 10https://gerrit.wikimedia.org/r/708286 (https://phabricator.wikimedia.org/T287445) [13:09:14] (03CR) 10Hashar: "Saving of the cache is done via a system I nicknamed "castor". It invokes a step after the build has completed which only save the cache w" [integration/config] - 10https://gerrit.wikimedia.org/r/708285 (https://phabricator.wikimedia.org/T287445) (owner: 10Hashar) [13:10:41] (03CR) 10Hashar: "That raises the timeout to 60 minutes since 45 minutes is not enough without a maven cache." [integration/config] - 10https://gerrit.wikimedia.org/r/708286 (https://phabricator.wikimedia.org/T287445) (owner: 10Hashar) [13:24:27] (03CR) 10Gehel: "> Patch Set 1:" [integration/config] - 10https://gerrit.wikimedia.org/r/708286 (https://phabricator.wikimedia.org/T287445) (owner: 10Hashar) [13:24:57] (03CR) 10Gehel: [C: 03+1] jjb: raise wikidata/query/rdf timeout [integration/config] - 10https://gerrit.wikimedia.org/r/708286 (https://phabricator.wikimedia.org/T287445) (owner: 10Hashar) [13:27:19] gehel: merci :) [13:29:06] (03CR) 10Hashar: [C: 03+2] "> The maven dependency plugin provides a goal to remove the dependencies used by a project, but not the other way around: https://maven.ap" [integration/config] - 10https://gerrit.wikimedia.org/r/708286 (https://phabricator.wikimedia.org/T287445) (owner: 10Hashar) [13:29:15] (03CR) 10Hashar: [C: 03+2] jjb: save maven cache in maven-release-docker jobs [integration/config] - 10https://gerrit.wikimedia.org/r/708285 (https://phabricator.wikimedia.org/T287445) (owner: 10Hashar) [13:30:18] hashar: using go-offline after a build should have mostly no effect (except wasting time) [13:30:34] meeting for the rest of the day, but ping me tomorrow if you want to discuss it [13:30:39] (03Merged) 10jenkins-bot: jjb: save maven cache in maven-release-docker jobs [integration/config] - 10https://gerrit.wikimedia.org/r/708285 (https://phabricator.wikimedia.org/T287445) (owner: 10Hashar) [13:30:41] (03Merged) 10jenkins-bot: jjb: raise wikidata/query/rdf timeout [integration/config] - 10https://gerrit.wikimedia.org/r/708286 (https://phabricator.wikimedia.org/T287445) (owner: 10Hashar) [13:32:04] 10Release-Engineering-Team (Doing), 10Wikidata, 10Wikidata-Query-Service, 10Patch-For-Review: wikidata-query-rdf-maven-release-docker build is too slow and always times out - https://phabricator.wikimedia.org/T287445 (10hashar) The timeout for https://integration.wikimedia.org/ci/job/wikidata-query-rdf-mav... [13:33:58] 10Release-Engineering-Team (Doing), 10Wikidata, 10Wikidata-Query-Service, 10Patch-For-Review: wikidata-query-rdf-maven-release-docker build is too slow and always times out - https://phabricator.wikimedia.org/T287445 (10dcausse) @hashar thanks! I launched a build for testing (https://integration.wikimedia.... [13:35:03] 10Release-Engineering-Team (Doing), 10Wikidata, 10Wikidata-Query-Service, 10Discovery-Search (Current work), 10Patch-For-Review: wikidata-query-rdf-maven-release-docker build is too slow and always times out - https://phabricator.wikimedia.org/T287445 (10dcausse) [14:12:15] dcausse: j'éspère que ca passera sinon je bump le timeout à genre 3 heures [14:13:00] et on voit bien qu'archiva est trèèèèès lent :-\ [14:13:57] hashar: ok, si le cache reste même en cas d'échec ca devrait être bon sinon il faudra augmenter le timeout en effet :/ [14:14:21] le cache n'est sauvé qu'en cas de succès [14:14:29] :| [14:14:51] (4.4 kB at 701 B/s) ... [14:35:48] (03PS1) 10Hashar: jjb: raise timeout wikidata query rdf release job [integration/config] - 10https://gerrit.wikimedia.org/r/708301 (https://phabricator.wikimedia.org/T273086) [14:36:10] dcausse: bumped it to four hours timeout :( [14:36:14] can you rerun it? [14:36:22] and archiva.wikimedia.org definitely has an issue of some sort today [14:37:00] (03CR) 10Hashar: [C: 03+2] jjb: raise timeout wikidata query rdf release job [integration/config] - 10https://gerrit.wikimedia.org/r/708301 (https://phabricator.wikimedia.org/T273086) (owner: 10Hashar) [14:38:05] (03Merged) 10jenkins-bot: jjb: raise timeout wikidata query rdf release job [integration/config] - 10https://gerrit.wikimedia.org/r/708301 (https://phabricator.wikimedia.org/T273086) (owner: 10Hashar) [14:40:52] dcausse: I am rebuilding the last build [14:41:10] https://integration.wikimedia.org/ci/job/wikidata-query-rdf-maven-release-docker/62/console [14:49:39] hashar: thanks! [15:16:17] (03CR) 10Ahmon Dancy: scap backport command (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) (owner: 10Jeena Huneidi) [16:08:06] (03PS1) 10Jbond: deployments-calendar: update IRC name for jbond [tools/release] - 10https://gerrit.wikimedia.org/r/708316 [16:10:11] (03CR) 10Hashar: [C: 03+2] deployments-calendar: update IRC name for jbond [tools/release] - 10https://gerrit.wikimedia.org/r/708316 (owner: 10Jbond) [16:11:16] (03Merged) 10jenkins-bot: deployments-calendar: update IRC name for jbond [tools/release] - 10https://gerrit.wikimedia.org/r/708316 (owner: 10Jbond) [16:27:17] dcausse: 00:00:00.715 java.io.IOException: No space left on device :D [16:27:28] meh... [16:27:45] I guess that whole cache system is definitely obsolete now [16:28:02] something's wrong with this build [16:28:02] I don't get why the normal build works tho [16:28:10] oh it worked fine [16:28:28] Total time: 01:41 h [16:28:40] the step after the build failed but it does not cause the build to be marked as a failure [16:32:58] !log cleaned some obsolete caches under integration-castor03 /srv/jenkins-workspace/caches [16:32:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:34:00] James_F if you're around (or someone else with access) would you mind telling github to run tests on my pull request https://github.com/wikimedia/eslint-plugin-mediawiki/pull/65 ? "First-time contributors need a maintainer to approve running workflows." [16:37:47] (03CR) 10Jeena Huneidi: scap backport command (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) (owner: 10Jeena Huneidi) [16:38:32] (03PS23) 10Jeena Huneidi: scap backport command [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) [16:39:31] (03CR) 10Ahmon Dancy: [C: 03+1] "I haven't tested yet but code looks good to me." [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) (owner: 10Jeena Huneidi) [16:55:18] (03PS1) 1020after4: WIP: workboard metrics [releng/ddd] - 10https://gerrit.wikimedia.org/r/708319 [16:55:46] !log creating new gitlab runner instance runner-1002 for testing [16:55:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:56:29] brennen ^ fyi. i'm hacking on some puppet for the package installation and registration [16:57:20] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Miriam) HI @Aklapper. As a Senior Research Scientist in the Research team I would like to be able to create milestones to organize and structure our team's projects. Could you please... [16:59:36] dduvall: ack, thx [17:00:05] 10Release-Engineering-Team, 10GitLab: Puppetize GitLab runner configuration - https://phabricator.wikimedia.org/T287504 (10dduvall) [17:00:16] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10mmodell) @miriam: Done. Quoting Andre's disclaimer: >>! In T706#7171709, @Aklapper wrote: >//Usual disclaimer: Please follow [guidelines](https://www.mediawiki.org/wiki/Phabricator... [17:09:49] 10Release-Engineering-Team (Doing), 10GitLab: Puppetize GitLab runner configuration - https://phabricator.wikimedia.org/T287504 (10thcipriani) [17:12:27] 10Release-Engineering-Team (Doing), 10GitLab: Puppetize GitLab runner configuration - https://phabricator.wikimedia.org/T287504 (10brennen) [17:12:29] 10Release-Engineering-Team (Doing), 10GitLab (Initialization), 10User-brennen: Experiment with gitlab-runners project and provision a pool of shared runners for general use - https://phabricator.wikimedia.org/T287279 (10brennen) [17:12:47] 10Release-Engineering-Team (Doing), 10GitLab, 10User-brennen: Puppetize GitLab runner configuration - https://phabricator.wikimedia.org/T287504 (10brennen) [17:13:29] 10Release-Engineering-Team (Doing), 10GitLab (Initialization), 10User-brennen: Puppetize GitLab runner configuration - https://phabricator.wikimedia.org/T287504 (10brennen) [17:13:40] 10Continuous-Integration-Config, 10Release-Engineering-Team (Radar), 10MediaWiki-Vendor, 10Parsoid, and 2 others: `mediawiki-core-php72-phan-docker` job runs `composer install` instead of using packages from mediawiki/vendor - https://phabricator.wikimedia.org/T287419 (10thcipriani) [17:17:37] 10Release-Engineering-Team, 10GitLab, 10User-brennen: Document long-term requirements for GitLab job runners - https://phabricator.wikimedia.org/T286958 (10brennen) [17:18:26] 10Release-Engineering-Team (Doing), 10Wikidata, 10Wikidata-Query-Service, 10Release Pipeline (Blubber): PipelineBot for flink-rdf-streaming-updater does not generate proper deployment-chart patches - https://phabricator.wikimedia.org/T287374 (10thcipriani) Per our team triage this might be a naming issue,... [17:21:00] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10GitLab, 10Release Pipeline, 10User-brennen: Figure out the future of (or replacements for) PipelineLib in a GitLab world - https://phabricator.wikimedia.org/T287211 (10thcipriani) [17:26:43] 10Phabricator, 10Release-Engineering-Team (Doing), 10Security-Team: Grant access to OIT-LDAP Diffusion repo to contractor Danielattevelt - https://phabricator.wikimedia.org/T287124 (10thcipriani) a:03mmodell [17:27:41] 10Release-Engineering-Team (Next), 10GitLab (Initialization): Establish a routine GitLab deployment / update window - https://phabricator.wikimedia.org/T287117 (10thcipriani) [17:28:34] 10Phabricator, 10Release-Engineering-Team (Doing), 10Security-Team: Grant access to OIT-LDAP Diffusion repo to contractor Danielattevelt - https://phabricator.wikimedia.org/T287124 (10hashar) [17:28:51] 10Release-Engineering-Team (Seen), 10GitLab, 10User-brennen: Document long-term requirements for GitLab job runners - https://phabricator.wikimedia.org/T286958 (10brennen) [17:29:18] 10Phabricator, 10Release-Engineering-Team (Doing), 10Security-Team: Grant access to OIT-LDAP Diffusion repo to contractor Danielattevelt - https://phabricator.wikimedia.org/T287124 (10mmodell) Simply adding the user to the policy on {rWOLT} should be fine. I'll take care of it if I can access that ACL. [17:29:22] 10Release-Engineering-Team (Doing), 10Wikidata, 10Wikidata-Query-Service, 10Release Pipeline (Blubber): PipelineBot for flink-rdf-streaming-updater does not generate proper deployment-chart patches - https://phabricator.wikimedia.org/T287374 (10jeena) Currently the promote step doesn't support updating the... [17:32:06] 10Release-Engineering-Team (Radar), 10Growth-Team, 10StructuredDiscussions, 10Patch-For-Review, and 2 others: Core tests failing due to Flow HTTP requests and ServiceContainer access - https://phabricator.wikimedia.org/T287001 (10thcipriani) [17:34:13] 10Phabricator, 10Release-Engineering-Team (Doing), 10Security-Team: Grant access to OIT-LDAP Diffusion repo to contractor Danielattevelt - https://phabricator.wikimedia.org/T287124 (10mmodell) Ok I added @Danielattevelt to the //visibility// `custom policy` via the rWOLT repository's [[ /diffusion/edit/2045/... [17:35:16] 10Release-Engineering-Team (Radar), 10Growth-Team, 10StructuredDiscussions, 10Patch-For-Review, and 2 others: Core tests failing due to Flow HTTP requests and ServiceContainer access - https://phabricator.wikimedia.org/T287001 (10thcipriani) Moving to watching as we're not clear on the solution, but once a... [17:41:40] 10Release-Engineering-Team (Radar), 10Community-Relations, 10Quality-and-Test-Engineering-Team (QTE), 10serviceops-radar: Expand the list of group 1 wikis to contain at least one (preferably 2) smaller "top ten size" wikis - https://phabricator.wikimedia.org/T286664 (10thcipriani) Adding #serviceops-radar... [17:43:48] (03Abandoned) 10Accraze: Review access change [machinelearning/liftwing] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/692082 (owner: 10Accraze) [17:43:50] 10Release-Engineering-Team (Doing), 10Scap: Scap should release lock on SIGTERM - https://phabricator.wikimedia.org/T192703 (10thcipriani) 05Open→03Resolved Still awaiting release, but calling this solved for now. Reopen if there are issues. [17:45:13] 10Release-Engineering-Team: java.io.NotSerializableException: groovy.lang.IntRange in Jenkins CI - https://phabricator.wikimedia.org/T287507 (10cmassaro) [17:45:27] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Seen): Switch back to upstream jenkins xunit plugin after PHPUnit fix is released - https://phabricator.wikimedia.org/T194096 (10thcipriani) a:05hashar→03None [17:46:08] 10Gerrit, 10Release-Engineering-Team (Seen): Install rename-project plugin - https://phabricator.wikimedia.org/T201953 (10thcipriani) [17:48:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Elasticsearch: Set up data storage to collect loosely structured data from CI - https://phabricator.wikimedia.org/T211904 (10thcipriani) p:05High→03Low a:05mmodell→03None [17:48:43] 10Continuous-Integration-Config, 10Release-Engineering-Team (Seen): Make test pipline vote Verified+1 instead of +2 to avoid unintentional submit - https://phabricator.wikimedia.org/T226123 (10thcipriani) [17:50:06] 10Release-Engineering-Team (Seen), 10Code-Review-Workgroup: Define code review metrics - https://phabricator.wikimedia.org/T229510 (10thcipriani) p:05Medium→03Low a:05Jdforrester-WMF→03None @Jdforrester-WMF feel free to re-claim if you're still working on this :) [17:53:52] 10Release-Engineering-Team, 10Release Pipeline (Blubber): java.io.NotSerializableException: groovy.lang.IntRange in Jenkins CI - https://phabricator.wikimedia.org/T287507 (10thcipriani) [17:54:04] 10Release-Engineering-Team, 10Release Pipeline: java.io.NotSerializableException: groovy.lang.IntRange in Jenkins CI - https://phabricator.wikimedia.org/T287507 (10hashar) [17:56:43] 10Release-Engineering-Team (Doing), 10Release Pipeline: java.io.NotSerializableException: groovy.lang.IntRange in Jenkins CI - https://phabricator.wikimedia.org/T287507 (10dduvall) p:05Triage→03Medium a:03dduvall This is likely a [[ https://github.com/cloudbees/groovy-cps | Groovy-CPS ]] related error st... [18:11:53] (03PS1) 10Zoranzoki21: [zuul/layout.yaml] Archive EUCopyrightCampaign related extension and skin [integration/config] - 10https://gerrit.wikimedia.org/r/708328 (https://phabricator.wikimedia.org/T256690) [18:37:45] 10Phabricator, 10Release-Engineering-Team (Doing), 10Security-Team: Grant access to OIT-LDAP Diffusion repo to contractor Danielattevelt - https://phabricator.wikimedia.org/T287124 (10sbassett) Thanks, @mmodell. [18:38:02] 10Phabricator, 10Release-Engineering-Team (Doing), 10Security-Team: Grant access to OIT-LDAP Diffusion repo to contractor Danielattevelt - https://phabricator.wikimedia.org/T287124 (10sbassett) 05Open→03Resolved [19:34:53] jeena, dduvall: does the failure at https://integration.wikimedia.org/ci/blue/organizations/jenkins/wikimedia-toolhub-pipeline-publish/detail/service-pipeline-test-and-publish/3/pipeline/ ("pull access denied for prep") mean that my publish pipeline needs to explicitly build the prep image? [19:36:18] I was trying to be "nice" with resources and not re-run all the tests in the post-merge trigger, but I think I was assuming a lot more blubber magic than there seems to be happening in that run. [19:37:12] bd808: hmm, blubber should have included the prep stage for you [19:38:14] ack. when I call the blubber service to generate a dockerfile for my "production" target it looks like it has all the things [19:44:25] looks like maybe the `includes: [production]` in demo and `copies: [ { from: prep } ]` in production is not being transitive like it should be [19:45:56] bd808: congrats! you have discovered a bug in blubber :) [19:48:51] oh wow. yeah, my local `make .pipeline/demo.Dockerfile` is weirdly broken now. I guess if I figure out how to fix that things may go better. [19:50:22] i'm knee deep in puppet atm but if you can file a bug i can take a look later or tomorrow [19:50:52] a temporary workaround would be just to duplicate the production variant config in your demo variant [19:50:54] ack [20:01:05] (03CR) 10Jeena Huneidi: [C: 04-1] "I don't see an editquality pipeline in your .pipeline/config.yaml for machinelearning/liftwing/inference-services" [integration/config] - 10https://gerrit.wikimedia.org/r/708175 (https://phabricator.wikimedia.org/T279004) (owner: 10Accraze) [20:09:38] hey releng! Anyone have experience/docs on how to test a helm chart using a local docker image in minikube (instead of the published images on the wmf registry)? ive created a local registry but was having some issues, wondering if anyone has experience doing so [20:09:51] yup I can help you [20:10:12] !!! yay! [20:10:39] you wanna jump into a private chat? or here ? [20:11:12] if you do "eval $(minikube docker-env)" in your terminal and then build your docker image with whatever tag you want then it will be in your minikube's registry [20:12:07] lmk if you're still having trouble/have more questions [20:13:11] ok that is VASTLY simpler than what i was attempting before - i will give it a shot! [20:13:17] hehe [20:14:35] so after you do the eval command then in that terminal window you will be using minikube's docker daemon [20:20:21] (need to prune some old containers first looks like its out of memory) [20:29:15] ok! so once i build the image, and tag it with an arbitrary tag, then i would update my helm chart's values.yaml with the correct image and version? [20:31:28] yup [20:32:05] oh umm if you didn't tag it with the wikimedia registry url you might need to make some extra changes to the values [20:32:07] and erm, the registry? [20:32:19] oh ok i will tag it with the registry name in that case! [20:32:26] okay cool [20:33:04] and also change the pull policy to IfNotPresent if it isn't that already [20:35:08] i think the scaffolding makes that the default! thats how it is already et [20:40:09] Hi all! seeking a bit of help figuring out why we're unable to locally reproduce test failures that we see in CI for a change [20:40:24] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/708333/ [20:40:41] It's a complicated CI setup but I think we've got it running the same locally, but there must be something different [20:40:43] thanks in advance! [20:41:54] looking [20:46:15] AndyRussG: I don't really know enough about quibble but hopefully hashar can check when he is around [20:49:04] jeena: ok thanks much! [20:49:15] it's a really hacky setup [20:59:41] jeena: thank you so much that worked :) [21:00:22] glad to hear it! [21:11:39] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T281157 (10mmodell) [21:17:20] 10Release-Engineering-Team (Doing), 10GitLab (Initialization), 10User-brennen: Remove Speed & Function blockers for GitLab work - https://phabricator.wikimedia.org/T274458 (10colewhite) [21:18:03] dduvall, hi, I'm Dylan, engineering manager for fundraising tech. We're having some CI trouble and I got your name from the wiki, and was wondering if you could help us. [21:18:31] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [21:18:53] We've got a test erroring on us, and we're running up against a campaign deadline. We were hoping you could help us force merge a patch so we can move forward. [21:19:09] We're in a google hangouts call if that would be helpful for us to explain the situation in more detail. [21:24:56] Or thcipriani, if Dan isn't the right person to ask, if you could point me in the right direction. [21:25:39] uh, sure, do you need to bypass CI for one of your repos? [21:26:05] thcipriani: yep! DonationInterface [21:26:10] I can change permissions to let you push to it if you really need to get something out quickly [21:26:19] thcipriani: yes that'd be fantastic, thanks so so much!!!!! [21:26:32] Yea, that would be exactly what we need, thank you for the quick help [21:26:33] what's failing in CI? [21:26:43] uhhhhh a thing? [21:26:56] heh, link? [21:27:02] thcipriani: 32 phpcs sniffs [21:27:07] it's the core special page test, yes one sec [21:27:10] it passes locally though [21:27:19] https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/708333/ [21:27:28] https://integration.wikimedia.org/ci/job/mwgate-composer-php73-docker/1176/console [21:27:48] no the php linting is not an issue forus [21:28:15] you should have the ability to push directly to a ref now (i.e., git push origin master should work right now) [21:28:16] the php linting is just on some hacky test patches we're pushing up [21:28:20] lemme take a look at this error [21:28:53] thcipriani: this is the patch: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/708333/ [21:29:09] relevant failure is here: https://integration.wikimedia.org/ci/job/quibble-donationinterface-REL1_35-php73-docker/636/console [21:29:25] it's something to do with an issue on CI that doesn't exist locally [21:29:33] pls ignore the follow-on patches btw :) [21:29:47] looks like they're all coming from, "Undefined index: result" [21:30:06] yeah we don't get that locally, but that one we figured out and fixed, it's the first one that's an issue [21:30:12] k [21:30:17] that's odd [21:30:32] latest update is it might have to do with us not having properly mocked another service [21:30:37] have you tried just re-running them on ci? [21:31:01] hrm, I guess the first error is Call to a member function getAccountConfig() on null [21:32:43] yeah we have been trying to get that fixed all day, there is some config missing but that error isn't the actual error [21:32:52] ah, k [21:33:02] we hadn't been able to replicate it locally hence the CI testing spam [21:33:07] anyway: like I said you all should be able to push to whichever branch you need now [21:33:15] thanks thcipriani ! [21:33:36] (03CR) 10Accraze: "> Patch Set 2: Code-Review-1" [integration/config] - 10https://gerrit.wikimedia.org/r/708175 (https://phabricator.wikimedia.org/T279004) (owner: 10Accraze) [21:33:38] and hopefully that will give you the space you need for a proper fix [21:33:58] let me know if you can't push [21:34:41] (03PS1) 10Ahmon Dancy: fix sudo_check_call str/bytes handling [tools/scap] - 10https://gerrit.wikimedia.org/r/708356 [21:34:54] we are trying one last fix attempt but we'll let you know if we can't thcipriani [21:35:26] cstone: good luck :) [21:35:29] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [21:42:57] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments, 10User-jeena (Sprint 7/28-8/10): 1.37.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T281159 (10jeena) [21:45:54] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments, 10User-jeena: 1.37.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T281160 (10jeena) [21:49:27] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline, 10Patch-For-Review, 10User-jeena (Sprint 7/12-7/27): Scap Backport --list command - https://phabricator.wikimedia.org/T287041 (10jeena) [21:49:50] (03PS1) 10Ahmon Dancy: www container improvements [tools/train-dev] - 10https://gerrit.wikimedia.org/r/708362 [21:50:11] (03CR) 10Ahmon Dancy: [V: 03+2 C: 03+2] www container improvements [tools/train-dev] - 10https://gerrit.wikimedia.org/r/708362 (owner: 10Ahmon Dancy) [22:00:04] thcipriani: we have the power thanks again [22:00:31] cstone: glad to hear it :) [22:03:01] (03PS6) 10Brennen Bearnes: WIP: add buster php images [releng/dev-images] - 10https://gerrit.wikimedia.org/r/698880 (https://phabricator.wikimedia.org/T273100) [22:03:25] 10Release-Engineering-Team (Doing), 10Quibble: Establish communication channel for Quibble development (plot twist: Slack channel) - https://phabricator.wikimedia.org/T286770 (10cscott) I generally use a Matrix client, to which I've bridged all my IRC and slack channels. So @Legoktm's solution would work for m... [22:06:43] thcipriani: I spoke too soon we don't have the power [22:07:06] cstone: heh, what's the message you're seeing? [22:07:13] thcipriani: looks like we can push to the adyen branch but not to master [22:07:21] via the cli `git push origin master` route [22:07:26] git push origin master [22:07:27] Enumerating objects: 16, done. [22:07:29] Counting objects: 100% (16/16), done. [22:07:31] Delta compression using up to 8 threads [22:07:33] Compressing objects: 100% (6/6), done. [22:07:35] Writing objects: 100% (6/6), 1.61 KiB | 1.61 MiB/s, done. [22:07:37] Total 6 (delta 5), reused 0 (delta 0), pack-reused 0 [22:07:39] remote: Resolving deltas: 100% (5/5) [22:07:41] remote: error: branch refs/heads/master: [22:07:43] remote: To push into this reference you need 'Push' rights. [22:07:45] remote: User: andyrussg [22:07:47] remote: Contact an administrator to fix the permissions [22:07:49] remote: Processing changes: refs: 1, done [22:07:51] To ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/DonationInterface.git [22:07:53] ! [remote rejected] master -> master (prohibited by Gerrit: not permitted: update) [22:07:55] error: failed to push some refs to 'ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/DonationInterface.gi [22:07:57] thcipriani: ^ [22:08:02] thanks [22:08:11] thcipriani: thank u!!!!! :) :) [22:08:12] * jgleeson it finishes! [22:08:22] also I'm being scolded for being an IRC spammer [22:09:04] oh! [22:09:21] I added push for refs but it looks like there's a special section for refs/heads/master [22:09:22] * thcipriani fixes [22:10:07] AndyRussG: try now [22:11:48] thcipriani: ! [remote rejected] master -> master (commit already exists (in the project): refs/changes/21/708221/1) [22:11:50] error: failed to push some refs to 'ssh://gerrit.wikimedia.org:29418/mediawiki/extensions/DonationInterface.git' [22:11:58] thcipriani: it's somehow merged [22:12:02] so well... hmmm [22:12:11] oh good? [22:12:14] :) [22:12:24] thcipriani: we'll also need this power for the deployment branch on the same repo btw [22:12:46] AndyRussG: I think you can push to all the branches for that repo now [22:13:28] (03PS24) 10Thcipriani: scap backport command [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) (owner: 10Jeena Huneidi) [22:16:09] jgleeson: AndyRussG: iirc lack of Push permission on the DonationInterface repo was to ensure every commit passed through review [22:16:09] thcipriani: ok thanks! [22:16:21] hashar: yes it's been through many review [22:16:24] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10SRE, 10serviceops, 10Patch-For-Review: Check out www-portals repo in the mediawiki-webserver and in the mediawiki-multiversion images - https://phabricator.wikimedia.org/T285325 (10jeena) 05Open→03Resolved a:03jeena We had updated the jenkins confi... [22:16:30] but it's urgent to push this change out [22:16:37] +1 ;-) [22:19:54] as for the adyen branch: the quibble-donationinterface-REL1_35-php73-docker job is configured with `--project-branch mediawiki/extensions/DonationInterface=master` [22:20:21] so eg https://gerrit.wikimedia.org/r/c/mediawiki/extensions/DonationInterface/+/708221 did not get tested with that change but whatever `master` was pointing to [22:22:31] the merge of `adyen` into `master` apparently got pushed directly to the repo bypassing CI [22:23:00] and I would guess the job might fail from now on since none of the change made to `adyen` actually got tested [22:23:14] yeah hashar we realized last night that they were testing on master and not adyen [22:23:35] we saw test results when we tried to merge adyen branch into master [22:24:02] yeah that is annoying [22:24:13] that job is neat when it is working properly and used as unexpected [22:24:19] (03CR) 10Thcipriani: [C: 03+2] "Seems to work as expected afaict. Defaulting to merge here, but there are a number of nitpicks for idiomatic changes that would be good in" (036 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) (owner: 10Jeena Huneidi) [22:24:23] but otherwise has too many way to explode :\ [22:24:54] anyway it is too late for me but jjb/wm-fundraising.yaml has a comment about: donation_interface_branch: master [22:25:11] yeah thanks for telling us where it is cause we want to fix in the future [22:25:12] (03Merged) 10jenkins-bot: scap backport command [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) (owner: 10Jeena Huneidi) [22:25:57] iirc the job is triggered from multiple repositories [22:26:10] each using fundraising/REL1_35 [22:26:30] with DonationInterface being the outlier and using `master` and that is hardcoded in the job [22:26:47] so even if a change targets another branch, CI is hardcoded to always use the `master` branch [22:27:59] ah found it [22:28:12] AndyRussG: cstone: my comment from last time about the branch stuff: https://phabricator.wikimedia.org/T261777#6541811 [22:28:38] (03PS1) 10Jeena Huneidi: Add prettyTable package [tools/train-dev] - 10https://gerrit.wikimedia.org/r/708371 [22:29:10] some mismatch due to some repositories lacking a fundraising/REL1_35 branch and CI at the time used `master` as a fallback. It is now using `REL1_35` as a fallback [22:29:28] the short story is that the system is not very smart ;) [22:29:31] hashar: ok cool thanks, we will definitely continue to dig in and make tests pass properly ASAP after we finish force-pushing the current stuff [22:29:46] hashar: it's ok, it's quite smart :) [22:30:11] and should theorically prevent failling tests from entering the repo :-] [22:30:20] hashar: I feel that in general we just need to document these gotchas, and potentially make CI slightly closer to to production [22:30:54] possibly yes [22:30:59] hashar: basically we have just today as a window to deploy something so that it can get some small-scalre real-world testing, and if we don't, it hast to get pushed out to September [22:31:00] I meant [22:31:13] definitely +1 on doc and making it simpler [22:32:58] AndyRussG: no worries, good luck with the deploy! :] [22:33:15] hashar: thanks so so much and apologies for the bother!! :) [22:36:02] thcipriani: the merge to our deployment branch worked fine btw [22:36:24] (03PS2) 10Thcipriani: fix sudo_check_call str/bytes handling [tools/scap] - 10https://gerrit.wikimedia.org/r/708356 (owner: 10Ahmon Dancy) [22:36:33] great :) [22:43:48] (03CR) 10Jeena Huneidi: scap backport command (035 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) (owner: 10Jeena Huneidi) [22:44:54] (03CR) 10Jeena Huneidi: "thanks for merging!" [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) (owner: 10Jeena Huneidi) [22:46:32] (03CR) 10Thcipriani: [C: 04-1] "I'm fine with the universional_newlines (I think :)), but I'm mildly opposed the simplification of the poll logic for the subprocess: thar" (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/708356 (owner: 10Ahmon Dancy) [22:52:57] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline, 10Patch-For-Review, 10User-jeena (Sprint 7/12-7/27): Scap Backport --list command - https://phabricator.wikimedia.org/T287041 (10jeena) 05Open→03Resolved [22:53:01] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline, 10Patch-For-Review, 10User-brennen: Design m8s deployment workflows and tooling - https://phabricator.wikimedia.org/T279322 (10jeena) [22:53:09] 10Project-Admins: Create project tag for Data-Engineering - https://phabricator.wikimedia.org/T287531 (10odimitrijevic) [22:53:15] 10Release-Engineering-Team (Doing), 10MW-on-K8s, 10Release Pipeline, 10Patch-For-Review, 10User-jeena (Sprint 7/12-7/27): Scap Backport --list command - https://phabricator.wikimedia.org/T287041 (10jeena) [23:02:36] (03CR) 10Thcipriani: scap backport command (033 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) (owner: 10Jeena Huneidi) [23:03:56] (03CR) 10Thcipriani: scap backport command (031 comment) [tools/scap] - 10https://gerrit.wikimedia.org/r/701615 (https://phabricator.wikimedia.org/T287041) (owner: 10Jeena Huneidi) [23:37:09] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10User-brennen: logspam-watch broken - https://phabricator.wikimedia.org/T281121 (10brennen) 05Open→03Resolved [23:44:11] (03CR) 10Ahmon Dancy: "> Patch Set 2: Code-Review-1" (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/708356 (owner: 10Ahmon Dancy) [23:51:56] (03PS3) 10Ahmon Dancy: fix sudo_check_call str/bytes handling [tools/scap] - 10https://gerrit.wikimedia.org/r/708356 [23:55:57] (03CR) 10Ahmon Dancy: fix sudo_check_call str/bytes handling (032 comments) [tools/scap] - 10https://gerrit.wikimedia.org/r/708356 (owner: 10Ahmon Dancy) [23:57:57] (03CR) 10Ahmon Dancy: [V: 03+2 C: 03+2] Add prettyTable package [tools/train-dev] - 10https://gerrit.wikimedia.org/r/708371 (owner: 10Jeena Huneidi)