[03:13:25] Project mwcore-phpunit-coverage-master build #1690: 04STILL FAILING in 13 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1690/ [03:58:47] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [05:06:17] PROBLEM - SSH on gerrit2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [06:06:42] RECOVERY - SSH on gerrit2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [07:15:51] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for **CBlanton** - https://phabricator.wikimedia.org/T292679 (10Aklapper) ([offtopic] @CBlanton: Please either strip unneeded full quotes when repying, or use a delimiter (space dash dash), to keep things more readable. Thanks... [09:11:21] 10Release-Engineering-Team, 10Analytics: Add Olja as a member of the Analytics team on Gerrit - https://phabricator.wikimedia.org/T292823 (10Gehel) [09:11:45] hashar: I just created the access request above ^ [09:12:17] I could not find a template for Gerrit access requests, I hope there is all the required info. Not sure who needs to validate the request since Olja is the manager of that team. [09:22:08] https://phabricator.wikimedia.org/tag/gerrit-privilege-requests/ et https://www.mediawiki.org/wiki/Gerrit/Privilege_policy [09:22:10] je vais le faire [09:22:26] you even gave us the direct to gerrit admin interface :-] [09:24:29] gehel: {done} ! [09:24:30] 10Release-Engineering-Team, 10Analytics, 10Gerrit-Privilege-Requests: Add Olja as a member of the Analytics team on Gerrit - https://phabricator.wikimedia.org/T292823 (10hashar) 05Open→03Resolved a:03hashar The [[https://gerrit.wikimedia.org/r/admin/groups/d34747bee94be39cff54b5fda1ae36b575107792 | Ana... [09:24:46] hashar: thanks! [09:24:53] potentially we could move that to ldap [09:24:56] but well hmm [09:25:00] lets not open that can of worms [09:25:05] :) [10:02:05] 10Gerrit, 10Release-Engineering-Team, 10Discovery: Update gerrit submit type for discovery repositories in gerrit - https://phabricator.wikimedia.org/T255509 (10hashar) The documentations about submit type is at https://gerrit.wikimedia.org/r/Documentation/config-project-config.html#submit-type `Fast forwar... [10:09:39] 10Continuous-Integration-Config, 10Gerrit, 10Release-Engineering-Team (Seen): Make test pipline vote Verified+1 instead of +2 to avoid unintentional submit - https://phabricator.wikimedia.org/T226123 (10hashar) [10:15:44] hashar: could we not patch gerrit to make quick approve appear on V+1 too [10:15:57] For someone with +2 rights [10:16:07] 10Continuous-Integration-Config, 10Gerrit, 10Release-Engineering-Team (Seen): Make test pipline vote Verified+1 instead of +2 to avoid unintentional submit - https://phabricator.wikimedia.org/T226123 (10hashar) A reply I have made to an internal discussion we had: The task is https://phabricator.wikimedia.o... [10:16:38] RhinosF1: it is complicated :] [10:16:57] there is some side discussion about it in an internal chat (Slack) [10:44:21] hashar: ah! [10:49:01] (03Abandoned) 10Hashar: Dockerfiles: [node12] Re-build to upgrade node12 from 12.21.0 to 12.22.5 [integration/config] - 10https://gerrit.wikimedia.org/r/727617 (owner: 10Jforrester) [10:53:32] (03PS2) 10Hashar: dockerfiles: rebuild Bullseye images for security repos [integration/config] - 10https://gerrit.wikimedia.org/r/724501 [10:58:50] (03CR) 10Hashar: [C: 03+2] "James raised that in another change which I have abandoned since it is a dupe of this one." [integration/config] - 10https://gerrit.wikimedia.org/r/724501 (owner: 10Hashar) [10:59:14] James_F: doing the bullseye / node 12 update. Thanks for raising it again since I forgot about this change ^ [11:00:28] (03Merged) 10jenkins-bot: dockerfiles: rebuild Bullseye images for security repos [integration/config] - 10https://gerrit.wikimedia.org/r/724501 (owner: 10Hashar) [11:01:12] !log Rebuilding CI Docker images based on Bullseye to include apt security updates [11:01:15] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:33:46] hashar: seems to run fine in core, fwiw https://integration.wikimedia.org/ci/job/wmf-quibble-core-vendor-mysql-php72-docker/64235/consoleFull#console-section-10 [12:07:15] kostajh: hi ;) [12:10:59] (03PS1) 10Kosta Harlan: Release Quibble 1.1.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/728416 [12:11:27] (03Abandoned) 10Kosta Harlan: [DNM] Test run [integration/quibble] - 10https://gerrit.wikimedia.org/r/727444 (owner: 10Kosta Harlan) [12:11:46] I am looking at the side effect of running `composer install` [12:12:10] hashar: hi! so in summary - it works for core as is, the patch provides support for extensions/skins by running composer install first, in the same way we handle it (afaict) for the composer-test stage [12:15:04] so for core, that would install the dependencies from composer [12:15:19] which might add stuff to vendor.git when using it for dependencies [12:15:32] and they will then be used by qunit/apitesting/phpunit database suite [12:16:17] then the step running after phpbench runs composer test && npm test, but it does not do any installation [12:16:48] so for the use case of mediawiki/core the phpbench dev dependency has already been installed either via: [12:16:49] I’m not sure I understand exactly. Do you want to do a quick meeting? [12:16:54] - composer install from root of mw [12:17:02] - vendor.git being augment with dev dependencies [12:17:06] yeah sure [12:17:07] sorry [12:17:19] that workflow is a stupid mess of complexity [12:17:23] I’m brewing a coffee but then can join in a few minutes [12:17:32] good idea [12:19:40] kostajh: https://meet.google.com/kor-vqms-nrz?authuser=1 ;) [12:19:56] grbmblbl without the authuser parameter: https://meet.google.com/kor-vqms-nrz [12:21:26] k just a minute [12:35:27] (03PS2) 10Kosta Harlan: phpbench: Run composer install first [integration/quibble] - 10https://gerrit.wikimedia.org/r/727443 (https://phabricator.wikimedia.org/T291549) [12:35:29] (03PS2) 10Kosta Harlan: Release Quibble 1.1.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/728416 [12:39:23] (03CR) 10jerkins-bot: [V: 04-1] phpbench: Run composer install first [integration/quibble] - 10https://gerrit.wikimedia.org/r/727443 (https://phabricator.wikimedia.org/T291549) (owner: 10Kosta Harlan) [12:39:25] (03CR) 10jerkins-bot: [V: 04-1] Release Quibble 1.1.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/728416 (owner: 10Kosta Harlan) [12:40:09] (03CR) 10Hashar: "Successfully published image docker-registry.discovery.wmnet/releng/node12-test-browser-php80-composer:0.2.0-s3" [integration/config] - 10https://gerrit.wikimedia.org/r/724501 (owner: 10Hashar) [12:41:27] (03PS3) 10Kosta Harlan: phpbench: Run composer install first [integration/quibble] - 10https://gerrit.wikimedia.org/r/727443 (https://phabricator.wikimedia.org/T291549) [12:41:32] (03PS3) 10Kosta Harlan: Release Quibble 1.1.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/728416 [12:43:12] (03Restored) 10Kosta Harlan: [DNM] Test run [integration/quibble] - 10https://gerrit.wikimedia.org/r/727444 (owner: 10Kosta Harlan) [12:43:15] (03PS11) 10Kosta Harlan: [DNM] Test run [integration/quibble] - 10https://gerrit.wikimedia.org/r/727444 [12:44:27] (03CR) 10jerkins-bot: [V: 04-1] Release Quibble 1.1.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/728416 (owner: 10Kosta Harlan) [12:44:33] (03CR) 10jerkins-bot: [V: 04-1] phpbench: Run composer install first [integration/quibble] - 10https://gerrit.wikimedia.org/r/727443 (https://phabricator.wikimedia.org/T291549) (owner: 10Kosta Harlan) [12:45:21] (03PS4) 10Kosta Harlan: phpbench: Run composer install first [integration/quibble] - 10https://gerrit.wikimedia.org/r/727443 (https://phabricator.wikimedia.org/T291549) [12:45:23] (03PS4) 10Kosta Harlan: Release Quibble 1.1.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/728416 [12:46:43] (03CR) 10jerkins-bot: [V: 04-1] [DNM] Test run [integration/quibble] - 10https://gerrit.wikimedia.org/r/727444 (owner: 10Kosta Harlan) [12:48:42] (03PS12) 10Kosta Harlan: [DNM] Test run [integration/quibble] - 10https://gerrit.wikimedia.org/r/727444 [12:53:46] (03CR) 10Hashar: "Kosta and I spoke about this patch ;)" [integration/quibble] - 10https://gerrit.wikimedia.org/r/727443 (https://phabricator.wikimedia.org/T291549) (owner: 10Kosta Harlan) [12:54:05] (03CR) 10jerkins-bot: [V: 04-1] [DNM] Test run [integration/quibble] - 10https://gerrit.wikimedia.org/r/727444 (owner: 10Kosta Harlan) [12:58:28] (03PS5) 10Kosta Harlan: phpbench: Run composer install first [integration/quibble] - 10https://gerrit.wikimedia.org/r/727443 (https://phabricator.wikimedia.org/T291549) [12:58:30] (03PS5) 10Kosta Harlan: Release Quibble 1.1.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/728416 [12:58:38] (03PS13) 10Kosta Harlan: [DNM] Test run [integration/quibble] - 10https://gerrit.wikimedia.org/r/727444 [13:07:03] 10Project-Admins, 10Turkish-Sites: trwiki Twinkle project - https://phabricator.wikimedia.org/T289095 (10Aklapper) @TmY_e12: ping? [13:45:58] (03CR) 10Hashar: "One tiny issue with git clean." [integration/quibble] - 10https://gerrit.wikimedia.org/r/727443 (https://phabricator.wikimedia.org/T291549) (owner: 10Kosta Harlan) [14:23:38] 10Continuous-Integration-Config, 10MW-1.35-release, 10MW-1.36-release, 10MW-1.37-release, 10PHP 8.0 support: Make PHP 8.0 voting on MW 1.35, 1.36, 1.37 - https://phabricator.wikimedia.org/T274965 (10Reedy) [14:23:44] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10Jelto) I imported the latest data to `gitlab2001` and everything except ssh looks fine. I prepared... [14:24:54] 10Continuous-Integration-Config, 10MediaWiki-General, 10MW-1.37-release, 10PHP 8.0 support: Make MW 1.37 tests pass on PHP 8.0 - https://phabricator.wikimedia.org/T292847 (10Reedy) [14:38:55] (03CR) 10Hashar: phpbench: Run composer install first (033 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/727443 (https://phabricator.wikimedia.org/T291549) (owner: 10Kosta Harlan) [14:39:02] (03PS6) 10Hashar: phpbench: Run composer install first [integration/quibble] - 10https://gerrit.wikimedia.org/r/727443 (https://phabricator.wikimedia.org/T291549) (owner: 10Kosta Harlan) [14:39:19] (03PS14) 10Hashar: [DNM] Test run [integration/quibble] - 10https://gerrit.wikimedia.org/r/727444 (owner: 10Kosta Harlan) [14:39:45] kostajh: we will see how it goes. I have done some minor adjustement [14:40:11] if you have time to look at it this evening, I can cut the release/rebuild images etc sometime tonight or over the week-end [15:10:45] CI is looking a bit backed up [15:13:51] I would like to re-enable Puppet on Production Gitlab (gitlab1001) in 15 minutes. Ping me here or in https://phabricator.wikimedia.org/T283076 if you have any concerns [15:14:45] Project mwcore-phpunit-coverage-master build #1691: 04STILL FAILING in 12 min: https://integration.wikimedia.org/ci/job/mwcore-phpunit-coverage-master/1691/ [15:23:19] (03PS2) 10Ahmon Dancy: Use the remote origin url for Gerrit/git heuristic [tools/release] - 10https://gerrit.wikimedia.org/r/727433 (owner: 10Dduvall) [15:26:04] (03CR) 10Ahmon Dancy: "Things are looking good. The last bit needed for ./train to work non-interactively is for something to act like Jenkins and to submit a c" [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 (owner: 10Dduvall) [15:43:28] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10Jelto) Puppet on `gitlab1001` is enabled again and the puppet run was successful. Web interface wor... [15:51:14] (03CR) 10Ahmon Dancy: Replace simple git server with Gerrit (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 (owner: 10Dduvall) [16:13:50] 10Continuous-Integration-Config, 10MediaWiki-General, 10MW-1.37-release, 10PHP 8.0 support: Make MW 1.37 tests pass on PHP 8.0 - https://phabricator.wikimedia.org/T292847 (10Reedy) TODO: See what breaks. Guessing some overlap with {T283208} and possibly {T274966} [16:14:07] 10Continuous-Integration-Config, 10MediaWiki-General, 10MW-1.37-release, 10PHP 8.0 support: Make MW 1.37 tests pass on PHP 8.0 - https://phabricator.wikimedia.org/T292847 (10Reedy) [16:21:50] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T281167 (10brennen) 05Open→03Resolved At all wikis overnight, things seem pretty chill in logs. Optimistically calling this res... [16:26:31] 10Gerrit, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10ci-test-error (WMF-deployed Build Failure): Jenkins job failing intermittently due to Gerrit HTTP 502 errors when interacting with repos - https://phabricator.wikimedia.org/T246763 (10hashar) 05Resolved→03Open @z... [16:29:45] (03CR) 10Dduvall: Replace simple git server with Gerrit (033 comments) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 (owner: 10Dduvall) [16:29:59] (03PS17) 10Dduvall: Replace simple git server with Gerrit [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 [16:30:05] 10Gerrit, 10Release-Engineering-Team-TODO (2020-04 to 2020-06 (Q4)), 10Patch-For-Review, 10ci-test-error (WMF-deployed Build Failure): Jenkins job failing intermittently due to Gerrit HTTP 502 errors when interacting with repos - https://phabricator.wikimedia.org/T246763 (10hashar) 05Open→03Resolved Th... [16:30:12] 10Gerrit, 10Release-Engineering-Team: Gerrit timeout when cloning mediawiki/core - https://phabricator.wikimedia.org/T292858 (10hashar) [16:31:43] (03CR) 10Ahmon Dancy: Replace simple git server with Gerrit (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 (owner: 10Dduvall) [16:32:50] (03CR) 10Ahmon Dancy: [C: 03+2] Use the remote origin url for Gerrit/git heuristic [tools/release] - 10https://gerrit.wikimedia.org/r/727433 (owner: 10Dduvall) [16:33:03] (03CR) 10Dduvall: Replace simple git server with Gerrit (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 (owner: 10Dduvall) [16:34:03] (03Merged) 10jenkins-bot: Use the remote origin url for Gerrit/git heuristic [tools/release] - 10https://gerrit.wikimedia.org/r/727433 (owner: 10Dduvall) [16:36:49] 10Project-Admins: Create a generic #Codex tag - https://phabricator.wikimedia.org/T292859 (10LZaman) [16:37:34] (03CR) 10Dduvall: Replace simple git server with Gerrit (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 (owner: 10Dduvall) [16:38:41] 10Project-Admins: Create a generic #Codex tag - https://phabricator.wikimedia.org/T292859 (10Jdforrester-WMF) [16:38:43] 10Project-Admins: Create project tag for Codex - https://phabricator.wikimedia.org/T292565 (10Jdforrester-WMF) [16:38:46] 10Gerrit, 10Release-Engineering-Team, 10User-zeljkofilipin: Gerrit timeout when cloning mediawiki/core - https://phabricator.wikimedia.org/T292858 (10zeljkofilipin) [16:40:16] (03CR) 10Ahmon Dancy: Replace simple git server with Gerrit (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 (owner: 10Dduvall) [16:43:26] 10Gerrit, 10Release-Engineering-Team, 10User-zeljkofilipin: Gerrit timeout when cloning mediawiki/core - https://phabricator.wikimedia.org/T292858 (10hashar) I did a bit of grep in /var/log/gerrit/error_log and /var/log/apache2/gerrit.wikimedia.org.https.error.log. In chronological order: **gerrit** [2021-... [16:45:48] (03CR) 10Jforrester: dockerfiles: rebuild Bullseye images for security repos (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/724501 (owner: 10Hashar) [16:53:17] Project beta-scap-sync-world build #23092: 04FAILURE in 8 min 43 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/23092/ [16:54:09] hmm... rsync segfault [16:56:03] nice [16:56:15] many processes segfaulting on deployment-deploy01.deployment-prep.eqiad.wmflabs [16:58:43] 10Gerrit, 10Release-Engineering-Team, 10User-zeljkofilipin: Gerrit timeout when cloning mediawiki/core - https://phabricator.wikimedia.org/T292858 (10hashar) The corresponding errors: ` 16:12:01.825140 http.c:690 <= Recv header: remote: Counting objects: 8957, done remote: Finding sources: 100... [16:58:49] Project beta-scap-sync-world build #23093: 04STILL FAILING in 4 min 4 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/23093/ [17:00:09] 00:02:22.068 urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='deployment-logstash03.deployment-prep.eqiad1.wikimedia.cloud', port=9200): Max retries exceeded with url: /logstash-*/_search (Caused by NewConnectionError(': Failed to establish a new connection: [Errno -2] Name or service not known',)) [17:00:11] joy [17:00:15] DNS issue maybe [17:04:48] (03PS1) 10Ahmon Dancy: Avoid cascading exception handle_keyboard_interrupt() [tools/scap] - 10https://gerrit.wikimedia.org/r/728567 (https://phabricator.wikimedia.org/T292638) [17:05:05] (03PS2) 10Ahmon Dancy: Avoid cascading exception in handle_keyboard_interrupt() [tools/scap] - 10https://gerrit.wikimedia.org/r/728567 (https://phabricator.wikimedia.org/T292638) [17:08:36] 10Release-Engineering-Team, 10Scap, 10Patch-For-Review: Bad behavior when scap is control-c'd - https://phabricator.wikimedia.org/T292638 (10dancy) [17:13:12] o/ brennen, just wondering if you managed to try out the larger artifacts config? I might try and do another release this weekend (sorry for double ping, apparently I was in #wikimedia-dev before) [17:16:29] hey addshore - i haven't, but i can probably get to it today. [17:16:43] <3 [17:18:50] so hmm deployment-deploy01 has broken puppet since Sept 28th :( [17:20:14] scap tries to hit `deployment-logstash03.deployment-prep.eqiad1.wikimedia.cloud` which well doesn't exist in DNS apparently [17:20:25] as to why it starts breaking now ... no idea [17:22:07] (03CR) 10Kosta Harlan: [C: 03+1] "thanks!" [integration/quibble] - 10https://gerrit.wikimedia.org/r/727443 (https://phabricator.wikimedia.org/T291549) (owner: 10Kosta Harlan) [17:23:32] puppet lacks a value for v [17:23:33] "profile::kubernetes::deployment_server::user_defaults" [17:23:36] well that will be for later ;) [17:23:41] nod [17:23:42] Project beta-scap-sync-world build #23094: 04STILL FAILING in 19 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/23094/ [17:23:48] dinner time [17:24:05] (03CR) 10Hashar: [C: 03+2] phpbench: Run composer install first [integration/quibble] - 10https://gerrit.wikimedia.org/r/727443 (https://phabricator.wikimedia.org/T291549) (owner: 10Kosta Harlan) [17:25:26] (03PS6) 10Hashar: Release Quibble 1.1.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/728416 (owner: 10Kosta Harlan) [17:25:57] (03PS1) 10Hashar: changelog: begin new 1.1.2 version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/728576 [17:26:12] kostajh: rebased stuff, amended CHANGELOG.md :) [17:26:19] will cut the tag after dinner [17:26:52] (03CR) 10Hashar: [C: 03+2] Release Quibble 1.1.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/728416 (owner: 10Kosta Harlan) [17:27:02] (03CR) 10Hashar: [C: 03+2] changelog: begin new 1.1.2 version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/728576 (owner: 10Hashar) [17:30:09] 10Beta-Cluster-Infrastructure: Puppet errors on deployment-logstash03.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T286567 (10dancy) >>! In T286567#7214310, @Majavah wrote: > I think the easiest option here would be to just delete logstash03. It's running an old version of ELK (T283013) and... [17:42:49] (03Merged) 10jenkins-bot: phpbench: Run composer install first [integration/quibble] - 10https://gerrit.wikimedia.org/r/727443 (https://phabricator.wikimedia.org/T291549) (owner: 10Kosta Harlan) [17:44:21] Yippee, build fixed! [17:44:21] Project beta-scap-sync-world build #23095: 09FIXED in 18 min: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/23095/ [17:45:15] hashar: cheers! [17:45:43] Shall I make the jjb and docker build patches? [17:45:56] (03Merged) 10jenkins-bot: Release Quibble 1.1.1 [integration/quibble] - 10https://gerrit.wikimedia.org/r/728416 (owner: 10Kosta Harlan) [17:45:58] (03Merged) 10jenkins-bot: changelog: begin new 1.1.2 version cycle [integration/quibble] - 10https://gerrit.wikimedia.org/r/728576 (owner: 10Hashar) [18:05:43] 10Release-Engineering-Team (Done by Wed 06 Oct), 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10Legoktm) >>! In T291978#7406715, @dduvall wrote: > In other wo... [18:12:48] (03PS1) 10Ahmon Dancy: Add mediawiki/extensions/NearbyPages to mirror repos list [tools/train-dev] - 10https://gerrit.wikimedia.org/r/728583 [18:15:55] 10Release-Engineering-Team (Done by Wed 06 Oct), 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10mmodell) So the containers will be running in a VM environment... [18:21:30] 10Phabricator, 10Release-Engineering-Team (Done by Wed 06 Oct), 10GitLab-Test, 10GitLab (Integrations), 10User-brennen: Experiment with GitLab-Phabricator integration - https://phabricator.wikimedia.org/T265617 (10mmodell) So one thing I think that we need is the equivalent to the "gerrit patches" table... [18:41:59] hmm, my gitlab 2fa is broken [18:43:06] broken in a new and different way? [18:43:20] yeah, none of my codes, including recovery ones are working [18:43:37] im guessing i last successfully logged in in the past day or 2 [18:44:52] addshore: hrm, i think i am able to repro. [18:45:31] investigating. [18:46:35] feel free to ping me if you want me to try again [18:46:45] * addshore gets icecream and continues listening to conference talks [18:47:51] application.log shows failures but nothing super obvious as to _why_. jelto switched gitlab1001 over to puppet for config earlier today, so i'm guessing something changed along the way. [18:48:03] well that's weird, I just got logged out [18:48:15] (of gitlab) [18:48:56] (03PS2) 10Ahmon Dancy: Add mediawiki/extensions/NearbyPages to mirror repos list [tools/train-dev] - 10https://gerrit.wikimedia.org/r/728583 [18:48:58] (03PS1) 10Ahmon Dancy: Use Debian 10 (Buster) exclusively. Previously the deploy and www containers used Debian 9 (stretch). [tools/train-dev] - 10https://gerrit.wikimedia.org/r/728594 [18:49:03] was trying to load https://gitlab.wikimedia.org/help, but it kept redirecting to https://gitlab.wikimedia.org/, tried again and got logged out [18:49:24] hit sign in, page refreshed and I was signed in again and /help is still broken [18:50:01] yeah, at least one weird thing is happening here, and possibly 2. [18:50:19] (03CR) 10Ahmon Dancy: "Verified that this can complete a full ./train run in the deploy container." [tools/train-dev] - 10https://gerrit.wikimedia.org/r/728594 (owner: 10Ahmon Dancy) [18:50:19] looks like every time I try to load /help it redirects me through idp again [18:51:16] correction: any link. [18:54:46] it's possible that what's really happening is sessions are dying instantly. [18:55:20] might explain 2fa seeming not to work. [18:55:30] im just sat on the https://gitlab.wikimedia.org/users/auth/cas3/callback page [18:56:29] hrm, i was able to log in as https://gitlab.wikimedia.org/rando but i can't go anywhere else without getting redirected to / [18:56:59] this actually sounds like behavior i saw on the replica after it was rebuilt on puppet, so that gives me something to go on [18:57:27] yes, it looks like the _gitlab_session cookie changes every time [18:57:54] even loading / actually redirects through idp back to / [19:00:53] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10Dzahn) No concerns and nice work, Jelto. 👍 [19:01:55] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10brennen) Login to gitlab.wikimedia.org seems to be broken for 2fa users currently (recurring prompt... [19:02:18] i bet i know what this is. [19:02:53] brennen@gitlab1001:~$ sudo grep session_duration /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml [19:02:55] ## cas3-specific settings, specifically session_duration: [19:02:57] session_duration: 1 [19:03:12] does that say what I think it says [19:05:18] yeah, probably. the particulars are... whatever, not worth going into here, but a config value from the old ansible config trickled through in a way that didn't take effect before 14.3.2 or thereabouts. [19:05:27] i think. let's try changing the value and restarting, one sec. [19:05:51] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10AntiCompositeNumber) [19:07:08] !log gitlab1001: restarting to check correct session_duration value of 604800 [19:07:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:08:01] !log Tag quibble 1.1.1 @ b54af2aa60 [19:08:03] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:08:33] kostajh: dinner finished, I have tagged Quibble 1.1.1 [19:08:35] brennen: is codfw broken too? [19:09:32] RhinosF1: gitlab-replica.wikimedia.org? yeah, probably. [19:09:56] i think resetting the value works as i thought. submitting a puppet patch to fix. [19:10:09] (gitlab.wikimedia.org logins should work ok now) [19:10:42] Ye [19:11:18] works for me! [19:14:01] (03PS1) 10Hashar: dockerfiles: bump quibble to 1.1.1 [integration/config] - 10https://gerrit.wikimedia.org/r/728605 [19:14:04] hashar: shall I make the docker / jjb patch? [19:14:05] oh [19:14:30] :) [19:14:33] kostajh: ah yeah hmm sorry :D [19:14:40] there's a (presumably) train regression related to history pages https://phabricator.wikimedia.org/T292791 [19:15:48] (03CR) 10Kosta Harlan: [C: 03+1] dockerfiles: bump quibble to 1.1.1 [integration/config] - 10https://gerrit.wikimedia.org/r/728605 (owner: 10Hashar) [19:15:51] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T281167 (10Legoktm) There's something wrong with history page ordering: {T292791}. [19:16:19] 10Release-Engineering-Team (Doing), 10Patch-For-Review, 10Release, 10Train Deployments, 10User-brennen: 1.38.0-wmf.3 deployment blockers - https://phabricator.wikimedia.org/T281167 (10brennen) 05Resolved→03Open [19:16:26] 10Release-Engineering-Team (Radar), 10Infrastructure-Foundations, 10GitLab (Infrastructure), 10Patch-For-Review, and 3 others: Puppetise gitlab-ansible playbook - https://phabricator.wikimedia.org/T283076 (10brennen) I think this is the culprit: ` brennen@gitlab1001:~$ sudo grep session_duration /opt/gitl... [19:17:00] GitLab is working for me now too [19:17:19] (03Abandoned) 10Kosta Harlan: [DNM] Test run [integration/quibble] - 10https://gerrit.wikimedia.org/r/727444 (owner: 10Kosta Harlan) [19:18:01] hashar: I'll let you do the jjb one too then, but lmk if you want me to do it [19:18:23] I can do it don't worry ) [19:19:22] ./utils/docker-updates |grep quibble|wc -l [19:19:22] 26 [19:19:27] so much repetition [19:19:48] (03CR) 10Hashar: [C: 03+2] dockerfiles: bump quibble to 1.1.1 [integration/config] - 10https://gerrit.wikimedia.org/r/728605 (owner: 10Hashar) [19:20:02] Project beta-update-databases-eqiad build #53762: 04FAILURE in 1.2 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/53762/ [19:21:45] (03Merged) 10jenkins-bot: dockerfiles: bump quibble to 1.1.1 [integration/config] - 10https://gerrit.wikimedia.org/r/728605 (owner: 10Hashar) [19:23:07] !log Building CI Docker images for Quibble 1.1.1 [19:23:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:23:14] (03PS1) 10Hashar: jjb: switch jobs to Quibble 1.1.1 [integration/config] - 10https://gerrit.wikimedia.org/r/728620 [19:24:25] played "but who wrote that code" [19:24:35] typical answer: me a year ago :-\ [19:29:07] :D [19:38:28] (03PS18) 10Dduvall: Replace simple git server with Gerrit [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 [19:38:56] (03CR) 10jerkins-bot: [V: 04-1] Replace simple git server with Gerrit [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 (owner: 10Dduvall) [19:39:45] (03CR) 10Kosta Harlan: [C: 03+1] jjb: switch jobs to Quibble 1.1.1 [integration/config] - 10https://gerrit.wikimedia.org/r/728620 (owner: 10Hashar) [19:39:48] (03CR) 10Dduvall: Replace simple git server with Gerrit (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 (owner: 10Dduvall) [19:50:10] I have tried to use some emoji in a console message, but that did not play well [19:50:48] (03PS1) 10Hashar: utils: show comments in docker-updates [integration/config] - 10https://gerrit.wikimedia.org/r/728632 [19:51:48] (03PS19) 10Dduvall: Replace simple git server with Gerrit [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 [19:59:05] (03CR) 10Hashar: "Images build and published" [integration/config] - 10https://gerrit.wikimedia.org/r/728605 (owner: 10Hashar) [20:00:19] !log Updating Jenkins jobs for Quibble 1.1.1 # https://gerrit.wikimedia.org/r/c/integration/config/+/728620 [20:00:21] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:00:31] kostajh: deploying the jobs ;) [20:00:51] (03CR) 10Hashar: [C: 03+2] "Jobs are being deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/728620 (owner: 10Hashar) [20:02:44] (03Merged) 10jenkins-bot: jjb: switch jobs to Quibble 1.1.1 [integration/config] - 10https://gerrit.wikimedia.org/r/728620 (owner: 10Hashar) [20:05:17] kostajh: done! [20:08:59] 10Release-Engineering-Team (Radar), 10GitLab (Auth & Access), 10Patch-For-Review, 10Upstream, 10User-brennen: Increase GitLab session lifetime to something reasonable - https://phabricator.wikimedia.org/T288757 (10brennen) 05Open→03Resolved a:03brennen [20:09:25] 10Release-Engineering-Team (Radar), 10GitLab (Auth & Access), 10Patch-For-Review, 10Upstream, 10User-brennen: Increase GitLab session lifetime to something reasonable - https://phabricator.wikimedia.org/T288757 (10brennen) [20:13:59] kostajh: funny I went to recheck the GrowthExperiments patch a minute after you ;) [20:15:30] taking a break, will check again in half an hour or so [20:20:02] Project beta-update-databases-eqiad build #53763: 04STILL FAILING in 1.6 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/53763/ [20:37:09] (03CR) 10Jforrester: [C: 03+2] "Neat." [integration/config] - 10https://gerrit.wikimedia.org/r/728632 (owner: 10Hashar) [20:38:33] James_F: i might improve µ§%Ml§: [20:38:50] I might improve that script to ease the images update in jjb config files [20:40:13] (03Merged) 10jenkins-bot: utils: show comments in docker-updates [integration/config] - 10https://gerrit.wikimedia.org/r/728632 (owner: 10Hashar) [20:40:58] hashar: Ha, sorry, should I not have merged? It is already better than what came before. [20:41:49] James_F: that one was ready for review & approval indeed [20:42:00] I can add the editing easer later on [20:42:21] Cool. [20:50:28] (03CR) 10Ahmon Dancy: "Nice improvements in the latest patchsets" [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 (owner: 10Dduvall) [20:51:48] (03PS1) 10Hashar: jjb: do not run phpbench in wmf-quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/728651 [20:52:02] and we need a command to craft the list of affected jobs [20:53:07] (03PS2) 10Hashar: jjb: do not run phpbench in wmf-quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/728651 [20:53:54] (03CR) 10Hashar: "I don't think we need to run phpbench in the wmf-quibble jobs which does integration tests of multiple extensions + core." [integration/config] - 10https://gerrit.wikimedia.org/r/728651 (owner: 10Hashar) [20:54:16] that is all for this week :] happy hackings [21:11:59] >/rxjs/operators/auditTime.js.map' [21:11:59] 22:02:20 npm WARN tar TAR_ENTRY_ERROR ENOSPC: no space left on device, open '/workspace/src/skins/MinervaNeue/node_modules/core-js-pure/es/string/repeat.js' [21:12:01] Fun times [21:16:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Doing), 10Zuul: Migrate Zuul alerting to Grafana / AlertManager - https://phabricator.wikimedia.org/T292284 (10hashar) 05Resolved→03Open AlertManager hasn't issued a notification on IRC. Maybe it needs an extra configuration to join our... [21:17:33] !log Purging Docker images on all CI agents [21:17:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:17:40] Reedy: too many images on the CI agents I guess [21:20:04] Project beta-update-databases-eqiad build #53764: 04STILL FAILING in 4 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/53764/ [21:43:47] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.38.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T281168 (10brennen) [21:46:56] Can someone kick beta [21:47:03] It was moaning about composer [21:49:39] Gotta wait for a patch to merge [21:50:00] Which Reedy [21:50:10] https://gerrit.wikimedia.org/r/728652 [21:51:09] Oh [21:58:57] next full run should be good [22:10:38] 10Release-Engineering-Team (Done by Wed 06 Oct), 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10dduvall) >>! In T291978#7413043, @Legoktm wrote: >>>! In T2919... [22:12:25] 10Release-Engineering-Team (Done by Wed 06 Oct), 10Security-Team, 10GitLab (CI & Job Runners), 10Patch-For-Review, and 2 others: Limit GitLab shared runners to images from Wikimedia Docker registry - https://phabricator.wikimedia.org/T291978 (10dduvall) >>! In T291978#7413065, @mmodell wrote: > So the cont... [22:21:32] Yippee, build fixed! [22:21:32] Project beta-update-databases-eqiad build #53765: 09FIXED in 1 min 31 sec: https://integration.wikimedia.org/ci/job/beta-update-databases-eqiad/53765/ [22:22:27] tada [22:33:11] (03PS20) 10Dduvall: Replace simple git server with Gerrit [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 [22:33:46] (03PS21) 10Dduvall: Replace simple git server with Gerrit [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 [22:33:52] (03CR) 10Dduvall: Replace simple git server with Gerrit (031 comment) [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 (owner: 10Dduvall) [22:35:13] (03CR) 10Dduvall: "I was able to successfully run `./train` without issue against this latest patchset. \o/" [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 (owner: 10Dduvall) [22:52:34] (03PS22) 10Dduvall: Replace simple git server with Gerrit [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 [22:53:56] (03CR) 10Dduvall: [C: 03+2] Use Debian 10 (Buster) exclusively. Previously the deploy and www containers used Debian 9 (stretch). [tools/train-dev] - 10https://gerrit.wikimedia.org/r/728594 (owner: 10Ahmon Dancy) [22:54:06] (03CR) 10Dduvall: [C: 03+2] Add mediawiki/extensions/NearbyPages to mirror repos list [tools/train-dev] - 10https://gerrit.wikimedia.org/r/728583 (owner: 10Ahmon Dancy) [22:56:12] (03Merged) 10jenkins-bot: Add mediawiki/extensions/NearbyPages to mirror repos list [tools/train-dev] - 10https://gerrit.wikimedia.org/r/728583 (owner: 10Ahmon Dancy) [22:56:14] (03Merged) 10jenkins-bot: Use Debian 10 (Buster) exclusively. Previously the deploy and www containers used Debian 9 (stretch). [tools/train-dev] - 10https://gerrit.wikimedia.org/r/728594 (owner: 10Ahmon Dancy) [22:57:55] (03PS23) 10Dduvall: Replace simple git server with Gerrit [tools/train-dev] - 10https://gerrit.wikimedia.org/r/726714 [23:56:14] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [23:58:49] it's just cwhite spamming patches ^ [23:58:55] :)