[08:05:43] 10Diffusion, 10Tool-spacemedia, 10Toolforge, 10Wikimedia-GitHub, 10User-bd808: Mirroring tool-spacemedia Diffusion repository to GitHub seems to be broken - https://phabricator.wikimedia.org/T284144 (10Don-vip) [08:40:37] beta cluster isn’t loading for me (times out after [I think] one minute with “Error: 502, Next Hop Connection Failed”), is there a known issue? [08:42:26] (SSHing into deployment-mwmaint01 works) [08:43:45] what about deployment-mediawiki11 [08:43:57] * Lucas_WMDE tries [08:44:17] seems to be reachable too [08:44:42] load average looks reasonable (0.3 range) [08:45:02] i'm on my phone so can't look myself [08:45:23] it sounds like the answer to my question is “not a known issue yet” ^^ [08:45:27] I can open a phab task [08:47:20] correct [08:48:22] huh, it works from the command line [08:48:30] oh, and in a private window too [08:49:07] (or at least, the private window loaded as much as the , though it’s not showing page contents yet) [08:55:23] <wikibugs> 10Beta-Cluster-Infrastructure: Beta cluster unreachable when interacting with sessions (Error: 502, Next Hop Connection Failed) - https://phabricator.wikimedia.org/T284279 (10Lucas_Werkmeister_WMDE) [08:55:37] <Lucas_WMDE> created ^ (feel free to retitle it, not sure if that title is good) [08:56:16] <wikibugs> 10Beta-Cluster-Infrastructure: Beta cluster unreachable when interacting with sessions (Error: 502, Next Hop Connection Failed) - https://phabricator.wikimedia.org/T284279 (10Lucas_Werkmeister_WMDE) [09:00:36] <wikibugs> 10Release-Engineering-Team (Doing), 10Gerrit (Gerrit 3.3), 10Patch-For-Review: Switch Gerrit from Java 8 to Java 11 - https://phabricator.wikimedia.org/T268225 (10hashar) We will remove the Java 8 packages on Monday (June 7th) [09:19:38] <wikibugs> 10Release-Engineering-Team-TODO, 10DNS, 10SRE, 10Traffic, and 3 others: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10jbond) [09:19:48] <wikibugs> 10Release-Engineering-Team (Doing), 10GitLab (Initialization), 10User-brennen: Remove Speed & Function blockers for GitLab work - https://phabricator.wikimedia.org/T274458 (10jbond) [09:19:52] <wikibugs> 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10Traffic, 10GitLab (Initialization), and 2 others: open firewall ports on gitlab1001.wikimedia.org (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10jbond) a:05Dzahn→03jbond The SSH port has... [09:20:19] <wikibugs> 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10Traffic, 10GitLab (Initialization), and 2 others: open firewall ports on gitlab1001.wikimedia.org (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10jbond) 05Resolved→03Open a:05jbond→03Dz... [09:52:16] <wikibugs> 10Phabricator: RuntimeException: "Undefined variable: url" at https://phabricator.wikimedia.org/u/0 - https://phabricator.wikimedia.org/T284059 (10IN) >>! In T284059#7125745, @Aklapper wrote: > Hi, how and why did you find and enter that URL? Why would you expect broken URLs not to display an error? This task la... [09:52:24] <wikibugs> 10Phabricator: RuntimeException: "Undefined variable: url" at https://phabricator.wikimedia.org/u/0 - https://phabricator.wikimedia.org/T284059 (10IN) >>! In T284059#7125745, @Aklapper wrote: > Hi, how and why did you find and enter that URL? Why would you expect broken URLs not to display an error? This task la... [10:02:44] <wikibugs> (03PS11) 10Hashar: [WMF] register our plugins as submodules [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684336 [10:02:50] <wikibugs> (03CR) 10Hashar: [WMF] register our plugins as submodules (031 comment) [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684336 (owner: 10Hashar) [10:36:06] <wikibugs> 10Continuous-Integration-Config: There should be a CI check for files differing only by case - https://phabricator.wikimedia.org/T204824 (10hashar) 05Open→03Declined On a case insensitive filesystems, git would set `core.ignorecase = true` when doing a clone/init. That does add some workarounds to make it w... [10:37:35] <wikibugs> 10Continuous-Integration-Config, 10Wikidata: SlowTimer for two PHPunit tests, possibly from Wikibase - https://phabricator.wikimedia.org/T211035 (10hashar) 05Open→03Resolved a:03hashar Closing as probably obsolete. It might have been related to the underlying MySQL database being too slow at the time for... [10:41:34] <wikibugs> 10Continuous-Integration-Config, 10Patch-For-Review: Replace all the npm* images with node10* ones - https://phabricator.wikimedia.org/T207087 (10hashar) 05Open→03Resolved a:03Jdforrester-WMF apparently the last one was mathoid which has been migrated to Pipeline via T261809 And the image have been all... [10:51:23] <wikibugs> 10Continuous-Integration-Config, 10Machine-Learning-Team, 10ORES: ORES wheels deployment repo Jenkins job should gate-and-submit - https://phabricator.wikimedia.org/T211041 (10hashar) 05Open→03Declined My understanding is that research/ores/wheels is abandoned or scheduled to be replaced entirely. So I d... [10:59:00] <wikibugs> 10Continuous-Integration-Config, 10Utilities-mwdumper: Build mwdumper automatically - https://phabricator.wikimedia.org/T213874 (10hashar) 05Open→03Declined mwdumper is not that much maintained. One can just use mvn to build it. Maybe we will eventually add some part to publish to Maven Central when a t... [11:03:42] <wikibugs> 10Release-Engineering-Team (Seen), 10Release Pipeline: The pipeline should provide a way to save artifacts from a stage - https://phabricator.wikimedia.org/T216272 (10hashar) [11:03:51] <wikibugs> 10Release-Engineering-Team (Seen), 10Release Pipeline: Pipeline should clone source to a subdirectory of the Jenkins workspace - https://phabricator.wikimedia.org/T216274 (10hashar) [11:10:12] <wikibugs> 10Release-Engineering-Team (Seen), 10Wikimedia-Site-requests, 10serviceops-radar: Consider creating a puppet-compiler equivalent for mediawiki-config.git - https://phabricator.wikimedia.org/T220775 (10hashar) [11:40:10] <icinga-wm> PROBLEM - SSH on contint2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [12:00:41] <wikibugs> (03PS1) 10Hashar: coverage: add link to clover.xml [integration/docroot] - 10https://gerrit.wikimedia.org/r/698180 (https://phabricator.wikimedia.org/T221510) [12:02:18] <wikibugs> (03CR) 10Hashar: "An apparently low hanging fruit that is tentatively easy to solve. I am not familiar with flex layout unfortunately, I have reorder the " [integration/docroot] - 10https://gerrit.wikimedia.org/r/698180 (https://phabricator.wikimedia.org/T221510) (owner: 10Hashar) [12:03:21] <wikibugs> 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10Zuul: Run testsuite when packaging integration/zuul - https://phabricator.wikimedia.org/T222952 (10hashar) 05Open→03Declined [12:03:40] <wikibugs> 10Continuous-Integration-Config, 10Tool-extjsonuploader, 10Patch-For-Review, 10Test-Coverage: Provide list of repos with coverage information in machine-readable format - https://phabricator.wikimedia.org/T221510 (10hashar) a:03hashar Two years later, it seems it is all about adding a link to the `clover... [12:05:51] <wikibugs> 10Phabricator: RuntimeException: "Undefined variable: url" at https://phabricator.wikimedia.org/u/0 - https://phabricator.wikimedia.org/T284059 (10Aklapper) So there is no use case why to fix this, I guess. [12:06:03] <wikibugs> 10Phabricator: RuntimeException: "Undefined variable: url" at https://phabricator.wikimedia.org/u/0 - https://phabricator.wikimedia.org/T284059 (10Aklapper) 05Stalled→03Declined [12:30:35] <wikibugs> 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen), 10Epic: Tracking task to support running arbitrary Docker images - https://phabricator.wikimedia.org/T283724 (10hashar) [12:30:38] <wikibugs> 10Continuous-Integration-Config, 10SRE: operations/docker-images/production-images has no CI - https://phabricator.wikimedia.org/T283855 (10hashar) [12:32:53] <wikibugs> 10Gerrit, 10Diffusion-Repository-Administrators, 10User-Kizule: Unarchive EditAccount repository on Gerrit - https://phabricator.wikimedia.org/T283935 (10Aklapper) Should be fixed now; I edited it at https://github.com/wikimedia/mediawiki-extensions-EditAccount/settings/branches [12:33:46] <wikibugs> 10Continuous-Integration-Config, 10Machine-Learning-Team: CI should check to see if our wheels are good - https://phabricator.wikimedia.org/T250746 (10hashar) 05Open→03Declined My understand is ORES is being phased out, does not seem we need to invest in adding CI for `research/ores/wheels` [12:34:17] <wikibugs> 10LibUp, 10Front-end-Standards-Group, 10MW-1.35-notes (1.35.0-wmf.30; 2020-04-28): Move eslint ignores out of Gruntfile.js or package.json and into .eslintignore for all repos - https://phabricator.wikimedia.org/T249858 (10hashar) [12:38:02] <wikibugs> 10Continuous-Integration-Config: Provide a PHP code coverage job for mediawiki/libs/* (non-MW context) - https://phabricator.wikimedia.org/T252120 (10hashar) 05Open→03Resolved That apparently has been fixed at some point. An example config would be: ` name=zuul/layout.yaml - name: mediawiki/libs/alea p... [12:39:12] <wikibugs> 10Continuous-Integration-Config, 10Release-Engineering-Team (Seen): CI should run composer test/npm test BEFORE running Quibble - https://phabricator.wikimedia.org/T211703 (10hashar) 05Open→03Declined Maybe later? [12:39:48] <wikibugs> 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing), 10observability: Extract mtail puppet.git tests to a standalone CI container - https://phabricator.wikimedia.org/T255534 (10hashar) 05Open→03Declined Declining since `mtail` is rather small per Filippo [12:41:58] <wikibugs> 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Radar), 10Patch-For-Review, 10Upstream: Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597 (1... [12:42:11] <wikibugs> 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Release-Engineering-Team (Seen): Use cron instead of Jenkins for beta deployments - https://phabricator.wikimedia.org/T188367 (10hashar) 05Open→03Declined Declining the cron job idea in favor of a dedicated infra T256168 [12:42:27] <wikibugs> 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Release-Engineering-Team (Doing): Move beta cluster automatic deployment to a dedicated infrastructure - https://phabricator.wikimedia.org/T256168 (10hashar) [12:42:29] <wikibugs> 10Continuous-Integration-Infrastructure, 10Jenkins, 10Release-Engineering-Team (Radar), 10Patch-For-Review, 10Upstream: Jenkins Gearman plugin has deadlock on executor threads (was: Beta Cluster stopped receiving code updates (beta-update-databases-eqiad hung) - https://phabricator.wikimedia.org/T72597 (1... [12:44:13] <wikibugs> (03PS1) 10Arturo Borrero Gonzalez: zuul/layout.yaml: add configuration for cloud/toolforge/jobs-framework-api [integration/config] - 10https://gerrit.wikimedia.org/r/698188 (https://phabricator.wikimedia.org/T283238) [12:46:08] <wikibugs> (03CR) 10Hashar: [C: 03+2] zuul/layout.yaml: add configuration for cloud/toolforge/jobs-framework-api [integration/config] - 10https://gerrit.wikimedia.org/r/698188 (https://phabricator.wikimedia.org/T283238) (owner: 10Arturo Borrero Gonzalez) [12:47:20] <wikibugs> (03Merged) 10jenkins-bot: zuul/layout.yaml: add configuration for cloud/toolforge/jobs-framework-api [integration/config] - 10https://gerrit.wikimedia.org/r/698188 (https://phabricator.wikimedia.org/T283238) (owner: 10Arturo Borrero Gonzalez) [13:59:49] <wikibugs> 10Gerrit, 10Diffusion-Repository-Administrators, 10User-Kizule: Unarchive EditAccount repository on Gerrit - https://phabricator.wikimedia.org/T283935 (10Kizule) >>! In T283935#7134185, @Aklapper wrote: > Should be fixed now; I edited it at https://github.com/wikimedia/mediawiki-extensions-EditAccount/settin... [14:24:29] <wikibugs> 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10karapayneWMDE) Hi! As a new Engineering Manager at Wikimedia Germany, I am asked to administrate projects, sub-projects and milestones for Fundraising Tech as well as Wikidata and he... [14:27:34] <wikibugs> 10Phabricator: Add new WMDE Engingeer Manager to Triagers group - https://phabricator.wikimedia.org/T284306 (10karapayneWMDE) [14:31:10] <wikibugs> 10Phabricator, 10User-Urbanecm: Add new WMDE Engingeer Manager to Triagers group - https://phabricator.wikimedia.org/T284306 (10Urbanecm) 05Open→03Resolved a:03Urbanecm Reasonable request, approved. You should now be in the #triagers group. [14:34:36] <wikibugs> 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Ladsgroup) >>! In T706#7134437, @karapayneWMDE wrote: > Hi! As a new Engineering Manager at Wikimedia Germany, I am asked to administrate projects, sub-projects and milestones for Fu... [14:43:40] <wikibugs> 10Phabricator, 10User-Urbanecm: Add new WMDE Engingeer Manager to Triagers group - https://phabricator.wikimedia.org/T284306 (10karapayneWMDE) group access confirmed. thanks! [15:16:37] <wikibugs> (03CR) 10Ahmon Dancy: [C: 03+1] [WMF] register our plugins as submodules [software/gerrit] (wmf/stable-3.2) - 10https://gerrit.wikimedia.org/r/684336 (owner: 10Hashar) [15:43:13] <icinga-wm> RECOVERY - SSH on contint2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [18:05:13] <wikibugs> 10Release-Engineering-Team (Next), 10GitLab (Initialization), 10User-brennen: Create a GitLab settings script / repo - https://phabricator.wikimedia.org/T284336 (10brennen) [18:16:29] <wikibugs> 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Seen): Create WMF CI image for Node.js 14 - https://phabricator.wikimedia.org/T267888 (10Catrope) [18:18:33] <wikibugs> 10Phabricator, 10Release-Engineering-Team (Seen), 10User-Matthewrbowker: Phacility (Maintainer of Phabricator) is winding down. Upstream support ending. - https://phabricator.wikimedia.org/T283980 (10mmodell) >>! In T283980#7130885, @dbarratt wrote: > I wonder if there is another organization (Linux Founda... [18:20:52] <wikibugs> 10Release-Engineering-Team (Doing), 10SRE, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10Dzahn) per T276144#7133694 the SSH port is open to the world now. So that means this must be done! Let's close it ? [18:22:47] <wikibugs> 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10Traffic, 10GitLab (Initialization), and 2 others: open firewall ports on gitlab1001.wikimedia.org (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10Dzahn) Thanks! I think the SSH part was T2761... [18:24:22] <wikibugs> 10Release-Engineering-Team-TODO, 10DNS, 10SRE, 10Traffic, and 3 others: DNS for GitLab - https://phabricator.wikimedia.org/T276170 (10Dzahn) [18:24:32] <wikibugs> 10Release-Engineering-Team (Doing), 10GitLab (Initialization), 10User-brennen: Remove Speed & Function blockers for GitLab work - https://phabricator.wikimedia.org/T274458 (10Dzahn) [18:25:02] <wikibugs> 10Release-Engineering-Team-TODO (2021-01-01 to 2021-03-31 (Q3)), 10SRE, 10Traffic, 10GitLab (Initialization), and 2 others: open firewall ports on gitlab1001.wikimedia.org (was: Port map of how Gitlab is accessed) - https://phabricator.wikimedia.org/T276144 (10Dzahn) 05Open→03Resolved ` ACCEPT tcp... [18:26:03] <wikibugs> 10Release-Engineering-Team (Doing), 10GitLab (Initialization), 10User-brennen: Remove Speed & Function blockers for GitLab work - https://phabricator.wikimedia.org/T274458 (10Dzahn) 05Open→03Resolved a:03Dzahn boldly closing this since there obviously aren't blockers anymore or contractors wouldn't hav... [18:26:45] <wikibugs> 10Phabricator, 10Release-Engineering-Team (Seen), 10User-Matthewrbowker: Phacility (Maintainer of Phabricator) is winding down. Upstream support ending. - https://phabricator.wikimedia.org/T283980 (10mmodell) >>! In T283980#7131487, @Bugreporter wrote: > I don't think Wikimedia have enough resource to mainta... [18:27:23] <wikibugs> 10Release-Engineering-Team (Doing), 10GitLab (Initialization), 10User-brennen: Remove Speed & Function blockers for GitLab work - https://phabricator.wikimedia.org/T274458 (10Dzahn) [18:28:20] <wikibugs> 10Release-Engineering-Team (Doing), 10SRE, 10GitLab (Initialization), 10Patch-For-Review, 10User-brennen: SSH Access of Git data in GitLab - https://phabricator.wikimedia.org/T276148 (10Dzahn) Sure you want to open ssh to the public before backups and logging tasks are done? [18:53:41] <wikibugs> 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Create WMF CI image for Node.js 12 - https://phabricator.wikimedia.org/T284343 (10Jdforrester-WMF) [18:54:35] <wikibugs> (03CR) 10Jforrester: "> Patch Set 3:" [integration/config] - 10https://gerrit.wikimedia.org/r/695657 (owner: 10Jforrester) [18:55:53] <wikibugs> 10Continuous-Integration-Config, 10Release-Engineering-Team: Upgrade all CI jobs for WMF-deployed projects from Node 10 to Node 12 - https://phabricator.wikimedia.org/T284345 (10Jdforrester-WMF) [18:56:21] <wikibugs> 10Continuous-Integration-Config, 10Release-Engineering-Team (Seen), 10User-zeljkofilipin: Upgrade all CI jobs for WMF-deployed projects from Node 12 to Node 14 LTS - https://phabricator.wikimedia.org/T267890 (10Jdforrester-WMF) [18:56:27] <wikibugs> 10Continuous-Integration-Config, 10Release-Engineering-Team: Upgrade all CI jobs for WMF-deployed projects from Node 10 to Node 12 - https://phabricator.wikimedia.org/T284345 (10Jdforrester-WMF) [18:56:29] <wikibugs> 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team: Create WMF CI image for Node.js 12 - https://phabricator.wikimedia.org/T284343 (10Jdforrester-WMF) [18:56:34] <wikibugs> (03PS4) 10Jforrester: dockerfiles: Provide node12* tree based on ci-bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/695657 (https://phabricator.wikimedia.org/T284343) [19:13:25] <wikibugs> 10Release-Engineering-Team (Doing), 10GitLab (Initialization), 10User-brennen: Early adoption signup for WMF GitLab - https://phabricator.wikimedia.org/T282842 (10LucasWerkmeister) If the TranslateWiki.net people are okay with it, I’d be interested in moving the source code of the [Wikidata Lexeme Forms tool... [19:14:34] <wikibugs> (03PS5) 10Jforrester: dockerfiles: Provide node12* tree based on ci-bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/695657 (https://phabricator.wikimedia.org/T284343) [19:14:47] <wikibugs> 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: Create WMF CI image for Node.js 12 - https://phabricator.wikimedia.org/T284343 (10Jdforrester-WMF) [19:14:52] <wikibugs> (03CR) 10Jforrester: [C: 03+2] dockerfiles: Provide node12* tree based on ci-bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/695657 (https://phabricator.wikimedia.org/T284343) (owner: 10Jforrester) [19:16:17] <wikibugs> (03Merged) 10jenkins-bot: dockerfiles: Provide node12* tree based on ci-bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/695657 (https://phabricator.wikimedia.org/T284343) (owner: 10Jforrester) [19:16:46] <wikibugs> 10Continuous-Integration-Config, 10Patch-For-Review: Add CI for mediawiki/extensions/WikibaseStatementUpdater - https://phabricator.wikimedia.org/T270369 (10Jdforrester-WMF) a:05Jdforrester-WMF→03None [19:20:00] <James_F> !log Docker: Publishing node12 CI images T284343 [19:20:02] <stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:20:02] <stashbot> T284343: Create WMF CI image for Node.js 12 - https://phabricator.wikimedia.org/T284343 [19:35:42] <wikibugs> 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: Create WMF CI image for Node.js 12 - https://phabricator.wikimedia.org/T284343 (10Jdforrester-WMF) 05Open→03Resolved [19:35:44] <wikibugs> 10Continuous-Integration-Config, 10Release-Engineering-Team: Upgrade all CI jobs for WMF-deployed projects from Node 10 to Node 12 - https://phabricator.wikimedia.org/T284345 (10Jdforrester-WMF) [19:45:53] <icinga-wm> PROBLEM - SSH on contint2001.mgmt is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [19:50:56] <wikibugs> (03PS1) 10Jforrester: jjb: Provide node12 versions of existing node10 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/698244 (https://phabricator.wikimedia.org/T284345) [19:50:59] <wikibugs> (03PS1) 10Jforrester: Zuul: [VisualEditor/VisualEditor] Switch node10 jobs to node12 [integration/config] - 10https://gerrit.wikimedia.org/r/698245 (https://phabricator.wikimedia.org/T284345) [19:51:01] <wikibugs> (03PS1) 10Jforrester: Zuul: Migrate all node10 jobs to node12 in bulk [integration/config] - 10https://gerrit.wikimedia.org/r/698246 (https://phabricator.wikimedia.org/T284345) [19:51:03] <wikibugs> (03PS1) 10Jforrester: jjb: Drop unused node10 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/698247 (https://phabricator.wikimedia.org/T284345) [19:51:05] <wikibugs> (03PS1) 10Jforrester: jjb: Switch special jobs from node10 to node12 images [integration/config] - 10https://gerrit.wikimedia.org/r/698248 (https://phabricator.wikimedia.org/T284345) [19:52:12] <wikibugs> (03CR) 10jerkins-bot: [V: 04-1] Zuul: Migrate all node10 jobs to node12 in bulk [integration/config] - 10https://gerrit.wikimedia.org/r/698246 (https://phabricator.wikimedia.org/T284345) (owner: 10Jforrester) [19:52:20] <wikibugs> (03CR) 10jerkins-bot: [V: 04-1] jjb: Drop unused node10 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/698247 (https://phabricator.wikimedia.org/T284345) (owner: 10Jforrester) [19:52:22] <wikibugs> (03CR) 10jerkins-bot: [V: 04-1] jjb: Switch special jobs from node10 to node12 images [integration/config] - 10https://gerrit.wikimedia.org/r/698248 (https://phabricator.wikimedia.org/T284345) (owner: 10Jforrester) [19:54:57] <wikibugs> (03PS2) 10Jforrester: Zuul: Migrate all node10 jobs to node12 in bulk [integration/config] - 10https://gerrit.wikimedia.org/r/698246 (https://phabricator.wikimedia.org/T284345) [19:54:59] <wikibugs> (03PS2) 10Jforrester: jjb: Drop unused node10 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/698247 (https://phabricator.wikimedia.org/T284345) [19:55:01] <wikibugs> (03PS2) 10Jforrester: jjb: Switch special jobs from node10 to node12 images [integration/config] - 10https://gerrit.wikimedia.org/r/698248 (https://phabricator.wikimedia.org/T284345) [19:55:55] <wikibugs> (03CR) 10Jforrester: [C: 03+2] jjb: Provide node12 versions of existing node10 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/698244 (https://phabricator.wikimedia.org/T284345) (owner: 10Jforrester) [19:55:58] <wikibugs> (03CR) 10Jforrester: [C: 03+2] Zuul: [VisualEditor/VisualEditor] Switch node10 jobs to node12 [integration/config] - 10https://gerrit.wikimedia.org/r/698245 (https://phabricator.wikimedia.org/T284345) (owner: 10Jforrester) [19:56:18] <wikibugs> (03CR) 10jerkins-bot: [V: 04-1] Zuul: Migrate all node10 jobs to node12 in bulk [integration/config] - 10https://gerrit.wikimedia.org/r/698246 (https://phabricator.wikimedia.org/T284345) (owner: 10Jforrester) [19:56:20] <wikibugs> (03CR) 10jerkins-bot: [V: 04-1] jjb: Drop unused node10 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/698247 (https://phabricator.wikimedia.org/T284345) (owner: 10Jforrester) [19:56:34] <wikibugs> (03CR) 10jerkins-bot: [V: 04-1] jjb: Switch special jobs from node10 to node12 images [integration/config] - 10https://gerrit.wikimedia.org/r/698248 (https://phabricator.wikimedia.org/T284345) (owner: 10Jforrester) [19:57:03] <wikibugs> (03Merged) 10jenkins-bot: jjb: Provide node12 versions of existing node10 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/698244 (https://phabricator.wikimedia.org/T284345) (owner: 10Jforrester) [19:57:07] <wikibugs> (03Merged) 10jenkins-bot: Zuul: [VisualEditor/VisualEditor] Switch node10 jobs to node12 [integration/config] - 10https://gerrit.wikimedia.org/r/698245 (https://phabricator.wikimedia.org/T284345) (owner: 10Jforrester) [20:07:25] <wikibugs> (03PS3) 10Jforrester: Zuul: Migrate all node10 jobs to node12 in bulk [integration/config] - 10https://gerrit.wikimedia.org/r/698246 (https://phabricator.wikimedia.org/T284345) [20:11:10] <wikibugs> (03CR) 10Jforrester: [C: 03+2] Zuul: Migrate all node10 jobs to node12 in bulk [integration/config] - 10https://gerrit.wikimedia.org/r/698246 (https://phabricator.wikimedia.org/T284345) (owner: 10Jforrester) [20:11:44] <James_F> !log Zuul: [VisualEditor/VisualEditor] Switch node10 jobs to node12 T284345 [20:11:46] <stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:11:46] <stashbot> T284345: Upgrade all CI jobs for WMF-deployed projects from Node 10 to Node 12 - https://phabricator.wikimedia.org/T284345 [20:12:16] <wikibugs> (03Merged) 10jenkins-bot: Zuul: Migrate all node10 jobs to node12 in bulk [integration/config] - 10https://gerrit.wikimedia.org/r/698246 (https://phabricator.wikimedia.org/T284345) (owner: 10Jforrester) [20:13:55] <James_F> !log Zuul: Switch almost all node10 jobs to node12 T284345 [20:13:58] <stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:46:38] <icinga-wm> RECOVERY - SSH on contint2001.mgmt is OK: SSH OK - OpenSSH_6.6 (protocol 2.0) https://wikitech.wikimedia.org/wiki/Dc-operations/Hardware_Troubleshooting_Runbook [20:56:07] <wikibugs> (03PS3) 10Jforrester: jjb: Drop unused node10 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/698247 (https://phabricator.wikimedia.org/T284345) [20:58:18] <wikibugs> 10Phabricator, 10Developer-Advocacy (Jul-Sep 2021), 10PM: Decrease issues created many years ago with no recent activity (aka stale tickets) - https://phabricator.wikimedia.org/T248034 (10stjn) One problem that I can see from the proposal on top is that, when encountered, this would definitely lead to more c... [21:02:29] <wikibugs> (03CR) 10Jforrester: [C: 03+2] jjb: Drop unused node10 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/698247 (https://phabricator.wikimedia.org/T284345) (owner: 10Jforrester) [21:03:41] <wikibugs> (03Merged) 10jenkins-bot: jjb: Drop unused node10 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/698247 (https://phabricator.wikimedia.org/T284345) (owner: 10Jforrester) [21:09:07] <Krinkle> James_F: that.. was quick? [21:09:14] <wikibugs> (03PS3) 10Jforrester: jjb: Switch special jobs from node10 to node12 images [integration/config] - 10https://gerrit.wikimedia.org/r/698248 (https://phabricator.wikimedia.org/T284345) [21:09:16] <Krinkle> everything green then? [21:09:36] <James_F> Krinkle: Yes, across a couple of dozen tested repos. [21:09:41] <Krinkle> awesome [21:09:45] <James_F> Krinkle: As expected, honestly. [21:09:54] <Krinkle> yeah, it's not a huge difference in terms of node compat [21:09:58] <Krinkle> was more worried about environment stuff [21:09:59] <James_F> The npm world is a lot more active than typical debian things. [21:10:08] <Krinkle> it's a difference debian and chrome/firefox package right? [21:10:15] <James_F> Yeah, porting kartotherian is a pain. [21:10:26] <James_F> Chrome and Firefox are newer, but not much different otherwise. [21:10:46] <James_F> (Except the back-compat. for the chromedriver package was removed in favour of chromium-driver only.) [21:19:13] <wikibugs> (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/698248 (https://phabricator.wikimedia.org/T284345) (owner: 10Jforrester) [21:20:33] <wikibugs> (03Merged) 10jenkins-bot: jjb: Switch special jobs from node10 to node12 images [integration/config] - 10https://gerrit.wikimedia.org/r/698248 (https://phabricator.wikimedia.org/T284345) (owner: 10Jforrester) [21:36:30] <wikibugs> (03PS1) 10Jforrester: [WIP] dockerfiles: Provide node12-*php*-composer images [integration/config] - 10https://gerrit.wikimedia.org/r/698257 (https://phabricator.wikimedia.org/T284345) [21:36:32] <wikibugs> (03PS1) 10Jforrester: [WIP] dockerfiles: Provide a node12-based Kartotherian image [integration/config] - 10https://gerrit.wikimedia.org/r/698258 (https://phabricator.wikimedia.org/T284345) [21:37:17] <wikibugs> 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Patch-For-Review: Upgrade all CI jobs for WMF-deployed projects from Node 10 to Node 12 - https://phabricator.wikimedia.org/T284345 (10Jdforrester-WMF) OK, this is mostly done except for two sets of jobs: * Jobs using the special Kartotherian im... [22:00:03] <wikibugs> 10Continuous-Integration-Config, 10Release-Engineering-Team, 10Patch-For-Review: Upgrade all CI jobs for WMF-deployed projects from Node 10 to Node 12 - https://phabricator.wikimedia.org/T284345 (10Jdforrester-WMF) Announced on wikitech-l in case it breaks anything: https://lists.wikimedia.org/hyperkitty/lis... [22:13:54] <James_F> Anyone know why the pipeline publish job would tag and immediately untag its build? https://integration.wikimedia.org/ci/job/service-pipeline-test-and-publish/1932/console -> confusion on T283902 [22:13:54] <stashbot> T283902: Evaluator service dies on receiving a request - https://phabricator.wikimedia.org/T283902 [22:34:31] <wikibugs> (03PS1) 10Jforrester: Update all the example SRE base image names [blubber] - 10https://gerrit.wikimedia.org/r/698267 [23:09:07] <wikibugs> (03PS1) 10DannyS712: Review access change [VisualEditor/VisualEditor] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/698042 [23:09:34] <wikibugs> (03PS2) 10DannyS712: Add `mediawiki` group as co-owners to allow +2 [VisualEditor/VisualEditor] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/698042 (https://phabricator.wikimedia.org/T284360) [23:12:41] <wikibugs> 10Release-Engineering-Team (Seen): Change notification email from jenkins-bot@wikimedia.org to releng internal list - https://phabricator.wikimedia.org/T151642 (10greg) I've recently been getting a few of these (mostly from @Jdforrester-WMF cleaning up portalsbuild?). It's not a major problem, but looking at the... [23:47:47] <Krinkle> !log Qemu jobs are stuck. Jenkins is unable to connect to integration-agent-qemu-1001 [23:47:49] <stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:47:55] <Krinkle> Probaly related to the restart earlier this week [23:47:56] <Krinkle> andrewbogott: ^ [23:48:06] <Krinkle> Did its IP address change as part of the reboot? [23:48:28] <Krinkle> appears to not have changed [23:48:35] <Krinkle> https://openstack-browser.toolforge.org/project/integration lists 172.16.1.187 [23:48:40] <Krinkle> which is what Jenkins is trying to connect to as well [23:49:13] <Krinkle> $ ssh integration-agent-qemu-1001.integration.eqiad1.wikimedia.cloud [23:49:13] <Krinkle> kex_exchange_identification: Connection closed by remote host [23:49:20] <Krinkle> (from my own laptop) [23:49:38] <andrewbogott> ok, checking [23:50:09] <Krinkle> it doesnt' receive jobs very often so presumably down since the reboot [23:50:15] <Krinkle> (less than once a day) [23:50:24] <andrewbogott> it's up, I can ssh as root and also as myself [23:50:42] <andrewbogott> dpkg is broken which could be something or nothing [23:51:21] <Krinkle> I can connect to deployment-deploy01.deployment-prep.eqiad1.wikimedia.cloud without issue [23:52:14] <andrewbogott> can you tell me what bastion you're using for your proxy jump? [23:52:58] <Krinkle> bastion.wmflabs.org:22 [23:52:59] <Krinkle> https://github.com/Krinkle/dotfiles/blob/8295825049b31a05e4e77d19550ab2a34f26a573/hosts/primary/sshconfig#L10-L13 [23:54:29] <andrewbogott> dns looks ok from there [23:55:11] <andrewbogott> I'm fixing puppet, then going to reboot (since apparently it isn't doing anything worthwhile anyway) [23:56:19] <andrewbogott> Krinkle: try ssh again now? [23:56:28] <Krinkle> I notice the osbrowser reports -CPU-Mem-Disk, which I guess doesn't mean anything (it shows fine in Horizon) [23:57:05] <andrewbogott> auth.log seems happy with your login... did it work? [23:57:10] <Krinkle> Yep, I'm in now. Let's see if Jenkins is equally lucky [23:57:17] <andrewbogott> 'k [23:57:31] <Krinkle> yes it is [23:57:33] <andrewbogott> Huh [23:57:40] <andrewbogott> well no idea why it worked for me all along :) [23:57:42] <Krinkle> !log integration-agent-qemu-1001 back up, Thanks andrewbogott [23:57:43] <stashbot> Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:57:52] <andrewbogott> sure thing [23:58:08] <Krinkle> andrewbogott: I see in my terminal log that my fourth attempt to ssh, just a second before you rebooted, worked [23:58:19] <Krinkle> > krinkle@integration-agent-qemu-1001:~$ Connection to integration-agent-qemu-1001.integration.eqiad1.wikimedia.cloud closed by remote host. [23:58:27] <Krinkle> I received a prompt just before you rebooted it [23:58:44] <Krinkle> in case that helps :)