[01:44:56] (03PS1) 10Jforrester: jjb: Update node10-test-browser-php80-composer users to latest [integration/config] - 10https://gerrit.wikimedia.org/r/720137 [01:44:58] (03PS1) 10Jforrester: Zuul: [wikipeg] Drop php72 special test job, the php80 one suffices [integration/config] - 10https://gerrit.wikimedia.org/r/720138 [01:45:00] (03PS1) 10Jforrester: jjb: Drop wikipeg-special-node10-plus-php72-composer-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720139 [01:45:02] (03PS1) 10Jforrester: Docker: Drop node{10,12}-test-browser-php72-composer, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/720140 (https://phabricator.wikimedia.org/T290651) [01:46:16] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/720137 (owner: 10Jforrester) [01:46:32] (03CR) 10Jforrester: [C: 03+2] Zuul: [wikipeg] Drop php72 special test job, the php80 one suffices [integration/config] - 10https://gerrit.wikimedia.org/r/720138 (owner: 10Jforrester) [01:47:56] (03Merged) 10jenkins-bot: jjb: Update node10-test-browser-php80-composer users to latest [integration/config] - 10https://gerrit.wikimedia.org/r/720137 (owner: 10Jforrester) [01:47:58] (03Merged) 10jenkins-bot: Zuul: [wikipeg] Drop php72 special test job, the php80 one suffices [integration/config] - 10https://gerrit.wikimedia.org/r/720138 (owner: 10Jforrester) [01:48:23] !log Zuul: [wikipeg] Drop php72 special test job, the php80 one suffices [01:48:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:49:16] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/720139 (owner: 10Jforrester) [01:49:21] (03CR) 10Jforrester: [C: 03+2] Docker: Drop node{10,12}-test-browser-php72-composer, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/720140 (https://phabricator.wikimedia.org/T290651) (owner: 10Jforrester) [01:50:43] (03Merged) 10jenkins-bot: jjb: Drop wikipeg-special-node10-plus-php72-composer-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720139 (owner: 10Jforrester) [01:50:47] (03Merged) 10jenkins-bot: Docker: Drop node{10,12}-test-browser-php72-composer, now unused [integration/config] - 10https://gerrit.wikimedia.org/r/720140 (https://phabricator.wikimedia.org/T290651) (owner: 10Jforrester) [01:51:45] (03PS1) 10Jforrester: jjb: Provide ooui-special-node12-plus-php80-composer-docker [integration/config] - 10https://gerrit.wikimedia.org/r/720141 [01:51:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: CI Docker images failing to build - https://phabricator.wikimedia.org/T290651 (10Jdforrester-WMF) 05Open→03Resolved a:03Jdforrester-WMF Fixed by dropping the PHP72 images. ;-) [01:51:47] (03PS1) 10Jforrester: Zuul: [oojs/ui] Add ooui-special-node12-plus-php80-composer-docker as experimental [integration/config] - 10https://gerrit.wikimedia.org/r/720142 [01:52:51] (03CR) 10jerkins-bot: [V: 04-1] Zuul: [oojs/ui] Add ooui-special-node12-plus-php80-composer-docker as experimental [integration/config] - 10https://gerrit.wikimedia.org/r/720142 (owner: 10Jforrester) [01:53:32] (03CR) 10Jforrester: [C: 03+2] jjb: Provide ooui-special-node12-plus-php80-composer-docker [integration/config] - 10https://gerrit.wikimedia.org/r/720141 (owner: 10Jforrester) [01:53:53] (03PS2) 10Jforrester: Zuul: [oojs/ui] Add ooui-special-node12-plus-php80-composer-docker as exp. [integration/config] - 10https://gerrit.wikimedia.org/r/720142 [01:54:43] (03Merged) 10jenkins-bot: jjb: Provide ooui-special-node12-plus-php80-composer-docker [integration/config] - 10https://gerrit.wikimedia.org/r/720141 (owner: 10Jforrester) [01:55:34] (03CR) 10Jforrester: [C: 03+2] Zuul: [oojs/ui] Add ooui-special-node12-plus-php80-composer-docker as exp. [integration/config] - 10https://gerrit.wikimedia.org/r/720142 (owner: 10Jforrester) [01:56:59] (03Merged) 10jenkins-bot: Zuul: [oojs/ui] Add ooui-special-node12-plus-php80-composer-docker as exp. [integration/config] - 10https://gerrit.wikimedia.org/r/720142 (owner: 10Jforrester) [01:58:25] !log Zuul: [oojs/ui] Add ooui-special-node12-plus-php80-composer-docker as experimental [01:58:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:07:48] (03PS1) 10Jforrester: Zuul: [oojs/ui] Switch special JS+PHP job from node10 to node12 [integration/config] - 10https://gerrit.wikimedia.org/r/720144 [02:07:50] (03PS1) 10Jforrester: jjb: Drop ooui-special-node10-plus-php80-composer-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720145 [02:07:52] (03PS1) 10Jforrester: jjb: Provide wikipeg-special-node12-plus-php80-composer-docker [integration/config] - 10https://gerrit.wikimedia.org/r/720146 [02:07:54] (03PS1) 10Jforrester: Zuul: [wikipeg] Provide wikipeg-special-node12-plus-php80-composer-docker as exp. [integration/config] - 10https://gerrit.wikimedia.org/r/720147 [02:07:56] (03PS1) 10Jforrester: Zuul: [wikipeg] Switch JS+PHP job from node10 to node12 [integration/config] - 10https://gerrit.wikimedia.org/r/720148 [02:07:58] (03PS1) 10Jforrester: jjb: Drop wikipeg-special-node10-plus-php80-composer-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720149 [02:08:00] (03PS1) 10Jforrester: debug-image: Point to a non-deprecated image as an example [integration/config] - 10https://gerrit.wikimedia.org/r/720150 [02:08:02] (03CR) 10Jforrester: [C: 03+2] Zuul: [oojs/ui] Switch special JS+PHP job from node10 to node12 [integration/config] - 10https://gerrit.wikimedia.org/r/720144 (owner: 10Jforrester) [02:09:59] (03Merged) 10jenkins-bot: Zuul: [oojs/ui] Switch special JS+PHP job from node10 to node12 [integration/config] - 10https://gerrit.wikimedia.org/r/720144 (owner: 10Jforrester) [02:10:09] !log Zuul: [oojs/ui] Switch special JS+PHP job from node10 to node12 [02:10:11] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:10:14] (03CR) 10jerkins-bot: [V: 04-1] Zuul: [wikipeg] Provide wikipeg-special-node12-plus-php80-composer-docker as exp. [integration/config] - 10https://gerrit.wikimedia.org/r/720147 (owner: 10Jforrester) [02:10:31] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/720145 (owner: 10Jforrester) [02:10:53] (03CR) 10Jforrester: [C: 03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/720146 (owner: 10Jforrester) [02:11:14] (03PS2) 10Jforrester: Zuul: [wikipeg] Provide wikipeg-special-node12-plus-php80-composer-docker [integration/config] - 10https://gerrit.wikimedia.org/r/720147 [02:11:18] (03CR) 10Jforrester: [C: 03+2] Zuul: [wikipeg] Provide wikipeg-special-node12-plus-php80-composer-docker [integration/config] - 10https://gerrit.wikimedia.org/r/720147 (owner: 10Jforrester) [02:12:15] (03Merged) 10jenkins-bot: jjb: Drop ooui-special-node10-plus-php80-composer-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720145 (owner: 10Jforrester) [02:12:39] (03Merged) 10jenkins-bot: jjb: Provide wikipeg-special-node12-plus-php80-composer-docker [integration/config] - 10https://gerrit.wikimedia.org/r/720146 (owner: 10Jforrester) [02:12:42] (03Merged) 10jenkins-bot: Zuul: [wikipeg] Provide wikipeg-special-node12-plus-php80-composer-docker [integration/config] - 10https://gerrit.wikimedia.org/r/720147 (owner: 10Jforrester) [02:13:16] !log Zuul: [wikipeg] Provide wikipeg-special-node12-plus-php80-composer-docker as an experimental job [02:13:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:16:15] (03PS2) 10Jforrester: Zuul: [wikipeg] Switch JS+PHP job from node10 to node12 [integration/config] - 10https://gerrit.wikimedia.org/r/720148 [02:16:19] (03CR) 10Jforrester: [C: 03+2] Zuul: [wikipeg] Switch JS+PHP job from node10 to node12 [integration/config] - 10https://gerrit.wikimedia.org/r/720148 (owner: 10Jforrester) [02:17:51] (03Merged) 10jenkins-bot: Zuul: [wikipeg] Switch JS+PHP job from node10 to node12 [integration/config] - 10https://gerrit.wikimedia.org/r/720148 (owner: 10Jforrester) [02:18:11] !log Zuul: [wikipeg] Switch JS+PHP job from node10 to node12 [02:18:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:18:19] (03CR) 10Jforrester: [C: 03+2] jjb: Drop wikipeg-special-node10-plus-php80-composer-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720149 (owner: 10Jforrester) [02:19:48] (03Merged) 10jenkins-bot: jjb: Drop wikipeg-special-node10-plus-php80-composer-docker, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720149 (owner: 10Jforrester) [02:20:06] (03CR) 10Jforrester: [C: 03+2] debug-image: Point to a non-deprecated image as an example [integration/config] - 10https://gerrit.wikimedia.org/r/720150 (owner: 10Jforrester) [02:21:51] (03Merged) 10jenkins-bot: debug-image: Point to a non-deprecated image as an example [integration/config] - 10https://gerrit.wikimedia.org/r/720150 (owner: 10Jforrester) [02:24:55] (03PS1) 10Jforrester: Docker: Drop node10-test-browser-php80-composer, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720152 [02:24:57] (03PS1) 10Jforrester: Docker: Drop node10-test-browser, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720153 [02:24:59] (03PS1) 10Jforrester: Docker: Drop node10-test, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720154 [02:25:01] (03PS1) 10Jforrester: Docker: Drop node10-portals, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720155 [02:25:03] (03PS1) 10Jforrester: Docker: Drop node10, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720156 (https://phabricator.wikimedia.org/T210704) [02:25:05] (03PS1) 10Jforrester: Docker: Update miscellaneous documentation mentions of node10 [integration/config] - 10https://gerrit.wikimedia.org/r/720157 [02:26:21] Krinkle: ^^^ Tah-dah. Only 900 days after it went EOL. [02:26:26] 10Release-Engineering-Team (Doing), 10GitLab, 10Patch-For-Review, 10Privacy, 10User-brennen: GitLab uses 'real name' as username (rather than 'shell name' or an user-specified name) - https://phabricator.wikimedia.org/T288392 (10brennen) > This is fairly easy to change however I'm not sure how the change... [02:26:30] (03CR) 10jerkins-bot: [V: 04-1] Docker: Drop node10-test, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720154 (owner: 10Jforrester) [02:26:55] James_F: but but but fresh stll need it. [02:27:05] can we keep them around just to avid them getting nuked? [02:27:06] Krinkle: Wait, really? [02:27:11] Oy. [02:27:32] You use node10-test-browser still? I thought you switched to 12? [02:27:40] Oh, right, Qemu issues. [02:27:42] * James_F sighs. [02:27:44] I'm still waiting for Qemu to be updated and/or for upstream to unbreak Node 12 compat with Qemu [02:27:55] Do you think either are going to happen ever? [02:27:59] Because I don't. [02:28:01] the latter, unlikey [02:28:11] the former, yes, the integration agent is already online [02:28:12] (03CR) 10Jforrester: [C: 04-2] "Krinkle still needs this. :-(" [integration/config] - 10https://gerrit.wikimedia.org/r/720153 (owner: 10Jforrester) [02:28:23] I'm waiting for Dave to create a Debian image for it. [02:28:32] (03PS2) 10Jforrester: Docker: Drop node10-portals, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720155 [02:28:34] (03PS2) 10Jforrester: Docker: Drop node10-test-browser, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720153 [02:28:36] Ah, cool. [02:28:36] (03PS2) 10Jforrester: Docker: Drop node10-test, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720154 [02:28:38] (03PS2) 10Jforrester: Docker: Drop node10, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720156 (https://phabricator.wikimedia.org/T210704) [02:28:40] (03PS2) 10Jforrester: Docker: Update miscellaneous documentation mentions of node10 [integration/config] - 10https://gerrit.wikimedia.org/r/720157 [02:28:42] (03CR) 10Jforrester: [C: 03+2] Docker: Drop node10-test-browser-php80-composer, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720152 (owner: 10Jforrester) [02:28:50] newer qemu was already in debian bullseye [02:29:00] Are you jumping straight to that? [02:29:05] but upgrade is non-trivial as it wasn't puppetized and we no longer have lars [02:29:23] who made the previous qemu debian image VM [02:29:30] Right. :-( [02:29:31] yep [02:29:40] Cool. The more stuff runs on bullseye the better. [02:29:43] https://phabricator.wikimedia.org/T284774 [02:29:48] I figured it's worth a shot [02:30:05] ran into a few early issues where I was apparently the first person to create a cloud vm with bullseye that uses /srv mounts [02:30:11] Having finished the 10->12 migration I find that we still have the 6->10 task open (because the Services team owns the work and don't exist any more). [02:30:11] (03Merged) 10jenkins-bot: Docker: Drop node10-test-browser-php80-composer, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720152 (owner: 10Jforrester) [02:30:23] (03CR) 10Jforrester: [C: 03+2] Docker: Drop node10-portals, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720155 (owner: 10Jforrester) [02:31:04] And of course it's about time to start the 12->14 migration. Maybe in January. [02:31:28] (03Merged) 10jenkins-bot: Docker: Drop node10-portals, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720155 (owner: 10Jforrester) [02:31:44] (03PS3) 10Jforrester: Docker: Update miscellaneous documentation mentions of node10 [integration/config] - 10https://gerrit.wikimedia.org/r/720157 [02:31:46] (03PS3) 10Jforrester: Docker: Drop node10-test-browser, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720153 [02:31:48] (03PS3) 10Jforrester: Docker: Drop node10-test, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720154 [02:31:50] (03PS3) 10Jforrester: Docker: Drop node10, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720156 (https://phabricator.wikimedia.org/T210704) [02:31:55] (03CR) 10Jforrester: [C: 03+2] Docker: Update miscellaneous documentation mentions of node10 [integration/config] - 10https://gerrit.wikimedia.org/r/720157 (owner: 10Jforrester) [02:33:37] (03Merged) 10jenkins-bot: Docker: Update miscellaneous documentation mentions of node10 [integration/config] - 10https://gerrit.wikimedia.org/r/720157 (owner: 10Jforrester) [03:21:15] 10Release-Engineering-Team (Seen), 10MediaWiki-Release-Tools: Branch script doesn't respect `branchpoint` when creating submodules, resulting in "implicit merges" error from gerrit - https://phabricator.wikimedia.org/T279718 (10Jdforrester-WMF) Ping. It'd be really really nice to get this fixed before we hit t... [03:21:57] 10Release-Engineering-Team (Seen), 10MediaWiki-Release-Tools: Branch script doesn't respect `branchpoint` when creating submodules, resulting in "implicit merges" error from gerrit - https://phabricator.wikimedia.org/T279718 (10Jdforrester-WMF) (Alternatively, I can just stay around and run this at the same ti... [07:40:27] (03CR) 10Addshore: [C: 03+1] "I think on the whole this looks pretty good" [integration/config] - 10https://gerrit.wikimedia.org/r/683589 (https://phabricator.wikimedia.org/T278706) (owner: 10Michael Große) [08:27:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: CI Docker images failing to build - https://phabricator.wikimedia.org/T290651 (10hashar) Awesome, thank you! [08:30:22] (03CR) 10Hashar: [C: 03+2] "It is a miracle! Well done." [integration/config] - 10https://gerrit.wikimedia.org/r/720156 (https://phabricator.wikimedia.org/T210704) (owner: 10Jforrester) [08:30:33] (03CR) 10Hashar: [C: 03+2] Docker: Drop node10-test, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720154 (owner: 10Jforrester) [08:30:45] (03CR) 10Hashar: [C: 03+2] Docker: Drop node10-test-browser, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720153 (owner: 10Jforrester) [08:35:18] (03PS4) 10Hashar: Docker: Drop node10-test-browser, unused [integration/config] - 10https://gerrit.wikimedia.org/r/720153 (owner: 10Jforrester) [09:29:20] (03PS8) 10Michael Große: Add and use job to deploy branch in node project [integration/config] - 10https://gerrit.wikimedia.org/r/683589 (https://phabricator.wikimedia.org/T278706) [09:29:22] (03CR) 10Michael Große: Add and use job to deploy branch in node project (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/683589 (https://phabricator.wikimedia.org/T278706) (owner: 10Michael Große) [10:25:26] Hey 🙂 [10:25:27] I want to add a job, that allows deploying of an unmerged gerrit patch of a static vuejs app to some hosting service like netlify or vercel (or something like patchdemo on toolforge, if that one day exists for the jamstack). The goal is that also UX (and sometimes even Product Owner) can review a patch before it is merged without having to set up a dev environment on their device. [10:25:27] This is my first time adding a whole new job and project to the integration config, and so I want to tread especially careful in order to not break anything. Could you have a look at https://gerrit.wikimedia.org/r/c/integration/config/+/683589 and give me some feedback there? [11:50:14] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [wikimedia/fundraising/privatebin] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/720285 [11:50:16] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [wikimedia/fundraising/privatebin] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/720285 (owner: 10QChris) [11:53:57] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [wikimedia/fundraising/privatebin] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/720288 [11:54:00] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [wikimedia/fundraising/privatebin] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/720288 (owner: 10QChris) [12:12:07] (03PS1) 10Arturo Borrero Gonzalez: zuul/layout.yaml: add configuration for cloud/toolforge/jobs-framework-emailer [integration/config] - 10https://gerrit.wikimedia.org/r/720296 [12:13:48] (03CR) 10Arturo Borrero Gonzalez: "This change is ready for review." [cloud/toolforge/jobs-framework-emailer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/720256 (owner: 10Arturo Borrero Gonzalez) [12:14:26] (03Abandoned) 10Arturo Borrero Gonzalez: Edit Repo Config [cloud/toolforge/jobs-framework-emailer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/720256 (owner: 10Arturo Borrero Gonzalez) [12:14:48] (03CR) 10Arturo Borrero Gonzalez: "This change is ready for review." [cloud/toolforge/jobs-framework-emailer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/720257 (owner: 10Arturo Borrero Gonzalez) [12:15:27] (03Abandoned) 10Arturo Borrero Gonzalez: Edit Repo Config [cloud/toolforge/jobs-framework-emailer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/720257 (owner: 10Arturo Borrero Gonzalez) [12:15:50] (03CR) 10Arturo Borrero Gonzalez: "This change is ready for review." [cloud/toolforge/jobs-framework-emailer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/720259 (owner: 10Arturo Borrero Gonzalez) [12:16:04] (03CR) 10Arturo Borrero Gonzalez: [V: 03+2 C: 03+2] Edit Repo Config [cloud/toolforge/jobs-framework-emailer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/720259 (owner: 10Arturo Borrero Gonzalez) [13:55:48] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Puppet agent failure on integration-agent-qemu-1001 in project integration - https://phabricator.wikimedia.org/T290615 (10hashar) Trying to install busybox yields: ` Unpacking busybox (1:1.22.0-19+deb9u2) ... dpkg: error processing archiv... [14:03:13] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Puppet agent failure on integration-agent-qemu-1001 in project integration - https://phabricator.wikimedia.org/T290615 (10hashar) And of course upon reboot the fs is borked. From the console https://horizon.wikimedia.org/project/instances... [14:04:06] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): integration-agent-qemu-1001 in project integration has corrupted disk / partition - https://phabricator.wikimedia.org/T290615 (10hashar) p:05Triage→03High [14:15:37] jbond, jelto: so i'm not sure how best to proceed on T288392 at this point. i spent a while digging into gitlab code last night and have not come up with much useful. [14:15:38] T288392: GitLab uses 'real name' as username (rather than 'shell name' or an user-specified name) - https://phabricator.wikimedia.org/T288392 [14:20:31] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): integration-agent-qemu-1001 in project integration has corrupted disk / partition - https://phabricator.wikimedia.org/T290615 (10hashar) Disk had a lot of errors. @Andrew kindly jumped in to run fsck manually through the console. Log of... [14:22:04] PROBLEM - Work requests waiting in Zuul Gearman server on contint2001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [14:25:52] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): integration-agent-qemu-1001 in project integration has corrupted disk / partition - https://phabricator.wikimedia.org/T290615 (10hashar) I have retriggerd an install of the linux kernel from stretch-backports as well as busybox since it i... [14:42:32] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): integration-agent-qemu-1001 in project integration has corrupted disk / partition - https://phabricator.wikimedia.org/T290615 (10hashar) Ultimately fixed by @Andrew . The instance is back, apt works. Running puppet I now have: `countere... [14:46:33] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): integration-agent-qemu-1001 in project integration has corrupted disk / partition - https://phabricator.wikimedia.org/T290615 (10hashar) For the lldp_neighbor fact: ` Debug: Facter: executing command: /bin/sh -c /usr/sbin/lldpctl -f xml D... [14:46:38] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): Puppet agent failure detected on instance integration-agent-pkgbuilder-1002 in project integration - https://phabricator.wikimedia.org/T288238 (10hashar) [14:46:54] 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): integration-agent-qemu-1001 in project integration has corrupted disk / partition - https://phabricator.wikimedia.org/T290615 (10hashar) 05Open→03Resolved a:03hashar [14:52:19] Krinkle: so the poor integration-agent-qemu1001 instance had a corrupted filesystem. A fsck more or less fixed it [14:52:35] but the fresh-test job fails to boot qemue instance for some reason and the job never manages to ssh into it [14:52:55] maybe the image is corrupted or some other files went to have an issue :-\ [14:52:58] hashar: still, or not anymore? [14:53:06] * dancy doesn't trust fsck to repair a filesystem that jacked up. [14:54:01] I've been blocked for several months now trying to roll out a seemingly simple nodejs update for the fresh dev environment. [14:54:07] apparently this morning (before fsck/reboot) it could not reach qemu https://integration.wikimedia.org/ci/job/fresh-test/162/console [14:54:33] the reason being that our new node images use a newer debian version, and that combination of nodejs and debian is not supported by nodejs when run inside the old stretch's qemu VM [14:54:53] and this in turn is relevant because we test our docker tools such as fresh by runing htem inside a VM since we don't want to run them directly on the agent. [14:55:18] any ideas for how to unblock this are welcome :) maybe we want to revisit the way we test this, with a method other than a docker container in a VM in a VM [14:55:31] I just noticed the agent to have faulty partition [14:55:51] T284696, T284774 [14:55:51] T284774: Provide one or more Qemu agents in CI that use a newer version than 2.x - https://phabricator.wikimedia.org/T284774 [14:55:52] T284696: Update Fresh from Node.js 10 LTS to Node.js 12 LTS - https://phabricator.wikimedia.org/T284696 [14:55:53] and the node 12 test browser image has an old node12 package (our base image does not included bullseye security update) [14:55:57] it is a whole can of worms [14:56:27] maybe given we limit CI jobs to trested people we can just run the shell script from the repo on the integration agent, and then invoke docker direclty like we do from other CI jobs. [14:57:22] I've lost track of what we all expose integration agents to, but when we first set this up afaik the only risk was someone messing up the agent itself in which case we can re-create it [14:57:38] if there is no other risk, I think that's a worthwhile bet since it's going to be rare and easily traceable [14:58:11] i hope we've not started trusting integration agents for anything security sensitive or access to prod etc in ways that code on the agent running could misuse [15:01:56] no idea [15:03:28] I'm not aware of any such trust being established. [15:05:16] I amnaged to boot/srv/vm-images/qemu-debian10buster-2020_05_04c.img [15:05:20] but sshd is not listening bah [15:06:05] oh no it does [15:07:58] but the hostfwd=tcp::4293-:22 does not seem to work :/ [15:12:26] I know nothing about qemu network / mapping etc :\ [15:13:13] Lemme know if you want to team up [15:16:24] [15:17:10] dancy: Krinkle https://meet.google.com/awx-noux-kbe if you wanna join the fun [15:18:19] > i hope we've not started trusting integration agents for anything security sensitive or access to prod etc in ways that code on the agent running could misuse [15:18:19] There seem to be credentials called `selenium-user-production` and a comment which indicates that this includes both username and password at https://github.com/wikimedia/integration-config/blob/f42b3a44421d41f8620042046392bfc3a5d02cad/jjb/job-templates-selenium.yaml#L73 [15:18:19] Does that qualify as "access to prod"? [15:23:26] MichaelG_WMDE: no, and these should only be sent to jobs that need it, they are not available on disk of other agents [15:24:28] ah, I see. Thank you for explanation 🙏 [15:46:20] (03PS1) 10Zfilipin: WIP selenium-daily-beta-WikibaseMediaInfo [integration/config] - 10https://gerrit.wikimedia.org/r/720339 (https://phabricator.wikimedia.org/T290740) [16:04:14] (03Abandoned) 10Zfilipin: WIP selenium-daily-beta-WikibaseMediaInfo [integration/config] - 10https://gerrit.wikimedia.org/r/720339 (https://phabricator.wikimedia.org/T290740) (owner: 10Zfilipin) [16:04:17] tldr: there is no working network [16:05:36] super productive debugging :D [16:06:06] RECOVERY - Work requests waiting in Zuul Gearman server on contint2001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [16:12:24] (03CR) 10Ahmon Dancy: "Very detailed!" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) (owner: 10Jeena Huneidi) [16:14:35] (03CR) 10Ahmon Dancy: [C: 03+1] Provide JSON schema for use in config validation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719332 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [16:43:58] 10Gerrit: https://18.138.195.109/egames/pragmaticplaywap - https://phabricator.wikimedia.org/T290748 (10Novendi55) [16:46:06] ^ disabled this user [16:56:42] (03PS1) 10Ebernhardson: wikidata: Include missing \ in multi-line bash for wdqs-gui [integration/config] - 10https://gerrit.wikimedia.org/r/720345 [16:57:43] (03CR) 10Ladsgroup: [C: 03+1] "oops. I deploy it now." [integration/config] - 10https://gerrit.wikimedia.org/r/720345 (owner: 10Ebernhardson) [16:59:15] (03CR) 10Ladsgroup: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/720345 (owner: 10Ebernhardson) [17:00:40] (03Merged) 10jenkins-bot: wikidata: Include missing \ in multi-line bash for wdqs-gui [integration/config] - 10https://gerrit.wikimedia.org/r/720345 (owner: 10Ebernhardson) [17:04:13] so hmm [17:04:21] qemu + dhcp + network bah [17:11:52] (03PS1) 10Ebernhardson: wikidata: Keep all files in build prefixed .git [integration/config] - 10https://gerrit.wikimedia.org/r/720351 [17:11:58] * hashar reads https://qemu.readthedocs.io/en/latest/system/devices/net.html [17:12:38] ebernhardson: I can update the job right now ;) [17:13:36] oh Amir1 is on it apparently [17:13:55] dancy: turns out qemu has some documentation! https://qemu.readthedocs.io/en/latest/system/devices/net.html#using-the-user-mode-network-stack [17:14:25] hashar: yup, I hope you don't mind. [17:14:46] Amir1: I am all for it ! [17:15:14] ^^ [17:19:50] hashar: Nod. I've read through some of that in the past. [17:20:19] maybe it relied on a sys setting that got manually set [17:23:35] (03CR) 10Ladsgroup: [C: 03+2] "Deployed, confirmed it fixes the issue." [integration/config] - 10https://gerrit.wikimedia.org/r/720351 (owner: 10Ebernhardson) [17:24:42] 10Release-Engineering-Team, 10MediaWiki-extensions-VipsScaler: Undeploy VipsScaler - https://phabricator.wikimedia.org/T290759 (10Jdforrester-WMF) [17:24:47] (03Merged) 10jenkins-bot: wikidata: Keep all files in build prefixed .git [integration/config] - 10https://gerrit.wikimedia.org/r/720351 (owner: 10Ebernhardson) [17:25:51] (03PS4) 10Jeena Huneidi: Add patch author/reviewer to promote patch [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) [17:26:17] (03CR) 10Jeena Huneidi: Add patch author/reviewer to promote patch (032 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) (owner: 10Jeena Huneidi) [17:26:52] (03CR) 10Ahmon Dancy: [C: 03+1] "Nice mod" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) (owner: 10Jeena Huneidi) [17:29:11] (03PS1) 10Jforrester: Stop branching VipsScalar for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/720359 (https://phabricator.wikimedia.org/T290759) [17:36:16] (03CR) 10Jeena Huneidi: [C: 04-1] "I think only the first reviewer added can has the '%' in front of the 'r'" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) (owner: 10Jeena Huneidi) [17:37:59] (03CR) 10Ahmon Dancy: [C: 03+2] Provide JSON schema for use in config validation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719332 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [17:38:43] (03Merged) 10jenkins-bot: Provide JSON schema for use in config validation [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719332 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [17:39:12] (03PS2) 10Jforrester: Zuul: [cloud/toolforge/jobs-framework-emailer] Add basic tox CI [integration/config] - 10https://gerrit.wikimedia.org/r/720296 (owner: 10Arturo Borrero Gonzalez) [17:39:19] (03PS3) 10Jforrester: Zuul: [cloud/toolforge/jobs-framework-emailer] Add basic tox CI [integration/config] - 10https://gerrit.wikimedia.org/r/720296 (owner: 10Arturo Borrero Gonzalez) [17:39:43] (03CR) 10Jforrester: [C: 03+2] Zuul: [cloud/toolforge/jobs-framework-emailer] Add basic tox CI [integration/config] - 10https://gerrit.wikimedia.org/r/720296 (owner: 10Arturo Borrero Gonzalez) [17:41:30] (03Merged) 10jenkins-bot: Zuul: [cloud/toolforge/jobs-framework-emailer] Add basic tox CI [integration/config] - 10https://gerrit.wikimedia.org/r/720296 (owner: 10Arturo Borrero Gonzalez) [17:41:43] !log Zuul: [cloud/toolforge/jobs-framework-emailer] Add basic tox CI [17:41:46] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:43:03] (03PS5) 10Jeena Huneidi: Add patch author/reviewer to promote patch [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) [17:45:19] (03PS6) 10Jeena Huneidi: Add patch author/reviewer to promote patch [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) [17:48:05] (03PS7) 10Dduvall: Perform validation using JSON schema and ajv-cli [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) [17:51:34] (03CR) 10Jforrester: "I've deployed this; I note that the repo has some python code style whines from black which is blocking merges there." [integration/config] - 10https://gerrit.wikimedia.org/r/720296 (owner: 10Arturo Borrero Gonzalez) [17:58:52] (03CR) 10Ahmon Dancy: Perform validation using JSON schema and ajv-cli (032 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [18:00:14] (03CR) 10Ahmon Dancy: [C: 03+1] Add patch author/reviewer to promote patch [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) (owner: 10Jeena Huneidi) [18:10:32] well I give up on the qemu networking issue [18:10:39] maybe the agent is broken after all :/ [18:10:45] have a sweet week-end [18:23:49] 10Release-Engineering-Team, 10MediaWiki-extensions-VipsScaler, 10Patch-For-Review: Undeploy VipsScaler from Wikimedia wikis - https://phabricator.wikimedia.org/T290759 (10Legoktm) [18:25:23] (03PS2) 10Legoktm: Stop branching VipsScaler for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/720359 (https://phabricator.wikimedia.org/T290759) (owner: 10Jforrester) [18:25:30] (03CR) 10Legoktm: [C: 03+1] Stop branching VipsScaler for Wikimedia production [tools/release] - 10https://gerrit.wikimedia.org/r/720359 (https://phabricator.wikimedia.org/T290759) (owner: 10Jforrester) [18:27:23] Ha, was just thinking about creating T290759 myself yesterday [18:27:24] T290759: Undeploy VipsScaler from Wikimedia wikis - https://phabricator.wikimedia.org/T290759 [18:31:10] (03CR) 10Dduvall: [C: 04-1] "A couple of legit concerns and a handful of a nits. I'm around for sync review and merge, and I'm super excited for this feature." [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) (owner: 10Jeena Huneidi) [18:31:27] AntiComposite: Great minds. :-) [18:33:46] (03CR) 10Ahmon Dancy: [C: 03+1] Add patch author/reviewer to promote patch (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) (owner: 10Jeena Huneidi) [18:37:37] (03CR) 10Dduvall: Perform validation using JSON schema and ajv-cli (032 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [18:39:00] (03CR) 10Ahmon Dancy: Perform validation using JSON schema and ajv-cli (032 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [18:43:31] (03CR) 10Dduvall: Perform validation using JSON schema and ajv-cli (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [18:58:35] James_F: I missed https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/720373 (oops), could you review that one too please? [19:00:10] (03PS1) 10Ebernhardson: integration/config: Apply shellcheck to scripts defined in jjb [integration/config] - 10https://gerrit.wikimedia.org/r/720377 [19:01:53] (03CR) 10Ebernhardson: "This is still missing adding shellcheck to whichever container runs these, haven't tracked that down yet. Also this is using a minimum sev" [integration/config] - 10https://gerrit.wikimedia.org/r/720377 (owner: 10Ebernhardson) [19:05:50] (03PS7) 10Jeena Huneidi: Add patch author/reviewer to promote patch [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) [19:06:53] (03CR) 10Dduvall: [C: 03+2] Add patch author/reviewer to promote patch [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) (owner: 10Jeena Huneidi) [19:07:30] (03Merged) 10jenkins-bot: Add patch author/reviewer to promote patch [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719319 (https://phabricator.wikimedia.org/T281392) (owner: 10Jeena Huneidi) [19:11:27] (03PS8) 10Dduvall: Perform validation using JSON schema and ajv-cli [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) [19:12:42] (03PS9) 10Dduvall: Perform validation using JSON schema and ajv-cli [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) [19:13:30] (03CR) 10Dduvall: Perform validation using JSON schema and ajv-cli (032 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [19:14:59] (03CR) 10Dduvall: [C: 04-1] Perform validation using JSON schema and ajv-cli (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [19:15:57] 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Scap: Scap should be clearer about the need for a revert after a failed canary check - https://phabricator.wikimedia.org/T290037 (10Tgr) Thanks @dancy, that makes much clearer what's happening. [19:16:29] (03PS10) 10Dduvall: Perform validation using JSON schema and ajv-cli [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) [19:16:47] (03CR) 10Dduvall: [C: 03+1] Perform validation using JSON schema and ajv-cli (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [19:20:35] (03CR) 10Ahmon Dancy: Perform validation using JSON schema and ajv-cli (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [19:26:28] (03CR) 10Jeena Huneidi: [C: 03+2] "LGTM!" [blubber] - 10https://gerrit.wikimedia.org/r/717582 (https://phabricator.wikimedia.org/T267685) (owner: 10Dduvall) [19:27:12] (03PS11) 10Dduvall: Perform validation using JSON schema and ajv-cli [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) [19:27:35] (03CR) 10Dduvall: [C: 03+1] Perform validation using JSON schema and ajv-cli (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/719382 (https://phabricator.wikimedia.org/T225335) (owner: 10Dduvall) [19:30:20] (03Merged) 10jenkins-bot: python: Support execution of site package modules in builder [blubber] - 10https://gerrit.wikimedia.org/r/717582 (https://phabricator.wikimedia.org/T267685) (owner: 10Dduvall) [19:36:31] 10Release-Engineering-Team (Yak Shaving 🐃🪒): pipeline-promote: Add patch author as reviewer to promote patch - https://phabricator.wikimedia.org/T281392 (10jeena) 05Open→03Resolved a:03jeena see https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/720384 for example of added reviewers [19:36:48] 10Release-Engineering-Team (Yak Shaving 🐃🪒): pipeline-promote: Add patch author as reviewer to promote patch - https://phabricator.wikimedia.org/T281392 (10jeena) [20:32:21] 10Release-Engineering-Team (Doing), 10Release Pipeline (Blubber): blubber: Execute python module as RUN command of dockerfile - https://phabricator.wikimedia.org/T267685 (10Jdforrester-WMF) Resolved? [20:39:26] 10Release-Engineering-Team (Doing), 10Release Pipeline (Blubber): blubber: Execute python module as RUN command of dockerfile - https://phabricator.wikimedia.org/T267685 (10jeena) I'll resolve this after deployment shortly [20:48:22] 10Release-Engineering-Team (Doing), 10Release Pipeline (Blubber): blubber: Execute python module as RUN command of dockerfile - https://phabricator.wikimedia.org/T267685 (10jeena) 05Open→03Resolved Deployed to production [20:48:52] Thank you wonderful RelEng. [20:49:06] (I was going to be specific to people, but really all of you are great, so…) [20:49:49] ^.^ seconded [20:52:06] aw shucks :) thanks! [20:57:54] <3 [21:06:23] 10Release-Engineering-Team (Doing), 10Release, 10Train Deployments: 1.37.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T281164 (10Legoktm) ##### Risky Patch! 🚂🔥 * **Change**: https://gerrit.wikimedia.org/r/716756 * **Summary**: ** SyntaxHighlight now shells out to pygments through the S... [21:19:46] (03PS2) 10Jforrester: jjb, Zuul: [integration/config] Add shellcheck job for scripts defined in jjb [integration/config] - 10https://gerrit.wikimedia.org/r/720377 (owner: 10Ebernhardson) [21:20:37] (03CR) 10Jforrester: [C: 03+2] "jjb part deployed; merging now to deploy the Zuul part and see if this works." [integration/config] - 10https://gerrit.wikimedia.org/r/720377 (owner: 10Ebernhardson) [21:22:07] (03Merged) 10jenkins-bot: jjb, Zuul: [integration/config] Add shellcheck job for scripts defined in jjb [integration/config] - 10https://gerrit.wikimedia.org/r/720377 (owner: 10Ebernhardson) [21:23:26] !log Zuul: [integration/config] Add shellcheck job for scripts defined in jjb as an experimental job [21:23:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:25:00] (03CR) 10Jforrester: "check experimental" [integration/config] - 10https://gerrit.wikimedia.org/r/714633 (https://phabricator.wikimedia.org/T280247) (owner: 10Ebernhardson) [21:25:57] James_F: not sure if it will, pretty sure we have to install the shellcheck package wherever that runs, [21:26:13] it wasn't particularly clear, but i suspect that runs on the host rather than in a docker container [21:26:45] ebernhardson: Yes, I believe it runs directly on the contint* machines. [21:26:56] 14:26:18 xargs: shellcheck: No such file or directory [21:26:59] Indeed. :-) [21:27:14] I guess we'd need a puppet patch adding that to the machines? [21:27:42] if it's contint machines, then yea. I can put one together, although i suppose i should check if it's packaged for debian and not just ubunutu where i installed it locally [21:27:52] yea it's in buster [21:27:53] Ah, wait, no. [21:28:04] It runs on integration-agent-docker-1002 [21:28:17] That's good, much easier to fix. [21:28:29] (Still a puppet change.) [21:30:23] wonder where it goes..profile::ci::slave::labs::common seems plausible [21:30:32] Yeah, was just looking. [21:30:40] That's the right one, I'm pretty sure. [21:32:37] And of course they're still running stretch. [21:32:39] * James_F sighs. [21:33:08] lol, old :) Looks like it has shellcheck still at least, guess it's been around awhile: https://packages.debian.org/stretch/shellcheck [21:33:47] Ack. [21:34:01] gets 0.4.4 instead of 0.7.1, maybe similar enough :) [21:34:20] The migration to GitLab CI can't come soon enough. ;-) [21:36:25] Oh, and of course I spotted this over a year ago, reported it into Phabricator, and then forgot about it: T252071 [21:36:26] T252071: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster - https://phabricator.wikimedia.org/T252071 [21:36:46] moving things is hard :P [21:37:00] And me leaving RelEng a month later probably didn't help. [21:40:09] Hmm. I'll have to decom one of the existing agents to add a new one. :-( [21:43:37] Let's give it a go. [21:44:45] !log Pulling oldest CI agent integration-agent-docker-1001 from rotation so it can be replaced by a bullseye one for T252071 [21:44:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:44:48] T252071: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster - https://phabricator.wikimedia.org/T252071 [21:48:39] !log Deleting CI agent integration-agent-docker-1001 for T252071 [21:48:42] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:51:01] Hmm, there's no replacement for g2.cores8.ram24.disk80.4xiops; closest is g3.cores8.ram24.disk20.ephemeral40.4xiops which'll need some special config. Joy. [21:52:27] !log Created experimental integration-agent-docker-1021 for T252071 [21:52:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:52:30] T252071: Move all Wikimedia CI (WMCS integration project) instances from stretch to buster - https://phabricator.wikimedia.org/T252071 [22:49:50] Any volunteer to team up to figure out why the webserver in CI is no longer compressing http responses with gzip? [22:51:44] Krinkle: Do you have a timeframe for when it broke? Possibly a quibble change? [22:52:03] Yeah, quibble likely [22:52:11] no useful timeframe other than less than 18 months [22:53:05] Eyeballing https://gerrit.wikimedia.org/r/plugins/gitiles/integration/quibble/+/refs/heads/master/CHANGELOG.rst I'm assuming it's not the switch from PHP -S to Apache in 0.0.46? [22:53:38] could be. I know the php -s server uses router.php which hasn't been changed and compressed things when I last changed it (which was the change that made it do compression) [22:53:56] * Krinkle files tasks [22:53:59] task* [22:54:15] But Apache should have gzip enabled by default, surely? [22:55:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Fresnel, 10Quibble, 10Performance-Team (Radar): Quibble webserver in CI no longer compressing http responses - https://phabricator.wikimedia.org/T290776 (10Krinkle) [22:56:55] James_F: for html/css/js static assets that it serves itself, yes, most definitely. [22:57:08] Where is the httpd.conf file stored nowadays? [22:57:13] for things via php-fpm, it depends on the config details. I don't think it always does per-se [22:57:30] (Why does every OS builder think they know what they're doing and have a better place than were it was 20 years ago?)_ [22:58:15] so, the new mediawiki-docker local env does compress things correctly fwiw [22:58:26] And that has a very short and similar config as quibble [22:58:40] Yeah. [22:58:48] two conf files at https://gitlab.wikimedia.org/releng/dev-images/-/tree/main/dockerfiles/buster-apache2 [22:58:57] OK, quibble-buster-php72-apache has mod_deflate loaded. [22:59:09] https://gerrit.wikimedia.org/r/plugins/gitiles/integration/quibble/+/refs/heads/master/docker/apache/ [23:00:49] the dev-image one is complete, that is, it doesn't do any includes from files not in version control (e.g. default things from debian or from other docker images) [23:00:58] * James_F nods. [23:01:11] the quibble one does a bunch of IncludeOptional statements [23:01:20] which may or may not be no-ops [23:01:55] Right, quibble-fresnel inherits from quibble-buster-php72. [23:01:56] it sounds like you have one of these images running locally? [23:02:01] Which doesn't have apache? [23:02:28] I have all the CI images locally for development of them; my MW-docker install runs mildly ahead of 'real' merged dev-images generally. [23:02:38] ack PhpWebserver [23:02:46] okay, so fresnel hasn't changed [23:02:51] which is good and bad. [23:02:54] Yeah. [23:03:18] good because it means apache probably doesn't have some problem that we can't seem to find [23:03:27] bad because the php server hasn't changed afaik and that's its own mystery [23:04:00] T219694 [23:04:01] T219694: Enable compression for MW web responses in Jenkins jobs (e.g. Quibble, Fresnel) - https://phabricator.wikimedia.org/T219694 [23:04:53] `'-d', 'output_buffering=Off'` is still there in latest quibble [23:05:05] so MW should be able to do its output buffer to compress as normal [23:05:30] as added in https://gerrit.wikimedia.org/r/c/integration/quibble/+/534468 linked from that task [23:07:04] Hmm. [23:10:02] probably something in MW's compression handler broke then.. [23:10:15] * Krinkle tries with a php -S locally using the same settings [23:10:26] Do we monitor production to make sure it's still working there? [23:10:38] yeah, it's working there. [23:10:53] Good. [23:10:57] we disable all compression internall at all layers until ats where it's compressed on the fly based on browser's headers [23:11:46] I don't know actually if it's monitored explicitly as such, but I imagine the ats part has tests, and we'd notice the increase in payload size in synthetic testing [23:12:10] And hopefully "outgoing traffic just jumped 50%" would show up… [23:13:23] yeah, that too [23:13:31] ok, so locally I ain't getting any compression [23:13:41] Oh dear. [23:13:48] That's an MW issue, then? [23:13:49] I thikn the output buffering changes Tim and Aaron made last month broke it [23:13:51] yeah [23:14:04] Because the production compression is done by ATS not MW? [23:14:40] indeed [23:14:54] We turn MW, PHP, Nginx, Apache compression all off for mw-related requests [23:14:59] and only the edge handles it [23:15:02] Right. [23:15:22] I've confirmed wgDisableOutputCompression is not true in CI and for me locally [23:15:48] wfWebStartSetup sees level 0 initially, which confirms `'-d', 'output_buffering=Off'` is working correclty and not undone by anything early at runtime [23:16:04] then in OutputHandler::handle we return early because the $phase is not clean for some reason