[00:29:29] <Daimona>	 Ohhhhh I think I may have a solution for the extensions thing in PHPUnit
[00:31:44] <Daimona>	 A PHP script that loads Setup.php and prints the ExtensionRegistry queue. The bootstrap would shell out and invoke the script, and read its output to know which extensions to load. And after I do that I'll watch out for dinosaurs https://xkcd.com/292/
[03:53:30] <wikibugs>	 10Release-Engineering-Team (Radar), 10Epic, 10Platform Team Initiatives (API Integration Tests), 10Quality-and-Test-Engineering-Team (QTE) (Test engineering), 10Testing-Roadblocks: Create and run a suite of end-to-end tests for the Wikimedia environment - https://phabricator.wikimedia.org/T248683 (10Tgr)
[05:15:04] <legoktm>	 /ms send Daimona re: "A PHP script that loads Setup.php and prints the ExtensionRegistry queue.", are you looking for maintenance/checkDependencies.php? (probably best to tweak that if it doesn't do exactly what you need)
[07:31:30] <wikibugs>	 (03PS1) 10Hashar: Add Puppet 7 catalogue compiler job [integration/config] - 10https://gerrit.wikimedia.org/r/937618 (https://phabricator.wikimedia.org/T236373)
[07:32:45] <wikibugs>	 (03CR) 10CI reject: [V: 04-1] Add Puppet 7 catalogue compiler job [integration/config] - 10https://gerrit.wikimedia.org/r/937618 (https://phabricator.wikimedia.org/T236373) (owner: 10Hashar)
[07:35:50] <wikibugs>	 (03PS2) 10Hashar: Add Puppet 7 catalogue compiler job [integration/config] - 10https://gerrit.wikimedia.org/r/937618 (https://phabricator.wikimedia.org/T236373)
[07:38:10] <wikibugs>	 (03CR) 10Hashar: "The job config looks fine then I don't know whether we want to run both compilation in parallel right now." [integration/config] - 10https://gerrit.wikimedia.org/r/937618 (https://phabricator.wikimedia.org/T236373) (owner: 10Hashar)
[07:38:22] <wikibugs>	 (03CR) 10Hashar: [C: 04-2] Add Puppet 7 catalogue compiler job [integration/config] - 10https://gerrit.wikimedia.org/r/937618 (https://phabricator.wikimedia.org/T236373) (owner: 10Hashar)
[07:59:28] <hashar>	 OH FUCK
[07:59:48] <RhinosF1>	 ?
[08:00:25] <hashar>	 !log integration: on CI Jenkins installing Node and Label parameter and updating the Token macro plugin
[08:00:26] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[08:00:30] <hashar>	 !log integration: on CI Jenkins installing Node and Label parameter and updating the Token macro plugin T236373
[08:00:32] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[08:00:33] <stashbot>	 T236373: puppet master command will be removed in puppet 6 - https://phabricator.wikimedia.org/T236373
[08:00:47] <hashar>	 ah no I freaked out because `wmf-insecte` had quit
[08:00:52] <hashar>	 that the irc bot for Jenkins
[08:00:58] <hashar>	 so when it dies, that indicates Jenkins is dead
[08:01:01] <hashar>	 or has troubles
[08:01:08] <hashar>	 but it just quitted and joined back cause the plugin got reloaded
[08:01:18] <RhinosF1>	 ah
[08:02:42] <wikibugs>	 10Phabricator, 10wikimedia.biterg.io: Closed tickets in Bugzilla migrated without a closing date - https://phabricator.wikimedia.org/T107254 (10Aklapper) >>! In T107254#9006822, @matmarex wrote: > It looks like you have no special handling for bugs that were closed and then reopened, so e.g. bug 1 (T2001) is l...
[08:05:08] <hashar>	 java.io.IOException: Failed to load: Node and Label parameter plugin (nodelabelparameter 1.12.0)
[08:05:08] <hashar>	  - Update required: Token Macro Plugin (token-macro 321.vd7cc1f2a_52c8) to be updated to 359.vb_cde11682e0c or higher
[08:05:09] <hashar>	 :)
[08:05:28] <hashar>	 so I have to update the token-macro plugin in order to install the Node and label parameter plugins
[08:05:31] <hashar>	 which requires a restart
[08:05:43] <hashar>	 will do it in a few, letting builds complete first
[08:09:34] <wikibugs>	 10Phabricator: Prevent adding new tasks in #Cloud-Services umbrella tag - https://phabricator.wikimedia.org/T329027 (10Aklapper) 05Open→03Resolved a:03Aklapper I created {H417} to automatically add a comment in such cases: > The #Cloud-Services project tag is not intended to have any tasks. Please check th...
[08:17:13] <wmf-insecte>	 Yippee, build fixed!
[08:17:13] <wmf-insecte>	 Project beta-code-update-eqiad build #452258: 09FIXED in 3 min 11 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/452258/
[09:23:52] <wikibugs>	 10Release-Engineering-Team (Radar), 10Research, 10User-brennen: Define Metrics for Change Failure Percentage - https://phabricator.wikimedia.org/T289567 (10Miriam) 05Open→03Resolved Resolving this as this collaboration has concluded a while ago. Thank you all!
[09:24:05] <wikibugs>	 10Phabricator, 10Privacy Engineering, 10Privacy: Difficulties registering a Phab account if third-party cookies are not enabled; email address requirement contradicts Privacy Policy - https://phabricator.wikimedia.org/T214251 (10Aklapper) I cannot reproduce any difficulties registering a Phab account if thir...
[09:27:57] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10collaboration-services, 10Patch-For-Review: Move micro sites from Ganeti to Kubernetes and from Gerrit to GitLab - https://phabricator.wikimedia.org/T300171 (10CodeReviewBot) jelto opened https://gitlab.wikimedia.org/repos/sre/miscweb/transparencyreport/-/merge_requ...
[09:32:40] <hashar>	 !log integration: on CI Jenkins, removed the Node and Label parameter plugin, it doesn't have the features I was looking for. T236373
[09:32:43] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[09:32:43] <stashbot>	 T236373: puppet master command will be removed in puppet 6 - https://phabricator.wikimedia.org/T236373
[09:32:53] <wikibugs>	 10Phabricator, 10Privacy Engineering, 10Privacy: Phabricator account email address requirement contradicts linked wiki Privacy Policy - https://phabricator.wikimedia.org/T214251 (10Aklapper)
[09:34:51] <wikibugs>	 10Phabricator, 10Privacy Engineering, 10Privacy: Phabricator account email address requirement contradicts linked wiki Privacy Policy - https://phabricator.wikimedia.org/T214251 (10Aklapper)
[09:35:06] <wikibugs>	 10Phabricator, 10Privacy Engineering, 10Privacy: Phabricator account email address requirement contradicts linked wiki Privacy Policy - https://phabricator.wikimedia.org/T214251 (10Aklapper) Currently, the page within Phabricator at https://phabricator.wikimedia.org/auth/start/?next=%2F which allows logging...
[10:04:51] <hashar>	 !log integration: updating Matrix Combination CI Jenkins plugin from 1.31 to 1.32 to fix: grey ball images are broken since Jenkins-2.333 (JENKINS-69777)  # T236373
[10:04:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:04:54] <stashbot>	 T236373: puppet master command will be removed in puppet 6 - https://phabricator.wikimedia.org/T236373
[10:06:15] <hashar>	 !log Restarting CI Jenkins for plugin update
[10:06:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:06:36] <hashar>	 o
[10:06:47] <hashar>	 so it reprocesses build after just 30 seconds, that is quite good :)
[10:18:46] <wmf-insecte>	 Yippee, build fixed!
[10:18:46] <wmf-insecte>	 Project beta-scap-sync-world build #111574: 09FIXED in 1 min 14 sec: https://integration.wikimedia.org/ci/job/beta-scap-sync-world/111574/
[10:42:03] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert)
[10:42:23] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Direct 1% of all traffic to mw-on-k8s - https://phabricator.wikimedia.org/T341463 (10Clement_Goubert) 05In progress→03Resolved Done, all looks ok. We'll now start preparing for 5%
[10:43:52] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Direct 1% of all traffic to mw-on-k8s - https://phabricator.wikimedia.org/T341463 (10Ladsgroup) >>! In T341463#9012011, @Clement_Goubert wrote: > Done, all looks ok. We'll now start preparing for 5%    {meme, src=itshappening}
[10:57:35] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Direct 5% of all traffic to mw-on-k8s - https://phabricator.wikimedia.org/T341780 (10Clement_Goubert)
[10:58:35] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Direct 5% of all traffic to mw-on-k8s - https://phabricator.wikimedia.org/T341780 (10Clement_Goubert) p:05Triage→03High
[11:04:00] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert)
[11:04:30] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Migrate group1 to Kubernetes - https://phabricator.wikimedia.org/T340549 (10Clement_Goubert) 05In progress→03Resolved After moving a couple group1 wikis, we have decided to go with a global traffic percentage to roll forwar...
[11:06:09] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert)
[11:11:38] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10collaboration-services, 10Patch-For-Review: Move micro sites from Ganeti to Kubernetes and from Gerrit to GitLab - https://phabricator.wikimedia.org/T300171 (10CodeReviewBot) jelto merged https://gitlab.wikimedia.org/repos/sre/miscweb/transparencyreport/-/merge_requ...
[11:12:02] <wmf-insecte>	 Yippee, build fixed!
[11:12:02] <wmf-insecte>	 Project mediawiki-core-doxygen-docker build #44794: 09FIXED in 7 min 43 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/44794/
[11:12:38] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert)
[11:23:34] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10collaboration-services, 10Patch-For-Review: Move micro sites from Ganeti to Kubernetes and from Gerrit to GitLab - https://phabricator.wikimedia.org/T300171 (10CodeReviewBot) jelto opened https://gitlab.wikimedia.org/repos/sre/miscweb/bienvenida/-/merge_requests/2...
[11:25:06] <wikibugs>	 10GitLab (Project Migration), 10collaboration-services: Migrate SRE repositories to GitLab - operations/software - https://phabricator.wikimedia.org/T341504 (10LSobanski)
[11:31:00] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10collaboration-services, 10Patch-For-Review: Move micro sites from Ganeti to Kubernetes and from Gerrit to GitLab - https://phabricator.wikimedia.org/T300171 (10CodeReviewBot) jelto merged https://gitlab.wikimedia.org/repos/sre/miscweb/bienvenida/-/merge_requests/2...
[11:43:34] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Find place in Phab where to display links to custom items (Privacy Policy; Help) - https://phabricator.wikimedia.org/T263 (10Aklapper)
[11:44:10] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10collaboration-services, 10Patch-For-Review: Move micro sites from Ganeti to Kubernetes and from Gerrit to GitLab - https://phabricator.wikimedia.org/T300171 (10CodeReviewBot) jelto opened https://gitlab.wikimedia.org/repos/sre/miscweb/annualreport/-/merge_requests/5...
[11:48:09] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Direct 1% of all traffic to mw-on-k8s - https://phabricator.wikimedia.org/T341463 (10Clement_Goubert) 05Resolved→03In progress I'm wondering if in the vein of   >>! In T290536#8466377, @Ladsgroup wrote: > This is not really...
[11:48:19] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert)
[11:48:42] <wikibugs>	 10GitLab (Pipeline Services Migration🐤), 10collaboration-services, 10Patch-For-Review: Move micro sites from Ganeti to Kubernetes and from Gerrit to GitLab - https://phabricator.wikimedia.org/T300171 (10CodeReviewBot) jelto merged https://gitlab.wikimedia.org/repos/sre/miscweb/annualreport/-/merge_requests/5...
[12:10:55] <wikibugs>	 10Differential, 10Phabricator: Phabricator does not provide an API to get Differential transaction data, similar to maniphest.gettasktransactions - https://phabricator.wikimedia.org/T123416 (10Aklapper)
[12:11:15] <wikibugs>	 10Differential, 10Phabricator, 10Wikibugs, 10NewFunctionality-Worktype: Create conduit method to query the feed and return records with relevant details populated instead of just a bunch of phids - https://phabricator.wikimedia.org/T123417 (10Aklapper) 05Open→03Stalled CC'ing @valhallasw as wikibugs' {...
[12:23:09] <wikibugs>	 10GitLab (Auth & Access), 10Release-Engineering-Team (They Live 🕶️🧟), 10CAS-SSO, 10Infrastructure-Foundations, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10SLyngshede-WMF) Depending on how the Gitlab OIDC pulls information we might have to change: u...
[13:00:59] <wikibugs>	 10Release-Engineering-Team (Deployment Training Requests): Deployment Training Request for xcollazo - https://phabricator.wikimedia.org/T341377 (10xcollazo)
[13:01:14] <wikibugs>	 10Release-Engineering-Team (Deployment Training Requests): Deployment Training Request for xcollazo - https://phabricator.wikimedia.org/T341377 (10xcollazo)
[13:23:45] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Add link to atom feed in each main blog page - https://phabricator.wikimedia.org/T205181 (10Aklapper) a:03Aklapper Since rPHAB3d33d1cceac7 we have a custom downstream `<link rel="alternate" type="application/atom+xml" href="/phame/blog/feed/7/" title="The Speed of Though...
[13:29:22] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10User-MModell: On Phabricator workboard, show status of associated Gerrit patches - https://phabricator.wikimedia.org/T215148 (10Aklapper)
[13:55:33] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10MW-on-K8s: Inform deployers of mw-debug deployment - https://phabricator.wikimedia.org/T341798 (10Clement_Goubert)
[13:57:57] <wikibugs>	 10Phabricator, 10Technical-Debt: Remove custom WMF Phab downstream hacks superseded by upstream Phorge changes - https://phabricator.wikimedia.org/T341799 (10Aklapper) p:05Triage→03Low
[13:58:06] <wikibugs>	 10Phabricator, 10Technical-Debt: Remove custom WMF Phab downstream hacks superseded by upstream Phorge changes - https://phabricator.wikimedia.org/T341799 (10Aklapper)
[13:58:15] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10collaboration-services, 10Patch-For-Review, 10User-brennen: Migrate phabricator.wikimedia.org to Phorge as upstream - https://phabricator.wikimedia.org/T333885 (10Aklapper)
[14:02:55] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10MW-on-K8s: Inform deployers of mw-debug deployment - https://phabricator.wikimedia.org/T341798 (10taavi) IMO if you want more people to use the k8s-experimental option you should update it to be the default option. That message is quite long already, and I think most peo...
[14:04:39] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10MW-on-K8s: Inform deployers of mw-debug deployment - https://phabricator.wikimedia.org/T341798 (10Clement_Goubert) That is a fair point, but I'm afraid that would mean people wouldn't test for bare-metal due to just not checking to what the option is set.
[14:20:42] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10MW-on-K8s: Inform deployers of mw-debug deployment - https://phabricator.wikimedia.org/T341798 (10Lucas_Werkmeister_WMDE) >>! In T341798#9012682, @taavi wrote: > That message is quite long already, and I think most people doing and testing deployments are familiar enough...
[14:23:18] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10MW-on-K8s: Inform deployers of mw-debug deployment - https://phabricator.wikimedia.org/T341798 (10taavi) Also, I would expect that a vast majority of the changes deployed via `scap backport` (config changes and relatively small bug fixes mostly) will work fine on both ba...
[14:27:17] <wikibugs>	 10Phabricator (Upstream), 10CSS, 10Upstream: In workboard columns, last bit of a vertically scrollable column cannot be seen when the column header has more than one line - https://phabricator.wikimedia.org/T255859 (10Aklapper)
[14:31:28] <wikibugs>	 10GitLab (Auth & Access), 10Release-Engineering-Team (They Live 🕶️🧟), 10CAS-SSO, 10Infrastructure-Foundations, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10SLyngshede-WMF) We didn't find a solution yet, but I'll spend some time looking into the CAS...
[14:45:59] <wikibugs>	 10Phabricator (Upstream), 10Browser-Support-Internet-Explorer, 10TestMe, 10Upstream: Upper horizontal scrollbar of the Workboard pages acting funny in Internet Explorer - https://phabricator.wikimedia.org/T190597 (10Aklapper) 05Open→03Stalled @gh87: Fire years later, is this "second horizontal scrollba...
[14:46:01] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Scrolling a Workboard page in various browsers - https://phabricator.wikimedia.org/T190604 (10Aklapper)
[14:46:24] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Scrolling a Workboard page in various browsers - https://phabricator.wikimedia.org/T190604 (10Aklapper) 05Open→03Stalled Stalling per open subtask
[14:46:28] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10MW-on-K8s: Inform deployers of mw-debug deployment - https://phabricator.wikimedia.org/T341798 (10Clement_Goubert) I sent an email to wikitech-l and ops-l. I don't necessarily expect every patch to be tested on both, but at least the information is out there on how to do...
[14:55:58] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10Documentation: Update [[mw:Phabricator/Code]] after migrating our upstream to Phorge - https://phabricator.wikimedia.org/T341808 (10Aklapper) 05Open→03Stalled p:05Triage→03Low
[14:56:03] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10Documentation: Update [[mw:Phabricator/Code]] after migrating our upstream to Phorge - https://phabricator.wikimedia.org/T341808 (10Aklapper)
[14:56:13] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (They Live 🕶️🧟), 10collaboration-services, 10Patch-For-Review, 10User-brennen: Migrate phabricator.wikimedia.org to Phorge as upstream - https://phabricator.wikimedia.org/T333885 (10Aklapper)
[15:10:37] <wikibugs>	 10GitLab (Auth & Access), 10Release-Engineering-Team (They Live 🕶️🧟), 10CAS-SSO, 10Infrastructure-Foundations, and 4 others: migrate gitlab away from the CAS protocol - https://phabricator.wikimedia.org/T320390 (10Arnoldokoth) @Jelto @SLyngshede-WMF I believe my login works now.   I did sign in to the repl...
[15:38:48] <wikibugs>	 10GitLab, 10Patch-For-Review: Add tests+linting to docpub - https://phabricator.wikimedia.org/T341225 (10CodeReviewBot) jnuche merged https://gitlab.wikimedia.org/repos/releng/docpub/-/merge_requests/3  add linting and tests to the project
[15:41:40] <wikibugs>	 10GitLab, 10Patch-For-Review: Add tests+linting to docpub - https://phabricator.wikimedia.org/T341225 (10jnuche) 05Open→03Resolved
[15:50:10] <wikibugs>	 10Differential, 10Phabricator, 10Wikibugs, 10NewFunctionality-Worktype: Create conduit method to query the feed and return records with relevant details populated instead of just a bunch of phids - https://phabricator.wikimedia.org/T123417 (10valhallasw) Wikibugs uses the task page html for  - getting the...
[16:00:11] <wikibugs>	 10Release-Engineering-Team, 10Wikimedia-Logstash: Kibana dashboard mediawiki-errors lacks channel errors and exceptions - https://phabricator.wikimedia.org/T341815 (10hashar)
[16:13:36] <wikibugs>	 10GitLab (Project Migration), 10collaboration-services: Migrate SRE repositories to GitLab - https://phabricator.wikimedia.org/T341468 (10LSobanski)
[16:22:13] <wikibugs>	 10GitLab (Auth & Access), 10Release-Engineering-Team, 10collaboration-services, 10Patch-For-Review, 10User-brennen: Create bot to sync LDAP groups with related GitLab groups - https://phabricator.wikimedia.org/T319211 (10dancy) a:03dancy
[16:37:08] <wikibugs>	 10Release-Engineering-Team (Radar), 10SRE, 10SRE-Access-Requests: Requesting access to release-engineering for aklapper - https://phabricator.wikimedia.org/T341749 (10cmooney)
[16:47:43] <wikibugs>	 10Release-Engineering-Team (Radar), 10SRE, 10SRE-Access-Requests, 10Patch-For-Review: Requesting access to release-engineering for aklapper - https://phabricator.wikimedia.org/T341749 (10Dzahn) We talked about this in meetings yesterday and today. Andre should get phabricator root. And this will be resolve...
[16:58:24] <wikibugs>	 10Release-Engineering-Team (Radar), 10SRE, 10SRE-Access-Requests, 10Patch-For-Review: Requesting access to release-engineering for aklapper - https://phabricator.wikimedia.org/T341749 (10cmooney) Path to add to "release-engineering" group directly is now merged.  @dzahn is there anything else I should do h...
[17:27:42] <wikibugs>	 10Differential, 10Phabricator: Phabricator does not provide an API to get Differential transaction data, similar to maniphest.gettasktransactions - https://phabricator.wikimedia.org/T123416 (10Aklapper)
[17:27:55] <wikibugs>	 10Differential, 10Phabricator, 10Wikibugs, 10NewFunctionality-Worktype: Create conduit method to query the feed and return records with relevant details populated instead of just a bunch of phids - https://phabricator.wikimedia.org/T123417 (10Aklapper) 05Stalled→03Declined Thanks *a lot* for elaboratin...
[17:30:12] <wikibugs>	 10Phabricator: One Phabricator task displayed in two workboard columns simultaneously - https://phabricator.wikimedia.org/T335443 (10Aklapper) p:05Triage→03Low cf https://secure.phabricator.com/D20901
[17:33:33] <wikibugs>	 10Phabricator, 10Release-Engineering-Team, 10collaboration-services, 10User-brennen: Allow tools to use phabricator webhooks - https://phabricator.wikimedia.org/T321790 (10Aklapper) p:05Triage→03Medium
[17:34:16] <wikibugs>	 10GitLab (Integrations), 10Phabricator, 10Release-Engineering-Team: GitLab repos are not correctly mirrored to Diffusion - https://phabricator.wikimedia.org/T324151 (10Aklapper) p:05Triage→03Medium
[17:35:02] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Unable to preview video in Phabricator task comments and descriptions - https://phabricator.wikimedia.org/T309222 (10Aklapper) p:05Triage→03Medium
[17:42:42] <wikibugs>	 10Phabricator, 10WMDE-TechWish-Maintenance, 10Technical-Debt: Kill Phab's sprint.phragile-uri config setting - https://phabricator.wikimedia.org/T275188 (10Aklapper) 05Open→03Resolved This somehow disappeared. https://phabricator.wikimedia.org/config/edit/sprint.phragile-uri/ says nowadays that it is unk...
[17:45:02] <wikibugs>	 10Phabricator, 10Code-Review-Workgroup, 10Developer Productivity, 10Epic: Determine options to improve work flow between Phabricator and Gerrit - https://phabricator.wikimedia.org/T232656 (10Aklapper)
[17:45:04] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10User-MModell: On Phabricator workboard, show status of associated Gerrit patches - https://phabricator.wikimedia.org/T215148 (10Aklapper)
[17:47:57] <wikibugs>	 10Release-Engineering-Team (Seen), 10Sustainability (Incident Followup): Analyze and amend (if necessary) workflow of user reporting and detecting large regressions/outages - https://phabricator.wikimedia.org/T219589 (10Aklapper) Removing #Phabricator as there doesn't seem anything actionable regarding Phabric...
[17:49:59] <wikibugs>	 10Phabricator, 10translatewiki.net, 10I18n: Adding sicilian language (scn) - https://phabricator.wikimedia.org/T299694 (10Aklapper) p:05Triage→03Low
[17:50:07] <wikibugs>	 10Phabricator, 10translatewiki.net, 10I18n: Slovenian language in Phabricator - https://phabricator.wikimedia.org/T318762 (10Aklapper) p:05Triage→03Low
[17:50:50] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Make assignees or setters of a past Due Date on a non-closed task more aware of that task - https://phabricator.wikimedia.org/T293734 (10Aklapper) p:05Triage→03Medium
[17:50:59] <wikibugs>	 10Phabricator: Due Date stamp rendering requires "deadline" task subtype; does not show for other task subtypes though Due Date field is filled - https://phabricator.wikimedia.org/T310188 (10Aklapper) p:05Triage→03Low
[17:51:16] <wikibugs>	 10Phabricator, 10Security-Team, 10SecTeam-Processed, 10Security: Audit members of acl*security for more than 12 months of no activity (May 2024) - https://phabricator.wikimedia.org/T337305 (10Aklapper) p:05Triage→03Low
[17:52:27] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Seen): Explore Phabricator → Slack integration - https://phabricator.wikimedia.org/T301189 (10Aklapper) p:05Triage→03Low Would {T321790} be relevant?
[17:54:03] <wikibugs>	 10Phabricator (Upstream), 10Upstream: "AphrontDuplicateKeyQueryException" when trying to drag a task from (sub-project) column to other column on parent workboard - https://phabricator.wikimedia.org/T139396 (10Aklapper) p:05Triage→03Low
[17:54:06] <wikibugs>	 10Phabricator, 10Tools: Publicly log account bans made using the phab-ban tool - https://phabricator.wikimedia.org/T200856 (10Aklapper) p:05Triage→03Low
[17:55:06] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Upload popup overflows box with a long filename - https://phabricator.wikimedia.org/T316305 (10Aklapper) p:05Triage→03Low
[18:01:06] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Incorrect height of class "phui-oi-frame" - https://phabricator.wikimedia.org/T307492 (10Aklapper) 05Open→03Stalled > Avatar inside menu shown incompletely.  @Stang: Is this still an issue? Could you attach a screenshot please?  > `+  min-height: 36px;`  Cannot reprodu...
[18:02:54] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Ask before leaving a Create Task form - https://phabricator.wikimedia.org/T301751 (10Aklapper) p:05Triage→03Low
[18:04:35] <wikibugs>	 10Phabricator (Upstream), 10Release-Engineering-Team (Radar), 10Developer Productivity, 10Upstream, 10User-brennen: Add Open Graph support to Phabricator Maniphest Tasks to have link preview on Telegram, Slack, and other messaging apps - https://phabricator.wikimedia.org/T288117 (10Aklapper) p:05Triage...
[18:05:30] <wikibugs>	 10Phabricator (phabricator-next): Less restrictive l image policy for drag and drop images for users or anons - https://phabricator.wikimedia.org/T333213 (10Aklapper) p:05Triage→03Low
[18:06:02] <wikibugs>	 10Phabricator, 10translatewiki.net: Remove trailing whitespace from Phabricator message documentation - https://phabricator.wikimedia.org/T294754 (10Aklapper) p:05Triage→03Low
[18:06:13] <wikibugs>	 10Phabricator: Phabricator failed to generate thumbnails for some 800-900KB files - https://phabricator.wikimedia.org/T290959 (10Aklapper) p:05Triage→03Low
[18:07:00] <wikibugs>	 10Release-Engineering-Team (Radar), 10SRE, 10SRE-Access-Requests: Requesting access to release-engineering for aklapper - https://phabricator.wikimedia.org/T341749 (10Dzahn) @Aklapper Feel free to try ssh to these hosts now. phab1004.eqiad.wmnet is prod phab, phab-test1001.eqiad.wmnet is the test machine, ph...
[18:09:01] <wikibugs>	 10Release-Engineering-Team (Radar), 10SRE, 10SRE-Access-Requests: Requesting access to release-engineering for aklapper - https://phabricator.wikimedia.org/T341749 (10Dzahn) 05Open→03Resolved a:03cmooney @cmooney Thanks! I confirm access should work now. No need to keep it open, looks all resolved to m...
[18:10:41] <wikibugs>	 10Phabricator, 10Upstream: Some task notifications include a @mention entry in the X-Phabricator-Stamps mail header for no obvious reason - https://phabricator.wikimedia.org/T266328 (10Aklapper) Should look into  `src/applications/transactions/editor/PhabricatorApplicationTransactionEditor.php` with its `publi...
[18:15:35] <wikibugs>	 10Release-Engineering-Team (Radar), 10SRE, 10SRE-Access-Requests: Requesting access to release-engineering for aklapper - https://phabricator.wikimedia.org/T341749 (10Dzahn) @Aklapper This came with a bunch of new access for you. Have fun with:  - deployment-docker - contint-docker - gerrit-root - deployment...
[18:24:13] <jgleeson>	 Hello beautiful release engineer people!!! I hope you are all doing well :) We had a patch merged on dev-images earlier today that we're keen to get it published to the wmf docker registry and I'm wondering do people usually write a phab task for that or just try to charm designated conint group members on IRC? (hence the attempted charm at the beginning)
[18:25:02] <jgleeson>	 the patch that was merged is here https://gitlab.wikimedia.org/repos/releng/dev-images/-/merge_requests/45
[18:31:34] <mutante>	 looks like this repo doesnt have a pipeline set up yet
[18:31:48] <mutante>	 but if it did, then it could auto build and publish the images in the future
[18:32:28] <mutante>	 possibly you can copy a .gitlab-ci.yaml and .pipeline dir from another repo, slightly adjust it and it would work
[18:32:59] <mutante>	 not sure about "manual push to registry" but others around here know all this much better
[19:13:36] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 2 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Reedy)
[19:14:00] <andrewbogott>	 dduvall: Docker is running my script which does "cd $DEPLOY_DIR/horizon && /usr/bin/python3 setup.py install" which does quite a few things but then fails with "error: could not create '/usr/lib/python3.9/site-packages': Permission denied"
[19:14:20] <andrewbogott>	 I would've expected the build process to be able to write to the thing it's building...
[19:14:52] <andrewbogott>	 The stage that launches the script looks like:
[19:14:54] <andrewbogott>	 https://www.irccloud.com/pastebin/ZX2QB3Gi/
[19:16:05] <mutante>	 andrewbogott: I saw you mentioned python 3.9 and the other day I saw that this is WIP: https://phabricator.wikimedia.org/T289222
[19:16:11] <mutante>	 which is about adding 3.9 to CI
[19:16:25] <mutante>	 makes me think maybe it's denied because it's just not there yet
[19:17:37] <andrewbogott>	 mutante: I don't think I've got as far as using that yet, I'm running this locally
[19:17:55] <andrewbogott>	 with base: docker-registry.wikimedia.org/python3-bullseye:latest
[19:18:02] <andrewbogott>	 isn't 3.9 standard in bullseye?
[19:18:11] <mutante>	 oh! local build environmnet. gotcha, yea, disregard that then
[19:18:38] <andrewbogott>	 but let me doublecheck that the parent dir exists...
[19:20:00] <andrewbogott>	 yeah, container contains e.g. /usr/local/lib/python3.9/dist-packages/virtualenv/__main__.py before the stage that fails
[19:26:19] <dduvall>	 andrewbogott: build processes can write to the application directory (by default `/srv/app` but whatever `lives.in` is configured as) and `/opt/lib`
[19:26:50] <dduvall>	 for python, it uses `/opt/lib/python/site-packages` as the site packages dir
[19:27:58] <dduvall>	 https://gitlab.wikimedia.org/repos/releng/blubber/-/blob/main/config/python.go#L13
[19:28:04] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Release-Engineering-Team (Radar), 10Security Team AppSec, and 4 others: Evaluate and confirm potential licensing issues for gitlab appsec pipeline tools - https://phabricator.wikimedia.org/T304737 (10sbassett)
[19:28:37] <andrewbogott>	 dduvall: ok, let me try changing the prefix.  But... shouldn't setup.py know where things go?
[19:29:43] <dduvall>	 andrewbogott: can you paste the entire blubber.yaml?
[19:29:45] <wikibugs>	 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Security: Implement an outdated modules check for golang - https://phabricator.wikimedia.org/T309997 (10sbassett)
[19:30:35] <andrewbogott>	 https://www.irccloud.com/pastebin/EDUpZyL5/
[19:32:34] <bd808>	 andrewbogott: I don't think setup.py would know. When blubber runs pip it adds `--target /opt/lib/python/site-packages` to the command. https://gitlab.wikimedia.org/repos/releng/blubber/-/blob/main/config/python.go#L160
[19:32:59] <andrewbogott>	 ok, adding that may be fixing things, still testing
[19:33:07] <andrewbogott>	 yeah, seems to work!
[19:33:11] * andrewbogott tries it everywhere
[19:33:16] <bd808>	 progress!
[19:33:20] <dduvall>	 blubber's python builder will set up `PYTHONPATH` as `/opt/lib/python/site-packages` but i don't know if setup.py will respect that
[19:33:59] <bd808>	 setup.py is the worst installer ever conceived, so probably not :)
[19:34:24] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Security: Design and Build Application Security Pipeline Components for Gitlab - https://phabricator.wikimedia.org/T289290 (10sbassett)
[19:34:45] <wikibugs>	 10GitLab, 10Gitlab-Application-Security-Pipeline, 10Release-Engineering-Team (Done by Fri 03 Sep), 10Security Team AppSec, and 3 others: Create Security Team group within gitlab.wikimedia.org - https://phabricator.wikimedia.org/T289292 (10sbassett)
[19:35:05] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Security: Create initial proof of concept application security pipeline repository - https://phabricator.wikimedia.org/T289293 (10sbassett)
[19:35:23] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Release-Engineering-Team (Doing), 10Security Team AppSec, and 4 others: Migrate existing proof-of-concept node ci templates to slim node wm node docker images - https://phabricator.wikimedia.org/T294306 (10sbassett)
[19:35:40] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, and 2 others: Research and design basic ci processing scripts (to exit 1 for tools that report errors and generate report artifacts) - https://phabricator.wikimedia.org/T294307 (10sbassett)
[19:35:47] <taavi>	 the blubberfile isn't actually using the python builder at that point, it's shelling out to something
[19:36:00] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, and 2 others: Finish node/npm initial tool ci templates for npm audit (Node 10, 12, 14) - https://phabricator.wikimedia.org/T294309 (10sbassett)
[19:36:14] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, and 2 others: Finish for node/npm initial tool ci templates for npm outdated (Node 10, 12, 14) - https://phabricator.wikimedia.org/T294310 (10sbassett)
[19:37:24] <dduvall>	 taavi: it looks like andrewbogott is using both. the python builder is defined in the `base-python` variant and `base-python` is included by `prep` which additionally defines the `builder` builder
[19:39:42] <dduvall>	 there's a new syntax as well to make ordering more clearly deterministic if that's helpful
[19:39:46] <dduvall>	 https://www.irccloud.com/pastebin/9n5FZ4US/
[19:40:19] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, and 3 others: Finish node/npm initial tool ci templates for auditjs (Node 10, 12, 14) - https://phabricator.wikimedia.org/T294311 (10sbassett)
[19:40:25] <dduvall>	 it's functionally the same in this case since the `python` builder is processed prior to `builder` but we added the new syntax to make things more controllable
[19:40:44] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Security: Investigate SAST template options now included with Gitlab CE and formulate use-cases and documentation - https://phabricator.wikimedia.org/T294312 (10sbassett)
[19:41:09] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, and 2 others: Add minimal yaml file linting as part of ci for security ci templates repository - https://phabricator.wikimedia.org/T294596 (10sbassett)
[19:41:19] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, and 2 others: Create a simple tool to check for bidirectional unicode characters - https://phabricator.wikimedia.org/T295333 (10sbassett)
[19:41:36] <andrewbogott>	 oh great now setup.py is installing things in /opt/lib/python/site-packages/lib/python3.9/site-packages/
[19:41:43] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, and 2 others: Create best practices / guidelines documentation for Gitlab application security ci templates - https://phabricator.wikimedia.org/T295374 (10sbassett)
[19:41:58] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Release-Engineering-Team (Yak Shaving 🐃🪒), 10Security Team AppSec, and 3 others: Confirm with releng which docker images make the most sense to use - https://phabricator.wikimedia.org/T296805 (10sbassett)
[19:42:20] <wikibugs>	 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Security, 10user-sbassett: Find optimal solution for projects with nested package.json files (within the context of Gitlab CI) - https://phabricator.wikimedia.org/T296806 (10sbassett)
[19:42:41] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, and 2 others: Create semgrep initial tool ci template - https://phabricator.wikimedia.org/T297991 (10sbassett)
[19:42:50] <wikibugs>	 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Security, 10user-sbassett: Create local-php-security-checker/php-security-checker ci yaml template - https://phabricator.wikimedia.org/T301828 (10sbassett)
[19:43:01] <wikibugs>	 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10SecTeam-Processed, 10Security: Create composer outdated ci yaml template - https://phabricator.wikimedia.org/T301829 (10sbassett)
[19:43:27] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Incorrect height of class "phui-oi-frame" - https://phabricator.wikimedia.org/T307492 (10Stang) @Aklapper {F37138269}
[19:43:29] <wikibugs>	 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10SecTeam-Processed, and 2 others: Create safety (w/ poetry support) ci yaml template - https://phabricator.wikimedia.org/T301830 (10sbassett)
[19:43:40] <wikibugs>	 10Gitlab-Application-Security-Pipeline, 10Release-Engineering-Team, 10Security Team AppSec, 10Security-Team, and 2 others: Create nancy ci yaml template - https://phabricator.wikimedia.org/T301831 (10sbassett)
[19:43:46] <wikibugs>	 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10SecTeam-Processed, and 2 others: Create phan/phan-taint-check plugin port as ci template - https://phabricator.wikimedia.org/T301832 (10sbassett)
[19:44:33] <wikibugs>	 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10SecTeam-Processed, and 2 others: Create python bandit ci template - https://phabricator.wikimedia.org/T301833 (10sbassett)
[19:44:35] <dduvall>	 andrewbogott: :/ i know little about `setup.py` behavior but what flag are you passing?
[19:44:41] <andrewbogott>	 --install-path seems to work better than --prefix
[19:44:48] <wikibugs>	 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10SecTeam-Processed, 10Security: Create gosec ci yaml template - https://phabricator.wikimedia.org/T301834 (10sbassett)
[19:44:49] <dduvall>	 i see `--install-lib` as maybe an option
[19:44:50] <andrewbogott>	 or rather --install-lib
[19:44:50] <dduvall>	 ah, ok
[19:45:03] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, and 2 others: Merge templates that are part of initial release to production branch - https://phabricator.wikimedia.org/T303482 (10sbassett)
[19:46:33] <andrewbogott>	 but the next thing I need to do is install things in /etc which... that has me back to having permission issues
[19:46:34] <dduvall>	 yeah we assumed pip usage when we implemented the python directive and then bd808 added support for poetry later on, but never thought to have bare easy_install support baked in
[19:46:42] <andrewbogott>	 unless /etc isn't where config goes in this context
[19:46:43] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, and 2 others: Research best ways to pass certain credentials to external services within application security-related ci templates - https://phabricator.wikimedia.org/T295508 (10sbassett) 05Open→...
[19:46:47] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Security: Design and Build Application Security Pipeline Components for Gitlab - https://phabricator.wikimedia.org/T289290 (10sbassett)
[19:47:44] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Security: Design and Build Application Security Pipeline Components for Gitlab - https://phabricator.wikimedia.org/T289290 (10sbassett)
[19:47:45] <dduvall>	 andrewbogott: typically we have people put everything other than external dependencies installed by npm/pip in the application directory
[19:48:21] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, and 2 others: Better support branches and add support for mw core to the phan-taint-check gitlab appsec template - https://phabricator.wikimedia.org/T305083 (10sbassett)
[19:48:33] <dduvall>	 heading out to lunch but i can help more in a bit
[19:48:37] <wikibugs>	 10Gitlab-Application-Security-Pipeline, 10Security Risk Management, 10Security Team AppSec, 10Security-Team, and 2 others: Create Risk Rating Calculator for Security Reviews / Gitlab AppSec CI - https://phabricator.wikimedia.org/T293138 (10sbassett)
[19:48:54] <bd808>	 containerizing 3rd party things with blubber can be a bit of extra "fun"
[19:49:31] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Security: Support conda environments with a new gitlab appsec ci template - https://phabricator.wikimedia.org/T305732 (10sbassett)
[19:49:42] <andrewbogott>	 dduvall: ok but if I'm running upstream libraries that expect to look in /etc... I need to rewrite them?
[19:49:44] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, 10Security: Create osv.dev ci includes - https://phabricator.wikimedia.org/T307514 (10sbassett)
[19:50:08] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, and 3 others: Write a phame / tech blog post detailing the appsec pipeline, as it currently exists - https://phabricator.wikimedia.org/T307517 (10sbassett)
[19:50:16] <wikibugs>	 10GitLab (CI & Job Runners), 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team, and 2 others: Investigate container scanning options within the context of the Gitlab appsec pipeline - https://phabricator.wikimedia.org/T307523 (10sbassett)
[19:50:52] <bd808>	 andrewbogott: we can do the trick of mounting an external file into the running container at /etc/whatever
[19:52:57] <andrewbogott>	 will that work if the files are in /srv/app but just need to be mounted in /etc to be found?  Mounting from one place inside the container to another place inside?
[19:54:12] <bd808>	 hmmmm... I don't know how to do that, no.
[19:56:40] <bd808>	 andrewbogott: is /etc/openstack-dashboard where you need to write things?
[19:57:27] <andrewbogott>	 just a second, let me see if i can configure where it looks for these files
[19:57:54] <bd808>	 I was looking for the same thing :)
[19:58:48] <andrewbogott>	 ok, as long as I put all the policy files in the same place I can change where it looks for them
[19:59:28] <andrewbogott>	 so I'll shove them all in /srv/app/policies and then we can change localsettings.py to look there
[19:59:36] <andrewbogott>	 my script can write to /srv/app right?
[19:59:46] <bd808>	 yes
[20:02:24] <andrewbogott>	 hmmm actually this may not help
[20:02:35] <andrewbogott>	 but also I'm not 100% sure those files are used, so I'll sweep this under the rug for now
[20:11:53] <bd808>	 I can't actually see where `/etc/openstack-dashboard` comes from in the horizon source repo. It's only mentioned in docs and tests. I guess it must be one of the requirements libraries that somehow makes that the default config override location?
[20:12:59] <dduvall>	 Yeah blubber was designed to containerize [mw] services in a consistent way and it stretched a bit to be more general purpose but it’s still lacking for building third party and base images
[20:20:17] <andrewbogott>	 bd808: I could be wrong that horizon looks in /etc and all, there's some fancy symlinking which I will try to do away with in this new build
[20:21:07] <bd808>	 andrewbogott: it's all over the docs as the suggested local config directory, I just haven't figured out where that is defined in code
[20:27:33] <andrewbogott>	 ok, the build completes now although I'm confident it won't do much. Do you think I should bake the apache config into the container, or leave it puppetized and mount it?  (I guess this is another case of 'how to write to /etc')
[20:30:30] <bd808>	 apache can be told to look elsewhere for config, but it really would be much easier if we could trick blubber into letting us actually install into /etc. I had a similar issue with the syntaxhighlight variant for shellbox
[20:31:15] <bd808>	 It was /usr/bin that I really wanted to write to there instead of /etc, but same idea.
[20:31:55] <mutante>	 fwiw, when we containerized an apache to put misc microsites into kubernetes we did it in this way that the ServerRoot is /srv/app.  It still loads modules from /etc/apache2 but those configs are installed by debian packages. The custom config is in /srv/ and the entrypoint.sh looks like:
[20:32:00] <mutante>	 /usr/sbin/apache2 -d /srv/app -f /srv/app/apache2.conf -DFOREGROUND -k start
[20:32:38] <mutante>	 that way we did not have to write to /etc and you could possibly copy some of it
[20:33:36] <andrewbogott>	 ok, I understand at least some of that
[20:35:12] <mutante>	 andrewbogott: maybe this is helpful: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/container/miscweb/+/refs/heads/master
[20:35:32] <mutante>	 there is en entrypoint.sh and a .pipeline dir with a blubber file
[20:35:39] <mutante>	 for an apache running out of srv
[20:35:56] <mutante>	 https://gerrit.wikimedia.org/r/plugins/gitiles/operations/container/miscweb/+/refs/heads/master/.pipeline/blubber.yaml
[20:36:25] <mutante>	 it uses the "httpd" imageas base and build on that
[20:36:48] <andrewbogott>	 mutante: where is your apache.conf?
[20:37:18] <mutante>	 https://gerrit.wikimedia.org/r/plugins/gitiles/operations/container/miscweb/+/refs/heads/master/production/httpd.conf
[20:37:33] <mutante>	 https://gerrit.wikimedia.org/r/plugins/gitiles/operations/container/miscweb/+/refs/heads/master/entrypoint.sh
[20:50:37] <andrewbogott>	 mutante: does this look semi-credible?  https://gitlab.wikimedia.org/repos/cloud/cloud-vps/horizon/deploy/-/commit/6c33f5be17b2863e20fe21329dbcde041ae70f01
[20:52:07] <mutante>	 andrewbogott: yea, it does. mostly. except one thing I think:
[20:52:09] <mutante>	 ErrorLog /var/log/apache2/horizon_error.log
[20:52:21] <andrewbogott>	 oh yeah
[20:52:24] <mutante>	 these lines.. because that also tries to write
[20:52:36] <mutante>	 so I only have:
[20:52:36] <mutante>	 ErrorLog /dev/stderr
[20:52:37] <mutante>	 ErrorLog /proc/self/fd/2
[20:52:41] <mutante>	 because that avoids it
[20:52:42] <andrewbogott>	 I also assume that the serverroot/docroot lines don't do anything and can be removed?
[20:52:46] <mutante>	 and somehow that gets into logstash
[20:53:20] <mutante>	 I am not sure about that, I think you need to set ServerRoot 
[20:53:27] <mutante>	 but use /srv
[20:53:30] <mutante>	 /srv/app that is
[20:53:45] <mutante>	 we also copied this from something else that already existed
[20:53:49] <mutante>	 /srv/app is somehow standard
[20:54:08] <andrewbogott>	 I didn't mean to merge horizon.conf at all, just apache.conf
[20:54:36] <andrewbogott>	 so, um, ignore that file and then imagine I asked my question again
[20:55:41] <mutante>	 you can merge both files into just one single config
[20:55:59] <andrewbogott>	 well I don't think there's anything in horizon.conf that's needed
[20:56:01] <mutante>	 when you start apache you start with .. -f /srv/app/apache2.conf
[20:56:08] <andrewbogott>	 at least, I believe all the useful bits to be in apache.conf
[20:57:01] <mutante>	 you will need the <VirtualHost> and ServerName
[20:57:29] <mutante>	 you can include one file in the other or just merge them into one. that's just how you prefer it 
[20:57:39] <mutante>	 for apache internally it just concats all that anyways
[20:58:22] <andrewbogott>	 ...
[21:00:30] <andrewbogott>	 if ServerName is baked in there then I'll need two different builds for labtesthorizon and horizon :(
[21:00:57] <mutante>	 yea, when a request comes in it will try to match the data in headers to the ServerName
[21:01:06] <mutante>	 but you can add them both
[21:01:12] <andrewbogott>	 Maybe apache doesn't work like this, I was hoping to configure apache along the lines of "if you get a request, then this is you!" and no vhost checking
[21:01:17] <mutante>	 ServerAlias 
[21:01:30] <mutante>	 You can have one ServerName and 10 ServerAlias
[21:01:33] <mutante>	 it will just work 
[21:01:51] <mutante>	 a server configured that way will respond to both 
[21:03:50] <mutante>	 the mediawiki appserver configs have a ServerAlias for each wiki all at once
[21:04:14] <andrewbogott>	 yep, but can't you also configure apache without vhosts at all and have it just be, like, one actual host?
[21:04:37] <mutante>	 yea, you can turn off virtual hosts altogether
[21:04:43] <mutante>	 then you need one IP for each website though
[21:05:10] <mutante>	 not sure if that matters anymore with the containers.. yea
[21:05:24] <mutante>	 but also not sure if you make your life easier by not using this very common setup
[21:05:39] <andrewbogott>	 I feel like in a container once traffic hits the container the hostname and source IP shouldn't matter at all
[21:05:44] <andrewbogott>	 yeah, that's a fair point
[21:06:26] <mutante>	 maybe install docker locally
[21:06:40] <mutante>	 then you can pull from WMF docker registry to local laptop
[21:06:46] <mutante>	 and run that standard httpd image
[21:06:49] <mutante>	 without any changes
[21:06:54] <mutante>	 and get a shell in the running container
[21:06:57] <mutante>	 and try it all out
[21:08:22] <mutante>	 yea, so one question would be "in the httpd image that we should build on.. does it enable the virtual host part"
[21:09:13] <mutante>	 because if that's in the default config and comes from the distro package.. it seems easier to use it rather than change "upstream"
[21:10:03] <mutante>	 also for apache questions in general, #httpd here on IRC is excellent. that guy "thumbs" does good support and has been forever
[21:10:27] <mutante>	 will know internal details like which config lines are really required and which are not
[21:11:32] <andrewbogott>	 good to know!
[21:11:47] <andrewbogott>	 I'm trying to see what my container will run now, will be a while before it gets as far as actually loading a page though
[21:11:56] <andrewbogott>	 thanks for your help!
[21:12:01] <andrewbogott>	 (and also thanks to dduvall and bd808 )
[21:12:35] <mutante>	 If you omit the ServerName directive from any name-based virtual host, the server will default to a fully qualified domain name (FQDN) derived from the system hostname.
[21:12:40] <mutante>	 ^ dont think you want that here
[21:14:25] <mutante>	 if it has "NameVirtualHost .." in the global config would decide whether you need VirtualHost and ServerName
[21:26:08] <bd808>	 I think it would be "most correct" in an Apache container to use port-based vhosts, but not name-based vhosts. That's `<VirtualHost *:8080>...</VirtualHost>` and you just make sure that you only use each port number once in the combined config for the server.
[21:29:37] <wikibugs>	 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Direct 1% of all traffic to mw-on-k8s - https://phabricator.wikimedia.org/T341463 (10Quiddity) Thanks for the draft, appreciated! I've [[https://meta.wikimedia.org/wiki/Tech/News/2023/29#Tech_News:_2023-29|added this to Tech Ne...
[21:33:18] <Daimona>	 Folks, has there been a change to CI in the last ~1 hour that might have broken things?
[21:33:27] <Daimona>	 Looks like something switched to contint2002 
[21:34:53] <Daimona>	 Hmmm actually, it's not as recent as I thought and the failures seem to be pre-existing
[21:39:03] <mutante>	 the switch from contint2001 to contint2002 happened 2 days ago
[21:39:26] <mutante>	 that was a hardware replacement where data was copied over but software versions should have stayed the same
[21:41:14] <Daimona>	 Yeah, I remember reading about it, that's why I was supposed to see some failures now
[21:41:22] <Daimona>	 I'm talking about this patch in particular: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/937559
[21:42:02] <Daimona>	 CI doesn't really "fail" in the usual sense, but the output of the wmf-quibble-core-vendor-mysql-php74-docker job is different in PS2 and PS3
[21:42:21] <Daimona>	 For the "PHPUnit unit tests" section in particular, PS2 works and runs all tests as expected
[21:42:49] <Daimona>	 But in PS3 it doesn't run any tests because it can't find LocalSettings.php
[21:42:57] <Daimona>	 And the only difference between PS2 and PS3 is a comment
[21:43:25] <Daimona>	 Now I just made another change that shouldn't really change much, and it's working again (PS9)
[21:44:31] <Daimona>	 When checking the diff of other similar failures I noticed that one mentioned contint1002 and the other one contint2002, but here PS2 and PS3 both mention contint2002
[21:46:46] <Daimona>	 Oooooooh it's because quibble parallelizes these steps!
[21:51:15] <andrewbogott>	 dduvall, bd808, one more docker question before day's end: what should I do about uids? Does apache still run as httpd? Will my 'horizon' service user still exist for running uwsgi scripts? Or is everything squashed down to one user and group?
[21:53:17] <bd808>	 andrewbogott: everything is one user and group. and the runtime user/group is different than the buildtime user/group.
[21:53:33] <dduvall>	 andrewbogott: if you're cramming it all into one container, it's going to be one process owner
[21:53:57] <dduvall>	 for apache and backend applications that speak over a socket, i think best practice is probably to run them in separate containers
[21:54:04] <dduvall>	 and build a specialized image for each
[21:54:30] <bd808>	 dduvall: in this case, Apache is the uwsgi container 
[21:55:20] <bd808>	 We are doing the slimest port of what we have done on bare metal into a container and that means using mod_wsgi
[21:55:35] <dduvall>	 oh, is it mod_uwsgi?
[21:55:40] <dduvall>	 ah
[21:55:46] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Performance-Team, 10Sustainability (MediaWiki-MultiDC): Performance Q2 2017/18 goal: Install and use mcrouter in deployment-prep - https://phabricator.wikimedia.org/T151466 (10Krinkle)
[21:55:51] <dduvall>	 then yeah, same process, same process owner
[21:57:56] <andrewbogott>	 great, what is that process named? I ask for the sake of "WSGIDaemonProcess horizon user=horizon group=horizon processes=3 threads=10 python-path=/opt/lib/python/site-packages display-name=horizon"
[22:00:40] <bd808>	 andrewbogott: I think that "runuser" is the runtime username. UID=900, GID=900. This is from looking at a blubber generated dockerfile I had laying around
[22:01:02] <andrewbogott>	 great, I'll try it
[22:01:13] <bd808>	 but I think you can basically drop the user/group params too
[22:03:21] <andrewbogott>	 seems like progress, now I'm at 'Permission denied: mod_wsgi (pid=8): Couldn't bind unix domain socket '/var/run/apache2/wsgi.8.0.1.sock''
[22:03:34] <andrewbogott>	 which could be a user/group thing or could be I'm just not allowed to write to /var/run at all
[22:05:07] <bd808>	 I'd bet on /var/run permissions.
[22:06:00] <bd808>	 andrewbogott: in the container I have that I think is close to your base, /var/run is 0755 root:root.
[22:06:27] <bd808>	 So you'll need to convince mod_wsgi to put the socket somewhere else
[22:06:37] <andrewbogott>	 Is there a tranditional place to put runtime files?  Just /tmp?
[22:07:24] <bd808>	 I think /tmp would work. Looks like the config is https://modwsgi.readthedocs.io/en/master/configuration-directives/WSGISocketPrefix.html
[22:08:05] <andrewbogott>	 yep
[22:08:46] <bd808>	 I think you can ignore the stuff on that page about /tmp being bad. Containers are different
[22:12:59] <bd808>	 dduvall: https://wikitech.wikimedia.org/wiki/Blubber/Pipeline could use a see also to kokkuri
[22:13:53] <dduvall>	 off, i'd like to get rid of that article
[22:13:58] <dduvall>	 *oof
[22:13:59] <dduvall>	 :)
[22:14:22] <dduvall>	 it's references a very old set of statically defined jenkins jobs prior to pipelinelib existing
[22:14:32] <dduvall>	 *referencing (geez. hands)
[22:15:10] <dduvall>	 but yeah, https://wikitech.wikimedia.org/wiki/PipelineLib should probably link to kokkuri
[22:16:26] <dduvall>	 https://wikitech.wikimedia.org/wiki/Blubber needs some serious cleanup and rewriting. i just wrote a bunch of acceptance tests in *gasp* cucumber that should make for better blubber examples. it's in review
[22:17:00] <andrewbogott>	 Hey, I got a 404 page!  counting that as a victory
[22:17:54] <dduvall>	 sounds like incremental progress!
[22:37:49] <wikibugs>	 10Release-Engineering-Team, 10Collection, 10SRE, 10Traffic, and 3 others: Strange error pattern noticed on viwiki during unrelated deploy - https://phabricator.wikimedia.org/T340850 (10Krinkle) 05Open→03Resolved a:03Reedy No entries in WMF Logstash for this after July 1st.  {F37138383 height=200}
[22:38:46] <wikibugs>	 10Release-Engineering-Team, 10Collection, 10SRE, 10Traffic, and 3 others: Strange error pattern noticed on viwiki during unrelated deploy - https://phabricator.wikimedia.org/T340850 (10Krinkle)
[22:59:50] <wikibugs>	 10Phabricator, 10DBA, 10Data-Persistence-Backup, 10collaboration-services, 10Patch-For-Review: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10brennen) I can access the db and do a dry run for the Phorge migrations:  ` brennen@phab-test1001:/srv/phab/phabr...
[23:12:08] <wikibugs>	 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T340245 (10dduvall) 05Open→03Resolved
[23:20:26] <wikibugs>	 10Phabricator, 10DBA, 10Data-Persistence-Backup, 10collaboration-services, 10Patch-For-Review: phabricator->phorge migration - database handling - https://phabricator.wikimedia.org/T335080 (10brennen) ` mysql:phuser@db1118.eqiad.wmnet [phabricator_project]> show slave status\G ***************************...
[23:26:33] <wikibugs>	 10GitLab, 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review: WMCS GitLab runners running frequently running out of disk space - https://phabricator.wikimedia.org/T340887 (10dduvall) I'm still unable to repro this issue locally, and according to upstream the ref counting is solely in memory...