[00:00:38] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:05:39] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [00:24:46] thcipriani: https://gerrit.wikimedia.org/r/c/mediawiki/extensions/OAuth/+/973247 could put wikitech back on the right train position if anyone has the energy to do that today. [00:25:58] it can certainly wait for Monday too. [00:47:42] bd808: nice! It's a little late for me to try to get it out the door + the roll forward. jnuche is not part of the US holiday and may be able to get it out tomorrow. Would a good test to run be: try to log in to wikitech and if it works then success? [00:49:09] thcipriani: https://horizon.wikimedia.org/ and https://toolsadmin.wikimedia.org/ were the logins that stopped working (along with stashbot) [00:49:53] or doing anything on wikitech with OAuth auth, but mostly humans don't do that [00:50:35] the other things broke because the talk to wikitech's Action API using OAuth to do what they do [00:51:14] ok, so, the steps to test would be (1) deploy patch (2) roll forward (3) login to horizon and if that works then we're done. Otherwise, revert rollforward and we're at the same state as now? [00:51:48] thcipriani: yeah, that sounds like a fair plan [00:53:30] ok, as long as rollback will put us back to a working state, seems like something we could try in the European morning. [00:54:11] yeah, t.aavi is probably around in EU morning too if jnuche wants help testing [00:55:05] okie doke. I'll update the task with steps. I'll leave it with jaime is he wants to defer 'til Monday when more folks are around. [00:56:22] <3 thanks for doing what y'all do [00:58:21] <3 for the work on the fix, looked like it required a lot of digging from a lot of smart folks. [01:00:28] I have a hunch that t.gr would have realized it was a session recursion bug pretty quickly, but matmarex got there once I remembered how to get debug output somewhere folks could see it. :) [01:03:47] 10Release-Engineering-Team (Quid Pro Crow 🦃), 10Patch-For-Review, 10Release, 10Train Deployments: 1.42.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T350080 (10thcipriani) Late Thursday notes for @jnuche There's a fix for wikitech, but it came in a little late in the day Thursday to sli... [01:59:08] 10Phabricator (Upstream), 10Upstream: Give transparent embedded image files in Phabricator a checkered background - https://phabricator.wikimedia.org/T78644 (10valerio.bozzolan) Ah, since somebody asked (?) yes, this feature is in this upstream changelog: https://we.phorge.it/w/changelog/2023.32/ [06:41:47] 10GitLab: Gitlab attempts to reach gitlab CDN when using the Web IDE - https://phabricator.wikimedia.org/T350931 (10Vgutierrez) [07:55:36] 10Release-Engineering-Team (Quid Pro Crow 🦃), 10Patch-For-Review, 10Release, 10Train Deployments: 1.42.0-wmf.4 deployment blockers - https://phabricator.wikimedia.org/T350080 (10jnuche) @thcipriani since it looks like there's no hurry to get the fix out, I'd rather wait until Monday so @bd808 and anyone el... [08:13:59] 10GitLab, 10Privacy: Gitlab attempts to reach gitlab CDN when using the Web IDE - https://phabricator.wikimedia.org/T350931 (10Peachey88) [08:21:56] 10Phabricator maintenance bot, 10API Platform, 10Content-Transform-Team, 10MediaWiki-Engineering: Stop adding Platform Engineering to new wiki creation tasks - https://phabricator.wikimedia.org/T346811 (10daniel) >>! In T346811#9318953, @Ladsgroup wrote: > That being said, this is not about creating new wi... [09:26:50] (03CR) 10Kosta Harlan: [C: 03+1] CoreComposerTest: Remove ancient PHP file extensions (031 comment) [integration/quibble] - 10https://gerrit.wikimedia.org/r/883885 (owner: 10Lucas Werkmeister (WMDE)) [09:28:41] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Uninstall Harbormaster (Phabricator application) - https://phabricator.wikimedia.org/T348115 (10Aklapper) @dzahn: I prefer to go one by one application (caution in case something blows up I was not aware of), and I prefer do such things at the beg... [09:44:44] 10GitLab, 10Privacy: Gitlab attempts to reach gitlab CDN when using the Web IDE - https://phabricator.wikimedia.org/T350931 (10Aklapper) That's the vscode asset: ` $:acko\> cd gitlab-web-ide-0.0.1-dev-20230921201626/ $:acko\> grep -r "cdn.web-ide" . ./packages/vscode-bootstrap/src/start.ts: 'https://{{uui... [09:56:43] 10GitLab (Pipeline Services Migration🐤), 10Release-Engineering-Team, 10MediaWiki-Docker, 10dev-images, and 3 others: MySQL/MariaDB images for development environments - https://phabricator.wikimedia.org/T238925 (10kostajh) a:05kostajh→03None [10:10:53] 10Continuous-Integration-Infrastructure, 10Quibble, 10Developer Productivity, 10MW-1.40-notes (1.40.0-wmf.26; 2023-03-06), and 3 others: Provide early feedback when a patch has job failures - https://phabricator.wikimedia.org/T323750 (10kostajh) 05Open→03Resolved [10:15:54] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services: Migrate to using new GitLab CI runner authentication scheme - https://phabricator.wikimedia.org/T344951 (10Jelto) One Shared Runner `runner-1021.gitlab-runners.eqiad1.wikimedia.cloud` and one Trusted Runne... [10:17:03] 10GitLab (CI & Job Runners), 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services: Migrate to using new GitLab CI runner authentication scheme - https://phabricator.wikimedia.org/T344951 (10Jelto) [10:29:26] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Technical-Debt: Remove custom WMF Phab downstream hacks superseded by upstream Phorge changes - https://phabricator.wikimedia.org/T341799 (10Aklapper) [10:37:08] 10GitLab (Project Migration), 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services, and 3 others: Migrate active repositories in Phabricator Differential to GitLab - https://phabricator.wikimedia.org/T191182 (10Aklapper) R2344: With boilerplate content containing terms l... [10:42:46] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10User-brennen: Clean up Phabricator production error logs - https://phabricator.wikimedia.org/T337500 (10Aklapper) Given the recurring `git cat-file` exceptions in our error log, I merged https://we.phorge.it/D25460 in upstream. Once pulled, th... [12:11:38] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Create a high-level deployment pipeline overview - https://phabricator.wikimedia.org/T349799 (10KBach) [12:14:03] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Create a high-level deployment pipeline overview - https://phabricator.wikimedia.org/T349799 (10KBach) Updated task description to reflect the current focus of this work. I pivoted away from writing an intermediate document and decided to... [12:23:23] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Deployment pipeline (GitLab/Kokkuri/Blubber) documentation improvements - https://phabricator.wikimedia.org/T342317 (10KBach) The overview document (T349799) is still in progress, as my approach to it has shifted and formulating the plan... [12:23:49] 10Release-Engineering-Team (Radar), 10Tech-Docs-Team, 10Documentation: Deployment pipeline (GitLab/Kokkuri/Blubber) documentation improvements - https://phabricator.wikimedia.org/T342317 (10KBach) [12:28:19] 10GitLab (Project Migration), 10Traffic, 10Patch-For-Review: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10LSobanski) @BCornwall For operations/software/varnish, looks like it should just be archived and not migrated? Let me know if that's the case. [14:02:25] 10GitLab (Pipeline Services Migration🐤), 10collaboration-services: move design.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T350791 (10Jelto) >>! In T350791#9321205, @Dzahn wrote: > Yea, style-guide is deployed by scap and other parts are not. This is because there were large files in there... [14:21:09] (03PS10) 10Hashar: Plugin to process Puppet Catalog Compiler results [software/gerrit] (deploy/wmf/stable-3.5) - 10https://gerrit.wikimedia.org/r/969981 [14:30:05] 10Continuous-Integration-Config, 10Release-Engineering-Team (Radar), 10ChangeProp, 10Data-Engineering, and 2 others: Run EventBus tests in MediaWiki core CI - https://phabricator.wikimedia.org/T257583 (10lbowmaker) [14:30:35] 10Release-Engineering-Team (Radar), 10Data-Engineering, 10Event-Platform: Stop using puppet + git pull for auto deployment of schema repos - https://phabricator.wikimedia.org/T274901 (10lbowmaker) [14:31:58] 10Beta-Cluster-Infrastructure, 10Data-Engineering, 10Event-Platform, 10MW-1.41-notes (1.41.0-wmf.12; 2023-06-06): cirrusSearchCheckerJob JobQueueErrors (Could not enqueue jobs) on Beta Cluster - https://phabricator.wikimedia.org/T322491 (10lbowmaker) [14:58:15] (03CR) 10DCausse: [C: 03+1] zuul: comment Blazegraph is sticking to Java 8 [integration/config] - 10https://gerrit.wikimedia.org/r/973088 (https://phabricator.wikimedia.org/T350587) (owner: 10Hashar) [15:41:59] (03CR) 10Hashar: [C: 03+2] zuul: comment Blazegraph is sticking to Java 8 [integration/config] - 10https://gerrit.wikimedia.org/r/973088 (https://phabricator.wikimedia.org/T350587) (owner: 10Hashar) [15:43:11] (03Merged) 10jenkins-bot: zuul: comment Blazegraph is sticking to Java 8 [integration/config] - 10https://gerrit.wikimedia.org/r/973088 (https://phabricator.wikimedia.org/T350587) (owner: 10Hashar) [15:47:09] 10Phabricator (Upstream), 10Release-Engineering-Team, 10Upstream, 10User-brennen: Hovercards: (RuntimeException) Undefined index: objectPHID - https://phabricator.wikimedia.org/T346876 (10Aklapper) One case in our logs so far: ` [2023-11-03 01:21:15] 'RequestURI: /search/hovercard/?cards=%7B%22phid-user-tl... [15:52:08] 10Phabricator, 10Release-Engineering-Team, 10collaboration-services, 10User-AKlapper: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708 (10acooper) I was assuming finding the person would be easy - it would be maybe their manager? Which is easily av... [15:54:20] 10Phabricator, 10Release-Engineering-Team, 10collaboration-services, 10User-AKlapper: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708 (10acooper) Oh and all that is required is for the verifier to also post a message authenticated to their account... [15:59:03] 10Phabricator, 10Release-Engineering-Team, 10collaboration-services, 10User-AKlapper: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708 (10acooper) You could also require messages of acknowledgement to be sent across at least two authentication doma... [16:03:05] (03Abandoned) 10Hashar: Add WikibaseClient-only job for ArticlePlaceholder [integration/config] - 10https://gerrit.wikimedia.org/r/438239 (owner: 10WMDE-leszek) [16:04:12] (03Abandoned) 10Hashar: dockerfiles: Remove hardcoded references to suite.xml and phpunit.php [integration/config] - 10https://gerrit.wikimedia.org/r/803487 (https://phabricator.wikimedia.org/T90875) (owner: 10Kosta Harlan) [16:04:33] (03Abandoned) 10Hashar: Zuul: Enforce Postgres and SQLite support for all MW tarball code [integration/config] - 10https://gerrit.wikimedia.org/r/797319 (owner: 10Jforrester) [16:15:28] (03CR) 10Hashar: [C: 03+2] "Deploying so we can test :)" [integration/config] - 10https://gerrit.wikimedia.org/r/973083 (https://phabricator.wikimedia.org/T350587) (owner: 10Hashar) [16:16:41] (03Merged) 10jenkins-bot: Add Java 11 jobs to some search/* repositories [integration/config] - 10https://gerrit.wikimedia.org/r/973083 (https://phabricator.wikimedia.org/T350587) (owner: 10Hashar) [16:35:06] (03CR) 10Hashar: [C: 03+2] "I have send dummy patches for each of the four repositories:" [integration/config] - 10https://gerrit.wikimedia.org/r/973083 (https://phabricator.wikimedia.org/T350587) (owner: 10Hashar) [16:36:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Patch-For-Review: Remove Java 8 images from integration/config - https://phabricator.wikimedia.org/T350587 (10hashar) I have send dummy patches for each of the four repositories: https://gerrit.wikimedia.org/r/c/search/extra/+/973355 and... [18:09:30] 10Phabricator, 10Release-Engineering-Team, 10collaboration-services, 10User-AKlapper: Establish a workflow that scales for requesting Phab 2FA resets - https://phabricator.wikimedia.org/T306708 (10Dzahn) >>! In T306708#9323224, @acooper wrote: > I was assuming finding the person would be easy - it would be... [18:10:33] 10GitLab (Pipeline Services Migration🐤), 10collaboration-services: move design.wikimedia.org to kubernetes - https://phabricator.wikimedia.org/T350791 (10Dzahn) Yay, I am glad to hear this @Volker_E Yea, let's have a kickoff :) [18:11:21] 10GitLab (Project Migration), 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥), 10collaboration-services, and 3 others: Migrate active repositories in Phabricator Differential to GitLab - https://phabricator.wikimedia.org/T191182 (10Dzahn) Cool :) thank you! @Aklapper [18:12:03] 10Phabricator, 10Release-Engineering-Team (Priority Backlog 📥): Uninstall Harbormaster (Phabricator application) - https://phabricator.wikimedia.org/T348115 (10Dzahn) > beginning of a week. fair enough !:)