[00:18:35] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 05Release, 05Train Deployments: 1.44.0-wmf.13 deployment blockers - https://phabricator.wikimedia.org/T382364#10490943 (10thcipriani) 05Openβ†’03Resolved [01:04:52] (03merge) 10lwatson: Releases: Bump Codex to 1.20.0 and Vue to 3.5.13 [repos/ci-tools/libup-config] - 10https://gitlab.wikimedia.org/repos/ci-tools/libup-config/-/merge_requests/58 (owner: 10egardner) [09:15:05] 10Gerrit: Allow OAuth login to Gerrit - https://phabricator.wikimedia.org/T48146#10491420 (10ExE-Boss) β†’14Duplicate dup:03T147864 [09:15:07] 10Gerrit: Support OAuth for login onto gerrit.wikimedia.org - https://phabricator.wikimedia.org/T147864#10491422 (10ExE-Boss) [10:22:04] (03PS2) 10Sohom Datta: zuul: Add MolecularPilot to CI allowlist [integration/config] - 10https://gerrit.wikimedia.org/r/1113739 [10:48:22] 10Phabricator, 10CAS-SSO, 06Infrastructure-Foundations: Phabricator should use IDP for developer account logins - https://phabricator.wikimedia.org/T377061#10491654 (10Aklapper) [11:35:31] 06Project-Admins: Create project tag for title normalizing - https://phabricator.wikimedia.org/T379989#10491820 (10Peachey88) [12:09:50] ugh, I did a bad thing on gerrit [12:10:01] can someone please create the %help branch in mediawiki/extensions/EntitySchema.git for me :( [12:10:07] no [12:10:10] s/create/delete/ [12:10:17] I accidentally created the branch but cannot delete it [12:10:35] (idk why – https://gerrit.wikimedia.org/r/admin/repos/mediawiki/extensions/EntitySchema,branches shows me β€œdelete” buttons for some of the branches but not others) [12:33:51] (03PS1) 10Arthur taylor: Update list of phpunit config files to copy to log directory [integration/quibble] - 10https://gerrit.wikimedia.org/r/1113983 (https://phabricator.wikimedia.org/T378797) [12:39:37] (03CR) 10CI reject: [V:04-1] Update list of phpunit config files to copy to log directory [integration/quibble] - 10https://gerrit.wikimedia.org/r/1113983 (https://phabricator.wikimedia.org/T378797) (owner: 10Arthur taylor) [12:40:28] (03CR) 10Arthur taylor: "recheck" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1113983 (https://phabricator.wikimedia.org/T378797) (owner: 10Arthur taylor) [12:42:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:42:36] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T384706 (10phaultfinder) 03NEW [12:46:34] Project beta-code-update-eqiad build #532097: 04FAILURE in 3 min 33 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/532097/ [12:48:44] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit1003) - https://phabricator.wikimedia.org/T384706#10492041 (10Jelto) [12:52:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [12:52:44] (03CR) 10Arthur taylor: "recheck" [integration/quibble] - 10https://gerrit.wikimedia.org/r/1113983 (https://phabricator.wikimedia.org/T378797) (owner: 10Arthur taylor) [12:55:03] Yippee, build fixed! [12:55:03] Project beta-code-update-eqiad build #532098: 09FIXED in 2 min 3 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/532098/ [13:01:34] Lucas_WMDE: done [13:25:25] taavi: thanks <3 [15:25:08] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit1003) - https://phabricator.wikimedia.org/T384706#10492493 (10Jelto) Two Digital Ocean IPs scraped gerrit and have throttled automatically. I blocked permanently with the change above. The traffic stopped after that. There were also quite a... [15:51:13] 10Beta-Cluster-Infrastructure: /api/rest_v1/page/summary/ is not working in Beta Cluster - https://phabricator.wikimedia.org/T384719 (10Diskdance) 03NEW [16:00:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [16:00:37] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T384721 (10phaultfinder) 03NEW [16:02:45] hashar: given that we don't even support RSA anymore on the CDN (dropped it after seeing <0.1% global usage) I think it's safe to remove it in gerrit as well [16:05:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [16:06:31] FIRING: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [16:06:35] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T384721#10492683 (10phaultfinder) [16:11:31] RESOLVED: [2x] ProbeDown: Service gerrit1003:443 has failed probes (http_gerrit_tls_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#gerrit1003:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [16:50:42] 10Phabricator, 06collaboration-services: Experiencing Phabricator 429 errors - https://phabricator.wikimedia.org/T383435#10492979 (10LSobanski) 05Openβ†’03Declined As @Dzahn stated before, using cloud IPs to access our services implies a risk of throttling. [16:51:28] 10Release-Engineering-Team (Doing 😎), 10wikimedia.biterg.io: Bitergia account for Levi Ferreira - https://phabricator.wikimedia.org/T384727 (10thcipriani) 03NEW [16:51:31] 10Release-Engineering-Team (Doing 😎), 10wikimedia.biterg.io: Bitergia account for Levi Ferreira - https://phabricator.wikimedia.org/T384727#10492995 (10thcipriani) p:05Triageβ†’03Medium [16:56:03] 10Release-Engineering-Team (Doing 😎), 10wikimedia.biterg.io: Bitergia account for Levi Ferreira - https://phabricator.wikimedia.org/T384727#10493020 (10thcipriani) [17:22:15] 06Release-Engineering-Team, 06Security-Team: Clarify what is the security patch deployment procedure - https://phabricator.wikimedia.org/T384439#10493107 (10sbassett) I haven't done a security deploy in a while, but the process I'd still use on a deployment host would be a `git apply --check`, then a `git am`... [17:24:57] (03PS1) 10Jgleeson: Revert "Zuul: Switch Fundraising jobs to REL1_43" [integration/config] - 10https://gerrit.wikimedia.org/r/1114025 [17:26:17] (03CR) 10CI reject: [V:04-1] Revert "Zuul: Switch Fundraising jobs to REL1_43" [integration/config] - 10https://gerrit.wikimedia.org/r/1114025 (owner: 10Jgleeson) [17:45:01] (03update) 10lwatson: JobLog: move console log above change info [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/642 (https://phabricator.wikimedia.org/T383855) [17:48:32] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap (SpiderPig πŸ•ΈοΈ), 06collaboration-services, 13Patch-For-Review: Scap SpiderPig: Routing for the web frontend - https://phabricator.wikimedia.org/T383946#10493174 (10thcipriani) >>! In T383946#10486564, @Dzahn wrote: > Once this is done you will need a v... [17:56:19] 06Release-Engineering-Team, 06Data-Engineering, 06Data-Platform-SRE, 10Dumps-Generation, and 4 others: WE 5.4 KR - Hypothesis 5.4.4 - Q3 FY24/25 - Migrate current-generation dumps to run on kubernetes - https://phabricator.wikimedia.org/T352650#10493191 (10BTullis) I have had some more thoughts about how t... [18:21:08] (03close) 10lwatson: JobLog: move console log above change info [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/642 (https://phabricator.wikimedia.org/T383855) [18:34:57] oh man [18:36:55] (03CR) 10Jgleeson: "We might need to revert this temporarily to get this passing https://gerrit.wikimedia.org/r/c/integration/config/+/1114025" [integration/config] - 10https://gerrit.wikimedia.org/r/1113585 (owner: 10Jforrester) [18:37:53] (03CR) 10Jgleeson: "and this :(" [integration/config] - 10https://gerrit.wikimedia.org/r/1113584 (owner: 10Jforrester) [18:39:12] fr-tech, wfan: it looks like it's not as simple as reverting one patch to bring back our old CI. the revert patch is failing due to the old job builds no longer existing :| they were removed here https://gerrit.wikimedia.org/r/c/integration/config/+/1113584/2 and I also realised there's a bigger patch chain which was added to upgrade our build to 8.2 that might not be as easy to roll back! [18:39:12] https://gerrit.wikimedia.org/r/c/integration/config/+/1113583/1 [18:39:21] sorry wrong channel! [18:39:40] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap (SpiderPig πŸ•ΈοΈ), 06collaboration-services: Scap SpiderPig: Routing for the web frontend - https://phabricator.wikimedia.org/T383946#10493334 (10Dzahn) After the merge above https://spiderpig.wikimedia.org now points to the deployment server. This is why... [18:41:53] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap (SpiderPig πŸ•ΈοΈ), 06collaboration-services: Scap SpiderPig: Routing for the web frontend - https://phabricator.wikimedia.org/T383946#10493350 (10Dzahn) If you want to get this going first in beta, you'll want to click in Horizon to get a domain proxy `htt... [18:52:44] 10Release-Engineering-Team (Priority Backlog πŸ“₯), 10Scap (SpiderPig πŸ•ΈοΈ), 06collaboration-services: Scap SpiderPig: Routing for the web frontend - https://phabricator.wikimedia.org/T383946#10493413 (10Dzahn) I think the next step needed is some envoy config, adding a snippet to `/etc/envoy/listeners.d` for spi... [19:13:29] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - https://phabricator.wikimedia.org/T384721#10493462 (10phaultfinder) [19:17:04] (03open) 10dancy: Update spiderpig local dev docker-compose config (v2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/651 [19:17:07] (03update) 10dancy: Update spiderpig local dev docker-compose config (v2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/651 [19:21:14] (03update) 10dancy: Update spiderpig local dev docker-compose config (v2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/651 [19:21:31] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - moscovium (RT) - https://phabricator.wikimedia.org/T384721#10493496 (10Dzahn) [19:21:46] (03update) 10dancy: Update spiderpig local dev docker-compose config (v2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/651 [19:21:50] 06Release-Engineering-Team, 06collaboration-services: ProbeDown - moscovium (RT) - https://phabricator.wikimedia.org/T384721#10493499 (10Dzahn) 05Openβ†’03Declined [19:25:13] 06Release-Engineering-Team, 06collaboration-services: ProbeDown (gerrit1003) - https://phabricator.wikimedia.org/T384706#10493522 (10Dzahn) 05Openβ†’03Resolved a:03Dzahn [19:27:14] 10GitLab (Infrastructure), 06collaboration-services, 13Patch-For-Review: GitLab OpenSSL 3 upgrade in 17.7 - https://phabricator.wikimedia.org/T379598#10493532 (10Dzahn) gitlab-ce 17.7 is now available in WMF APT but has not been installed yet. [20:07:59] (03approved) 10lwatson: Update spiderpig local dev docker-compose config (v2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/651 (owner: 10dancy) [20:08:21] (03merge) 10dancy: Update spiderpig local dev docker-compose config (v2) [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/651 [21:01:06] 10Scap (SpiderPig πŸ•ΈοΈ), 10Design-System-Team (DST-Sprint-39 (2025-01-06 to 2025-01-17)): SpiderPig UI: move the console log - https://phabricator.wikimedia.org/T383855#10493777 (10lwatson) 05In progressβ†’03Declined [21:06:01] 10Scap (SpiderPig πŸ•ΈοΈ), 10Design-System-Team (DST-Sprint-39 (2025-01-06 to 2025-01-17)): SpiderPig UI: move the console log - https://phabricator.wikimedia.org/T383855#10493786 (10lwatson) The console log positioning is still being discussed and we'll wait for further user feedback. [21:30:19] (03PS1) 10Jforrester: Revert "Zuul: Switch Fundraising jobs to REL1_43" [integration/config] - 10https://gerrit.wikimedia.org/r/1114068 [21:30:32] (03PS2) 10Jgleeson: Revert "Zuul: Switch Fundraising jobs to REL1_43" [integration/config] - 10https://gerrit.wikimedia.org/r/1114025 [21:31:02] (03PS1) 10Jforrester: Revert "jjb: Drop now-unused Fundraising REL1_39 jobs" [integration/config] - 10https://gerrit.wikimedia.org/r/1114069 [21:31:20] (03Abandoned) 10Jforrester: Revert "Zuul: Switch Fundraising jobs to REL1_43" [integration/config] - 10https://gerrit.wikimedia.org/r/1114068 (owner: 10Jforrester) [21:31:26] (03PS3) 10Jgleeson: Revert "Zuul: Switch Fundraising jobs to REL1_43" [integration/config] - 10https://gerrit.wikimedia.org/r/1114025 [21:31:56] (03Abandoned) 10Jforrester: Revert "Zuul: Switch Fundraising jobs to REL1_43" [integration/config] - 10https://gerrit.wikimedia.org/r/1114025 (owner: 10Jgleeson) [21:31:59] (03Restored) 10Jforrester: Revert "Zuul: Switch Fundraising jobs to REL1_43" [integration/config] - 10https://gerrit.wikimedia.org/r/1114068 (owner: 10Jforrester) [21:32:09] (03PS2) 10Jforrester: Revert "Zuul: Switch Fundraising jobs to REL1_43" [integration/config] - 10https://gerrit.wikimedia.org/r/1114068 [21:35:16] jgleeson: To confirm, you want to REL1_39 jobs back, but do you also want the REL1_43 jobs? [21:35:51] (03CR) 10Jforrester: [C:03+2] "Deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/1114069 (owner: 10Jforrester) [21:37:03] (03Merged) 10jenkins-bot: Revert "jjb: Drop now-unused Fundraising REL1_39 jobs" [integration/config] - 10https://gerrit.wikimedia.org/r/1114069 (owner: 10Jforrester) [21:42:27] (03CR) 10Jforrester: "I'll deploy this for now." [integration/config] - 10https://gerrit.wikimedia.org/r/1114068 (owner: 10Jforrester) [21:42:31] (03CR) 10Jforrester: [C:03+2] Revert "Zuul: Switch Fundraising jobs to REL1_43" [integration/config] - 10https://gerrit.wikimedia.org/r/1114068 (owner: 10Jforrester) [21:43:47] (03Merged) 10jenkins-bot: Revert "Zuul: Switch Fundraising jobs to REL1_43" [integration/config] - 10https://gerrit.wikimedia.org/r/1114068 (owner: 10Jforrester) [21:44:09] !log Revert "Zuul: Switch Fundraising jobs to REL1_43" [21:44:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [21:49:08] (03open) 10lwatson: JobCard: prevent event bubbling and fix highlighted card [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/652 (https://phabricator.wikimedia.org/T383835) [21:56:09] 10Scap (SpiderPig πŸ•ΈοΈ), 13Patch-For-Review: SpiderPig UI: address user interface feedback - https://phabricator.wikimedia.org/T383835#10493850 (10lwatson) [22:37:48] (03CR) 10Arlolra: [C:03+2] Hide cite number brackets for frwiki & eswiki [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1113880 (owner: 10Subramanya Sastry) [22:38:23] (03Merged) 10jenkins-bot: Hide cite number brackets for frwiki & eswiki [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1113880 (owner: 10Subramanya Sastry) [22:43:36] (03update) 10dancy: spiderpig: CAS auth integration [repos/releng/scap] - 10https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/625