[06:13:59] (03open) 10phedenskog: Add daily per-job medians to the core feedback dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/38 [06:14:06] (03merge) 10phedenskog: Add daily per-job medians to the core feedback dashboard [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/38 [07:11:36] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244160 (10Xqt) p:05Triage→03Unbreak! [13:24:53] All CI jobs are stuck. Any updates on T435699? [13:24:54] T435699: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699 [14:53:25] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor, 06SRE: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244373 (10Mahveotm) [15:05:34] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor, 06Collaboration-Services, 06SRE: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244415 (10Xqt) [15:17:36] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor, 06Collaboration-Services, 06SRE: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244441 (10cscott) All jenkins jobs have been stuck for 22 hours at this point. Granted, it's over... [17:07:27] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: CI seems to have not done anything for 24H - https://phabricator.wikimedia.org/T435731 (10Reedy) 03NEW [17:07:57] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: CI seems to have not done anything for 24H - https://phabricator.wikimedia.org/T435731#12244544 (10Reedy) p:05Triage→03Unbreak! [17:09:09] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team: CI seems to have not done anything for 24H - https://phabricator.wikimedia.org/T435731#12244552 (10Reedy) →14Duplicate dup:03T435699 [17:09:12] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor, 06Collaboration-Services, 06SRE: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244549 (10Reedy) [21:14:58] 10Beta-Cluster-Infrastructure: Beta cluster is being crawled to death by bot traffic coming from residential proxies - https://phabricator.wikimedia.org/T420833#12244804 (10AlexisJazz) >>! In T420833#12243359, @tstarling wrote: > I deployed Produnto to beta, but for the last 24 hours it's been too slow to verify... [21:31:37] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor, 06Collaboration-Services, 06SRE: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244833 (10Krinkle) ... clicking through shows: > integration-castor06 is offline > {F99733172 hei... [21:35:07] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor, 06Collaboration-Services, 06SRE: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244835 (10Les4353) Per https://integration.wikimedia.org/ci/computer/integration%2Dcastor06/ It se... [21:36:27] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor, 10Cloud-Services, and 3 others: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244837 (10Les4353) The #Cloud-Services project tag is not intended to have any tasks. Please check th... [21:37:14] 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Castor, 06cloud-services-team, and 2 others: Completion of castor-save-workspace-cache stucks - https://phabricator.wikimedia.org/T435699#12244839 (10Les4353) [21:38:26] 10Continuous-Integration-Infrastructure, 10Castor, 06cloud-services-team, 10Cloud-VPS (Quota-requests), 06Test Platform (Nairobi 29): Provide more disk I/O headroom for the Castor cache volume (integration-castor06) - https://phabricator.wikimedia.org/T435571#12244841 (10Les4353) p:05Triage→03High Pe... [22:25:49] (03PS1) 10Subramanya Sastry: Add a new config to compare mathjax.v2 server vs clientside rendering [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1328378 [23:01:34] (03PS2) 10Subramanya Sastry: Add a new config to compare mathjax.v2 server vs clientside rendering [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1328378 [23:01:34] (03PS1) 10Subramanya Sastry: Add a new config to compare mathjax.v2 vs mathjax.v4 rendering [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1328379 [23:14:07] (03PS3) 10Subramanya Sastry: Add a new config to compare mathjax.v2 server vs clientside rendering [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1328378 [23:14:07] (03PS2) 10Subramanya Sastry: Add a new config to compare mathjax.v2 vs mathjax.v4 rendering [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1328379 [23:18:31] (03PS3) 10Subramanya Sastry: Add a new config to compare mathjax.v2 vs mathjax.v4 rendering [integration/visualdiff] - 10https://gerrit.wikimedia.org/r/1328379 [23:26:35] 10GitLab (Auth & Access), 10Phabricator, 06Release-Engineering-Team (Radar), 06cloud-services-team, and 2 others: Look for ways to consolidate "we trust this human" access lists - https://phabricator.wikimedia.org/T364516#12244994 (10Nirmos)