[00:05:35] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [07:37:10] Dreamy_Jazz: for the SonarQube coverage issue on the master branch of extension, you can file a task in Phabricator against https://phabricator.wikimedia.org/project/view/6419/ and I think cc `pwangai` [07:37:44] side thing, the per changes analyzis could be reported as pull requests instead of short lived branches https://docs.sonarsource.com/sonarqube/latest/analyzing-source-code/pull-request-analysis/ [07:38:07] and we use Sonar Scanner 4.6.0 which looks outdated :) [07:42:25] (03CR) 10Noa wmde: [C: 03+1] "Or we can just try it and see if it makes a difference for the query builder job :)" [integration/config] - 10https://gerrit.wikimedia.org/r/961109 (owner: 10Hashar) [08:07:36] Thanks! [08:24:21] (03PS2) 10Driedmueller: Added integration config for BlueSpiceInterWikiSearch and others [integration/config] - 10https://gerrit.wikimedia.org/r/961392 [08:32:01] I have file dthe pull request analysis as https://phabricator.wikimedia.org/T347548 [08:32:17] Dreamy_Jazz: and you can file another task for the issue you are encountering :) [08:34:56] (03PS2) 10Hashar: jjb: use a job template for wikidata-query-*-build [integration/config] - 10https://gerrit.wikimedia.org/r/961109 [08:45:37] (03CR) 10Hashar: [C: 03+2] jjb: use a job template for wikidata-query-*-build (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/961109 (owner: 10Hashar) [08:50:25] I filed https://phabricator.wikimedia.org/T347407 [09:37:37] I have added a few things to the task but I don't quite understand what is going on :-\ [09:38:36] Thanks. The coverage seems to be also failing on CheckUser for individual changes. [09:39:05] But only for non-release branches [09:40:29] For example https://sonarcloud.io/component_measures?metric=new_coverage&selected=mediawiki-extensions-CheckUser%3Asrc%2FCheckUser%2FPagers%2FCheckUserGetEditsPager.php&view=list&branch=961127-2&id=mediawiki-extensions-CheckUser says no coverage when unit tests exist [09:42:01] Whereas a change on REL1_39 has unit test coverage reported https://sonarcloud.io/summary/new_code?id=mediawiki-extensions-CheckUser&branch=961263-15 [09:42:13] I'll add that detail to the task [09:49:37] and I noticed in the console PHPUnit fails due to lack of LocalSettings.php [09:49:43] so I guess something has changed in mediawiki/core [10:17:15] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, 10serviceops: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [10:47:48] (03PS1) 10Kosta Harlan: zuul: Require DiscussionTools in phan deps for ReportIncident [integration/config] - 10https://gerrit.wikimedia.org/r/961748 (https://phabricator.wikimedia.org/T340138) [10:49:17] (03CR) 10Hashar: [C: 03+2] zuul: Require DiscussionTools in phan deps for ReportIncident [integration/config] - 10https://gerrit.wikimedia.org/r/961748 (https://phabricator.wikimedia.org/T340138) (owner: 10Kosta Harlan) [10:50:35] (03Merged) 10jenkins-bot: zuul: Require DiscussionTools in phan deps for ReportIncident [integration/config] - 10https://gerrit.wikimedia.org/r/961748 (https://phabricator.wikimedia.org/T340138) (owner: 10Kosta Harlan) [10:55:15] (03CR) 10Hashar: [C: 03+2] "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/961748 (https://phabricator.wikimedia.org/T340138) (owner: 10Kosta Harlan) [10:55:28] 10Phabricator, 10TestMe: Phorge logs me out if accessing a task via URL - https://phabricator.wikimedia.org/T347570 (10MarcoAurelio) [11:02:45] (03CR) 10Kosta Harlan: zuul: Require DiscussionTools in phan deps for ReportIncident (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/961748 (https://phabricator.wikimedia.org/T340138) (owner: 10Kosta Harlan) [11:19:27] 10Phabricator, 10TestMe: Phorge logs me out if accessing a task via URL - https://phabricator.wikimedia.org/T347570 (10Aklapper) I cannot reproduce this. I assume this is reproducible? Does this also happen with another browser? [11:54:03] (03PS1) 10Hnowlan: jjb: add editor-analytics, edit-analytics and page-analytics [integration/config] - 10https://gerrit.wikimedia.org/r/961777 (https://phabricator.wikimedia.org/T336391) [11:55:53] (03PS1) 10Hnowlan: zuul: add edit-analytics and page-analytics [integration/config] - 10https://gerrit.wikimedia.org/r/961778 (https://phabricator.wikimedia.org/T336391) [12:24:21] 10Diffusion, 10Phabricator, 10Release-Engineering-Team: Understand which repositories we mirror, observe, host in Diffusion (and fix some findings) - https://phabricator.wikimedia.org/T347577 (10Aklapper) p:05Triage→03Low [12:30:16] 10Diffusion, 10Phabricator, 10Release-Engineering-Team: Understand which repositories we mirror, observe, host in Diffusion (and fix some findings) - https://phabricator.wikimedia.org/T347577 (10Urbanecm) I disabled the GitHub pushing URIs, those are legacy items that are no longer needed. [12:31:19] 10Diffusion, 10Phabricator, 10Release-Engineering-Team: Understand which repositories we mirror, observe, host in Diffusion (and fix some findings) - https://phabricator.wikimedia.org/T347577 (10Mbch331) @Aklapper The whole repo can be deleted. Tool isn't in use anymore. [12:38:54] (DatasourceError) firing: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [12:43:49] (DatasourceError) resolved: Queue (Jenkins jobs + Zuul functions) alert - https://grafana.wikimedia.org/alerting/grafana/iS0FSjJ4z/view - https://wikitech.wikimedia.org/wiki/Monitoring/DatasourceError - https://alerts.wikimedia.org/?q=alertname%3DDatasourceError [13:24:35] 10Release-Engineering-Team: logspam-watch doesn’t handle normalized exceptions well - https://phabricator.wikimedia.org/T347064 (10Lucas_Werkmeister_WMDE) IMHO the underlying issue (that unnormalized messages ended up in the log file) should be fixed, but it’s difficult to investigate. [14:18:50] (03PS1) 10Kosta Harlan: zuul: Require DiscussionTools for ReportIncident [integration/config] - 10https://gerrit.wikimedia.org/r/961816 (https://phabricator.wikimedia.org/T340138) [14:20:10] (03CR) 10Hashar: [C: 03+2] zuul: Require DiscussionTools for ReportIncident [integration/config] - 10https://gerrit.wikimedia.org/r/961816 (https://phabricator.wikimedia.org/T340138) (owner: 10Kosta Harlan) [14:21:59] (03Merged) 10jenkins-bot: zuul: Require DiscussionTools for ReportIncident [integration/config] - 10https://gerrit.wikimedia.org/r/961816 (https://phabricator.wikimedia.org/T340138) (owner: 10Kosta Harlan) [14:25:11] !log integration: resizing `integration-castor05` from `g3.cores8.ram36.disk20` to `g3.cores8.ram36.disk20.4xiops` ( +4xiops) # T345924 [14:25:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:25:15] T345924: Move castor instance to 4xiops disk flavor - https://phabricator.wikimedia.org/T345924 [14:26:12] and I guess that is how I broke CI entirely :/ [14:26:33] oh no [14:27:14] 10GitLab (Infrastructure), 10collaboration-services: Record metrics for Gitlab backup/restore times - https://phabricator.wikimedia.org/T347593 (10eoghan) [14:27:26] ah no it is booting [14:29:51] Invalid input received: Invalid volume: Volume 3f90c3f2-158d-4e45-a919-0f048f47c3b6 status must be available or downloading to reserve, but the current status is attaching. (HTTP 400) (Request-ID: req-ddd07558-b6b7-4ec6-8258-c4e5efb83a07) [14:29:53] * hashar whistles [14:31:30] so yeah essentially I broke the instance [14:34:38] (03PS1) 10Hashar: jjb: disable castor-save-workspace-cache [integration/config] - 10https://gerrit.wikimedia.org/r/961824 (https://phabricator.wikimedia.org/T345924) [14:34:59] (03CR) 10Hashar: [C: 03+2] jjb: disable castor-save-workspace-cache [integration/config] - 10https://gerrit.wikimedia.org/r/961824 (https://phabricator.wikimedia.org/T345924) (owner: 10Hashar) [14:35:31] (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/961816 (https://phabricator.wikimedia.org/T340138) (owner: 10Kosta Harlan) [14:36:51] (03Merged) 10jenkins-bot: jjb: disable castor-save-workspace-cache [integration/config] - 10https://gerrit.wikimedia.org/r/961824 (https://phabricator.wikimedia.org/T345924) (owner: 10Hashar) [14:42:52] I guess I will recreate it [14:45:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review: Move castor instance to 4xiops disk flavor - https://phabricator.wikimedia.org/T345924 (10hashar) ` lang=irc 16:29:49 <•hashar> Invalid input received: Invalid volume: Volume 3f90c3f2-158d-4e45-a... [14:46:00] (03PS1) 10Jforrester: Zuul: Add Terasail to the allow list [integration/config] - 10https://gerrit.wikimedia.org/r/961827 [14:46:34] (03CR) 10Jforrester: [C: 03+2] Zuul: Add Terasail to the allow list [integration/config] - 10https://gerrit.wikimedia.org/r/961827 (owner: 10Jforrester) [14:46:39] 10GitLab (Infrastructure), 10collaboration-services: Record metrics for Gitlab backup/restore times - https://phabricator.wikimedia.org/T347593 (10eoghan) p:05Triage→03Medium [14:48:08] (03Merged) 10jenkins-bot: Zuul: Add Terasail to the allow list [integration/config] - 10https://gerrit.wikimedia.org/r/961827 (owner: 10Jforrester) [14:48:48] !log Zuul: Add Terasail to the allow list [14:48:51] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:01:59] (03PS1) 10Hashar: Revert "jjb: disable castor-save-workspace-cache" [integration/config] - 10https://gerrit.wikimedia.org/r/961831 (https://phabricator.wikimedia.org/T345924) [15:04:15] integration-castor05.integration.eqiad1.wikimedia.cloud. 55 IN A 172.16.1.98 [15:04:15] integration-castor05.integration.eqiad1.wikimedia.cloud. 55 IN A 172.16.0.78 [15:04:20] looks like the good old bugs are still around [15:04:21] :/ [15:24:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Cloud-VPS (Quota-requests), 10castor: Request volume quota increase for integration Cloud VPS project - https://phabricator.wikimedia.org/T304080 (10hashar) As a followup, I had to recreate the instance and reattach the volume but it wa... [15:26:24] !log Rettaching integration-castor05 to Jenkins after its ssh host fingerprint got changed when I recreated the instance # T345924 [15:26:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:26:27] T345924: Move castor instance to 4xiops disk flavor - https://phabricator.wikimedia.org/T345924 [15:26:59] !log Reenabling castor-save-workspace-cache job # T345924 [15:27:02] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:28:05] (03CR) 10Hashar: [C: 03+2] Revert "jjb: disable castor-save-workspace-cache" [integration/config] - 10https://gerrit.wikimedia.org/r/961831 (https://phabricator.wikimedia.org/T345924) (owner: 10Hashar) [15:32:06] * kostajh hashar: I am still seeing phan errors on https://integration.wikimedia.org/ci/job/mwext-php74-phan-docker/72164/console, any ideas? [15:32:33] (03Merged) 10jenkins-bot: Revert "jjb: disable castor-save-workspace-cache" [integration/config] - 10https://gerrit.wikimedia.org/r/961831 (https://phabricator.wikimedia.org/T345924) (owner: 10Hashar) [15:32:59] (03CR) 10Kosta Harlan: "Something seems not quite right, see https://integration.wikimedia.org/ci/job/mwext-php74-phan-docker/72164/console" [integration/config] - 10https://gerrit.wikimedia.org/r/961748 (https://phabricator.wikimedia.org/T340138) (owner: 10Kosta Harlan) [15:39:47] (03CR) 10Kosta Harlan: zuul: Require DiscussionTools in phan deps for ReportIncident (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/961748 (https://phabricator.wikimedia.org/T340138) (owner: 10Kosta Harlan) [15:55:25] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Priority Backlog 📥), 10castor: Move castor instance to 4xiops disk flavor - https://phabricator.wikimedia.org/T345924 (10hashar) I have recreated the instance with the same hostname `integration-castor05`. Since the ssh host key changed I h... [16:24:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Priority Backlog 📥), 10castor: Move castor instance to 4xiops disk flavor - https://phabricator.wikimedia.org/T345924 (10hashar) And to complete I have added panels to the cloud grafana instance to show the number of disk IO in progress as... [17:07:06] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T345889 (10matmarex) [17:16:45] (03CR) 10Hashar: [C: 03+2] zuul: Require DiscussionTools in phan deps for ReportIncident (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/961748 (https://phabricator.wikimedia.org/T340138) (owner: 10Kosta Harlan) [17:39:11] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap, 10MediaWiki-Platform-Team (Radar): "scap backport" unable to deploy change that has abandoned sibling in another branch - https://phabricator.wikimedia.org/T345304 (10CodeReviewBot) jnuche updated https://gitlab.wikimedia.org/repos/releng/scap/-/merge_... [18:16:35] 10Diffusion, 10Phabricator, 10Release-Engineering-Team: Understand which repositories we mirror, observe, host in Diffusion (and fix some findings) - https://phabricator.wikimedia.org/T347577 (10Aklapper) [18:25:01] 10Release-Engineering-Team (Priority Backlog 📥), 10Patch-For-Review, 10Release, 10Train Deployments: 1.41.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T345889 (10dduvall) [19:06:17] 10GitLab (Project Migration), 10Traffic: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10Aklapper) [19:07:08] 10GitLab (Project Migration), 10Traffic: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10Aklapper) [19:07:16] 10GitLab (Project Migration), 10collaboration-services: Migrate SRE repositories to GitLab - https://phabricator.wikimedia.org/T341468 (10Aklapper) [19:07:33] 10GitLab (Project Migration), 10Traffic: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10Aklapper) Note that there is some potential overlap with {T341504} [19:09:40] 10GitLab (Project Migration), 10Traffic: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10BCornwall) [19:11:08] 10GitLab (Project Migration), 10Traffic: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10BCornwall) Ugh, not fond of mega-tickets. I can move it over to there if that's clearer/more useful though. [19:13:54] 10GitLab (Project Migration), 10Traffic: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10BCornwall) Oh, I see, it's broken down by namespace. Hm, I think I'll keep this one around since this is an effort specifically for traffic and it mixes a whole bunch of th... [20:25:42] 10Release-Engineering-Team (Priority Backlog 📥), 10Scap: Store MediaWiki image refs in scap history - https://phabricator.wikimedia.org/T347630 (10dduvall) [20:45:26] dancy: did you find why the "Undefined index: DEFAULT" error from mwmaint didn't show up? [20:47:26] Yeah. Brennen reminded me about https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/refs/heads/production/modules/role/files/logging/logspam.pl#196 [20:51:44] hm.. ok. wondering why, but up to y'all. seems intentinal at least :) [20:52:37] Hmm. I added that exclusion in 2021... I'm prepared to undo it now. [20:53:18] Brennen suggested a mode to toggle it in or out. [20:53:51] no objections to just removing it honestly [20:54:03] at this point it might make for a bit less confusion between systems [20:54:17] Nod.. which is my goal. [20:57:28] Krinkle: I saw 100k+ errors at cewiki re Undefined index a couple of days ago [20:57:50] They're still rolling in heavy. [20:57:51] iirc the fix landed in .27 or .28 ? [20:58:00] (errors for .27, that is) [20:58:10] presumed to be a long-running job. [20:58:41] it was in mwmaint yep, but I couldn't find in puppet any systemd that would trigger that script [20:58:59] granted, I was quick search [20:59:59] /srv/mediawiki-staging/multiversion/MWScript.php migrateLinksTable --wiki=cewiki --table pagelinks --batch-size 10000 --sleep 0.1 [21:12:21] next logspam-watch goal: con somebody into +2ing a patch for installing fzf in production and rebuild the interface so i can fuzzy-find everything [23:10:13] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.41.0-wmf.30 deployment blockers - https://phabricator.wikimedia.org/T347081 (10thcipriani) p:05Triage→03Medium a:03hashar