[00:06:24] 10Beta-Cluster-Infrastructure: UpdateImplementationsJob not being run on beta cluster - https://phabricator.wikimedia.org/T332207 (10Jdforrester-WMF) [00:06:54] 10Beta-Cluster-Infrastructure: UpdateImplementationsJob not being run on beta cluster - https://phabricator.wikimedia.org/T332207 (10Jdforrester-WMF) 05Open→03Resolved a:03TheresNoTime >>! In T332207#8700016, @TheresNoTime wrote: > Can you confirm this has been resolved? Yup, thank you so much. [05:15:02] 10Beta-Cluster-Infrastructure: UpdateImplementationsJob not being run on beta cluster - https://phabricator.wikimedia.org/T332207 (10DMartin-WMF) @TheresNoTime - My thanks, also! [07:57:22] 10Release-Engineering-Team, 10SRE, 10Wikimedia-Logstash: mediawiki-errors logstash dashboard's "errors over time" panel broken - https://phabricator.wikimedia.org/T332273 (10kostajh) [08:06:29] 10Release-Engineering-Team, 10SRE, 10SRE Observability, 10Wikimedia-Logstash: mediawiki-errors logstash dashboard's "errors over time" panel broken - https://phabricator.wikimedia.org/T332273 (10MoritzMuehlenhoff) [08:19:27] 10Phabricator: Cannot access workboard: Unhandled Exception: Undefined variable: other_assignees - https://phabricator.wikimedia.org/T332234 (10Aklapper) [08:20:25] !log github: created https://github.com/wikimedia/mediawiki-services-machinetranslation for replication from Gerrit # T331256 [08:20:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:20:29] T331256: Create Gerrit repository for /services/machinetranslation and migrate code from Gitlab - https://phabricator.wikimedia.org/T331256 [08:23:07] 10Gerrit, 10Machine-Learning-Team, 10serviceops-radar, 10Language-Team (Language-2023-January-March): Create Gerrit repository for /services/machinetranslation and migrate code from Gitlab - https://phabricator.wikimedia.org/T331256 (10hashar) I have created the GitHub repo https://github.com/wikimedia/med... [08:58:15] (03PS1) 10Hashar: Review access change [extensions/FileSystemImageServer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/900128 [08:58:48] (03PS2) 10Hashar: Add permissions to import GitHub repository [extensions/FileSystemImageServer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/900128 (https://phabricator.wikimedia.org/T240786) [08:58:56] (03CR) 10Hashar: [V: 03+2 C: 03+2] Add permissions to import GitHub repository [extensions/FileSystemImageServer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/900128 (https://phabricator.wikimedia.org/T240786) (owner: 10Hashar) [09:01:32] (03PS1) 10Hashar: Revert "Add permissions to import GitHub repository" [extensions/FileSystemImageServer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/900129 (https://phabricator.wikimedia.org/T240786) [09:01:42] (03CR) 10Hashar: [V: 03+2 C: 03+2] Revert "Add permissions to import GitHub repository" [extensions/FileSystemImageServer] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/900129 (https://phabricator.wikimedia.org/T240786) (owner: 10Hashar) [09:05:37] (03PS1) 10Hashar: Zuul: add extension FileSystemImageServer [integration/config] - 10https://gerrit.wikimedia.org/r/900201 (https://phabricator.wikimedia.org/T240786) [09:06:23] (03CR) 10Hashar: [C: 03+2] Zuul: add extension FileSystemImageServer [integration/config] - 10https://gerrit.wikimedia.org/r/900201 (https://phabricator.wikimedia.org/T240786) (owner: 10Hashar) [09:06:37] (03CR) 10CI reject: [V: 04-1] Zuul: add extension FileSystemImageServer [integration/config] - 10https://gerrit.wikimedia.org/r/900201 (https://phabricator.wikimedia.org/T240786) (owner: 10Hashar) [09:07:35] (03Abandoned) 10Hashar: Zuul: add extension FileSystemImageServer [integration/config] - 10https://gerrit.wikimedia.org/r/900201 (https://phabricator.wikimedia.org/T240786) (owner: 10Hashar) [10:09:56] 10Release-Engineering-Team (Seen), 10MW-on-K8s, 10SRE, 10Traffic, and 3 others: Serve production traffic via Kubernetes - https://phabricator.wikimedia.org/T290536 (10Clement_Goubert) [10:29:45] (03PS2) 10Hashar: dockerfiles: upgrade castor Stretch > Bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/899505 (https://phabricator.wikimedia.org/T278203) [10:29:57] (03CR) 10Hashar: [C: 03+2] dockerfiles: upgrade castor Stretch > Bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/899505 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [10:31:07] (03Merged) 10jenkins-bot: dockerfiles: upgrade castor Stretch > Bullseye [integration/config] - 10https://gerrit.wikimedia.org/r/899505 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [10:41:16] !log Switching https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ to a Bullseye based image (was Stretch based) [10:41:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:41:28] !log Switching https://integration.wikimedia.org/ci/job/castor-save-workspace-cache/ to a Bullseye based image (was Stretch based) T278203 [10:41:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:41:32] T278203: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 [10:48:12] (03CR) 10Hashar: [C: 03+2] jjb: switch jobs to releng/castor:0.3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/899508 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [10:48:38] !log Updating all jobs to switch Castor from Stretch to Bullseye based image | https://gerrit.wikimedia.org/r/c/integration/config/+/899508/ | T278203 [10:48:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [10:48:41] T278203: Migrate all CI jobs from stretch to buster or later and drop stretch testing support - https://phabricator.wikimedia.org/T278203 [10:50:37] (03Merged) 10jenkins-bot: jjb: switch jobs to releng/castor:0.3.0 [integration/config] - 10https://gerrit.wikimedia.org/r/899508 (https://phabricator.wikimedia.org/T278203) (owner: 10Hashar) [11:14:22] jnuche: You around here? I'm having a scap bootstrap issue putting new mw hosts in prod [11:14:58] Strangely it works on some, but not on others [11:15:15] Those that don't fail [11:15:17] Error: /usr/local/bin/bootstrap-scap-target.sh: line 31: scap/bin/install_local_version.sh: No such file or directory [11:15:19] [11:15:21] Error: /Stage[main]/Scap/Exec[bootstrap-scap-target]/returns: change from 'notrun' to ['0'] failed: /usr/local/bin/bootstrap-scap-target.sh: line 31: scap/bin/install_local_version.sh: No such file or directory [11:15:23] (corrective) [11:19:42] The script looks wonky [11:20:55] it does /usr/bin/rsync --archive --delay-updates --delete --delete-delay --compress --new-compress --timeout=5 --contimeout=2 deploy2002.codfw.wmnet::scap-install-staging/scap-wheels/buster deploy2002.codfw.wmnet::scap-install-staging/scap/bin/install_local_version.sh /var/lib/scap/ [11:21:06] Then cd to /var/lib/scap/ [11:21:15] And tries to execute scap/bin/install_local_version.sh [11:21:33] But the rsync puts install_local_version.sh at the root of /var/lib/scap [11:25:48] https://gerrit.wikimedia.org/r/c/operations/puppet/+/900252 [11:35:17] I'll +2 myself so I can move on, I can revert later [11:51:07] claime: back from lunch, looking [11:51:50] fix looks good, thank you [11:54:21] > Strangely it works on some, but not on others [11:54:23] if Scap already existed on the machine for whatever reason, the bootstrap script would have worked, because there is an install_local_version script included at scap/bin [11:54:35] probably also why I missed the bug [11:55:23] jnuche: Nah, it's just wonky puppet reporting it as not a failure [13:14:16] (03PS1) 10Hashar: jjb: refactor pipeline jobs definition [integration/config] - 10https://gerrit.wikimedia.org/r/900311 [13:14:37] (03CR) 10Hashar: [C: 04-1] jjb: refactor pipeline jobs definition [integration/config] - 10https://gerrit.wikimedia.org/r/900311 (owner: 10Hashar) [13:15:28] (03CR) 10CI reject: [V: 04-1] jjb: refactor pipeline jobs definition [integration/config] - 10https://gerrit.wikimedia.org/r/900311 (owner: 10Hashar) [13:25:06] 10Release-Engineering-Team (Deployment Training Requests): Deployment training request for allanjeremy.thisdot - https://phabricator.wikimedia.org/T332313 (10allanjeremy.thisdot) [13:33:35] (03CR) 10Hashar: [C: 04-1] jjb: refactor pipeline jobs definition (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/900311 (owner: 10Hashar) [14:09:58] :q [14:12:01] you can never quit vim. [14:31:28] hehehe :D [14:58:38] 10Release-Engineering-Team, 10Observability-Logging, 10SRE, 10Wikimedia-Logstash, 10SRE Observability (FY2022/2023-Q3): mediawiki-errors logstash dashboard's "errors over time" panel broken - https://phabricator.wikimedia.org/T332273 (10colewhite) The `MediaWiki errors over time by channel` visualization... [15:39:20] 10Beta-Cluster-Infrastructure: UpdateImplementationsJob not being run on beta cluster - https://phabricator.wikimedia.org/T332207 (10DMartin-WMF) @TheresNoTime - follow-up question related to this ticket: Although `UpdateImplementationsJob `is now running, our extension's other job, `CacheTesterResultsJob`, stil... [15:45:52] 10Beta-Cluster-Infrastructure: UpdateImplementationsJob not being run on beta cluster - https://phabricator.wikimedia.org/T332207 (10TheresNoTime) >>! In T332207#8702786, @DMartin-WMF wrote: > @TheresNoTime - follow-up question related to this ticket: Although `UpdateImplementationsJob `is now running, our exten... [20:54:57] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), 10Patch-For-Review, 10Release, and 2 others: 1.40.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T330205 (10brennen) [21:21:06] 10Beta-Cluster-Infrastructure: UpdateImplementationsJob not being run on beta cluster - https://phabricator.wikimedia.org/T332207 (10DMartin-WMF) @TheresNoTime - Thanks for offering to take a look! Turns out I was wrong above; in fact `CacheTesterResultsJob` //does //appear to be running. (There was some con... [22:22:33] 10Continuous-Integration-Config, 10MediaWiki-Core-Tests, 10Browser-Tests, 10Patch-For-Review, and 2 others: Make MediaWiki Wdio tests less slow (Sept 2019) - https://phabricator.wikimedia.org/T234002 (10Jdforrester-WMF) [22:32:23] 10Release-Engineering-Team (Priority Backlog 📥), 10MW-1.40-notes (1.40.0-wmf.27; 2023-03-13), 10Patch-For-Review, 10Release, and 2 others: 1.40.0-wmf.27 deployment blockers - https://phabricator.wikimedia.org/T330205 (10brennen) 05In progress→03Resolved [23:35:41] Hi. How do you stop gerrit from restoring the content of a previous edit in edit mode? [23:40:18] Solved, thanks anyway [23:44:05] I didn't understand the question, but I also felt that I know very little about editing via the gerrit UI so I didn't bother to ask for clarification.