[00:26:50] 06cloud-services-team, 10Toolforge: Can the ruby-dev package be made available on bastion and on the ruby3.3 images? - https://phabricator.wikimedia.org/T411226 (10Ijon) 03NEW [00:31:15] 06cloud-services-team, 10Toolforge: [lima-kilo] error mounting docker cache - https://phabricator.wikimedia.org/T411208#11414532 (10Raymond_Ndibe) >>! In T411208#11413945, @fnegri wrote: > @Volans ran into this issue some time ago, and I ran into it today. The workaround is using `./start-devenv.sh --no-cache`... [03:54:04] FIRING: PuppetConstantChange: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [04:28:49] RESOLVED: PuppetConstantChange: Puppet performing a change on every puppet run on cloudweb2002-dev:9100 - https://puppetboard.wikimedia.org/nodes?status=changed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetConstantChange [04:30:00] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#11414747 (10Marostegui) @taavi @fnegri clouddb1022 and clouddb1023 (both x3 and s3) are ready to be added to the LB. [04:30:11] 06cloud-services-team, 10Data-Services, 06DBA, 13Patch-For-Review: Productionize new clouddb* hosts (clouddb1022-1033) - https://phabricator.wikimedia.org/T409557#11414748 (10Marostegui) [06:38:22] 10Cloud-VPS (Project-requests): Request creation of Gutendex VPS project - https://phabricator.wikimedia.org/T411158#11414818 (10Ijon) Okay, confirmed that it runs well on MariaDB as well, so no need for a VPS. Is there anything in particular I need to do to be able to host a 500MB database on MariaDB from my t... [06:47:13] 10Cloud-VPS (Project-requests): Request creation of Gutendex VPS project - https://phabricator.wikimedia.org/T411158#11414838 (10Ijon) 05Open→03Resolved a:03Ijon Looks like I'm all set. Thanks! [07:32:31] 06cloud-services-team, 10Toolforge: Can the ruby-dev package be made available on bastion and on the ruby3.3 images? - https://phabricator.wikimedia.org/T411226#11414865 (10Ijon) Okay, it's coming back to me: I remember now that I need to use toolforge jobs. But trying to do so, and running **bundle** (Ruby eq... [07:33:01] 06cloud-services-team, 10Toolforge: Can the ruby-dev and libyaml-dev packages be made available on the ruby3.3 images? - https://phabricator.wikimedia.org/T411226#11414867 (10Ijon) [08:02:27] 06cloud-services-team, 10Toolforge: Can the ruby-dev and libyaml-dev packages be made available on the ruby3.3 images? - https://phabricator.wikimedia.org/T411226#11414916 (10Ijon) So, yeah, [[ https://wikitech.wikimedia.org/wiki/Help:Toolforge/My_first_Ruby_on_Rails_tool | this page has all the clues ]]. I... [09:15:29] 10Cloud-VPS (Project-requests): Request creation of Gutendex VPS project - https://phabricator.wikimedia.org/T411158#11415038 (10JJMC89) 05Resolved→03Declined a:05Ijon→03None [11:20:06] 06cloud-services-team, 10Cloud-VPS: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248 (10fgiunchedi) 03NEW [11:25:38] 06cloud-services-team, 10Cloud-VPS, 10Data-Services: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#11415400 (10taavi) [11:32:51] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 10Toolforge: If the inactive clouddumps host goes down, it causes a ripple effect on Cloud VPS and Toolforge - https://phabricator.wikimedia.org/T391369#11415427 (10fgiunchedi) I briefly looked the clouddumps1002 downtime from yesterday, and of course the... [13:46:53] 10VPS-project-Codesearch, 06collaboration-services: Stop pulling netbox-exported-dns repo from Phabricator Diffusion (which mirrors netbox-exports.wikimedia.org) - https://phabricator.wikimedia.org/T410624#11415931 (10Ladsgroup) I'm forcing a restart of the services, let's see if that makes an impact. [13:52:52] 06cloud-services-team, 10Cloud-VPS, 10Data-Services: Plan to make clouddumps more resilient and easier to operate - https://phabricator.wikimedia.org/T411248#11415949 (10fnegri) [13:52:53] 06cloud-services-team (FY2025/26-Q1-Q2), 10Cloud-VPS, 10Toolforge: If the inactive clouddumps host goes down, it causes a ripple effect on Cloud VPS and Toolforge - https://phabricator.wikimedia.org/T391369#11415950 (10fnegri) [14:25:55] 10VPS-project-Codesearch, 06collaboration-services: Stop pulling netbox-exported-dns repo from Phabricator Diffusion (which mirrors netbox-exports.wikimedia.org) - https://phabricator.wikimedia.org/T410624#11416056 (10Ladsgroup) Write config is broken: ` Nov 28 14:24:49 codesearch9 write_config.py[705103]: Ski... [14:30:03] 06cloud-services-team (FY2025/26-Q1-Q2), 10Toolforge, 10Wiki-Loves-Monuments-Database, 07Sustainability (Incident Followup): [toolsdb] ibdata1 growing on primary - https://phabricator.wikimedia.org/T409716#11416072 (10fnegri) > Just to be sure, would it be helpful if I stop my operations for a few days?... [14:39:40] 10VPS-project-Codesearch, 06collaboration-services: Stop pulling netbox-exported-dns repo from Phabricator Diffusion (which mirrors netbox-exports.wikimedia.org) - https://phabricator.wikimedia.org/T410624#11416114 (10Ladsgroup) Don't ask how I got this: ` Nov 28 14:38:09 codesearch9 write_config.py[705745]: <... [14:41:25] (03CR) 10Ladsgroup: "This broke the codesearch update since the merge: T410624#11416114 You shouldn't add .update on repos when it's one repo only." [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1165049 (owner: 10David Caro) [14:42:28] 06cloud-services-team (FY2025/26-Q1-Q2), 10Toolforge, 10Wiki-Loves-Monuments-Database, 07Sustainability (Incident Followup): [toolsdb] ibdata1 growing on primary - https://phabricator.wikimedia.org/T409716#11416129 (10fnegri) Something happened last night that caused the history length to grow again, but i... [14:44:58] (03PS1) 10Ladsgroup: Fix PAWS [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1212595 (https://phabricator.wikimedia.org/T410624) [14:46:06] (03CR) 10Ladsgroup: [C:03+2] Fix PAWS [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1212595 (https://phabricator.wikimedia.org/T410624) (owner: 10Ladsgroup) [14:46:59] (03Merged) 10jenkins-bot: Fix PAWS [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1212595 (https://phabricator.wikimedia.org/T410624) (owner: 10Ladsgroup) [14:58:31] 06cloud-services-team: Audit and standardize on UTC timezone for grafana.wmcloud.org dashboards - https://phabricator.wikimedia.org/T411274 (10fgiunchedi) 03NEW [14:59:34] 06cloud-services-team: Audit and standardize on UTC timezone for grafana.wmcloud.org dashboards - https://phabricator.wikimedia.org/T411274#11416233 (10fgiunchedi) As to effectively do the audit, we can adapt search-grafana-dashboards.js from https://wikitech.wikimedia.org/wiki/Grafana#Search/audit_metrics_usage... [16:01:16] 10VPS-project-Codesearch, 06collaboration-services: Stop pulling netbox-exported-dns repo from Phabricator Diffusion (which mirrors netbox-exports.wikimedia.org) - https://phabricator.wikimedia.org/T410624#11416311 (10Ladsgroup) 05Open→03Resolved It has finally stopped [16:18:24] (03open) 10fnegri: images: resolve the image every time (without cache) [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/255 [16:24:03] (03close) 10fnegri: images: resolve the image every time (without cache) [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/255 [16:24:21] 10Cloud-VPS (Project-requests): Request creation of Gutendex VPS project - https://phabricator.wikimedia.org/T411158#11416322 (10bd808) >>! In T411158#11414818, @Ijon wrote: > Okay, confirmed that it runs well on MariaDB as well, so no need for a VPS. Is there anything in particular I need to do to be able... [16:31:38] 10VPS-project-Codesearch, 06collaboration-services: Stop pulling netbox-exported-dns repo from Phabricator Diffusion (which mirrors netbox-exports.wikimedia.org) - https://phabricator.wikimedia.org/T410624#11416343 (10Aklapper) Thank you! <3 [17:30:33] (03Abandoned) 10Wandji collins: Remove app pycache files from git [labs/tools/wdaudiolex-be] - 10https://gerrit.wikimedia.org/r/1133587 (https://phabricator.wikimedia.org/T386329) (owner: 10Bovimacoco) [18:17:40] (03PS1) 10Wandji collins: Adding gerrit review to the project [labs/tools/wdaudiolex-fe] - 10https://gerrit.wikimedia.org/r/1212620 (https://phabricator.wikimedia.org/T391419) [18:26:52] (03PS2) 10Wandji collins: Adding gerrit review to the project [labs/tools/wdaudiolex-fe] - 10https://gerrit.wikimedia.org/r/1212620 (https://phabricator.wikimedia.org/T391419) [19:29:34] 06cloud-services-team, 10Toolforge: Can the ruby-dev and libyaml-dev packages be made available on the ruby3.3 images? - https://phabricator.wikimedia.org/T411226#11416613 (10Ijon) Okay, I'm all set. For the benefit of anyone stumbling across this ticket: switching to envvars was simple enough, and the page I... [19:29:47] 06cloud-services-team, 10Toolforge: Can the ruby-dev and libyaml-dev packages be made available on the ruby3.3 images? - https://phabricator.wikimedia.org/T411226#11416614 (10Ijon) 05Open→03Resolved a:03Ijon