[00:30:34] 14Grid-Engine-to-K8s-Migration: 14Migrate dibot from Toolforge GridEngine to Toolforge Kubernetes - 14https://phabricator.wikimedia.org/T319676#9633055 (10MBH) 14I will publish them after my working day, but you can edit this php files directly in the Toolforge folder, they aren't binaries like my tools. [01:46:56] (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:48:19] (03PS1) 10Krinkle: frontend: Fix excludeFiles mismatch between form read and URL write [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1011214 (https://phabricator.wikimedia.org/T321347) [03:48:48] (03CR) 10Krinkle: [C:03+2] frontend: Fix excludeFiles mismatch between form read and URL write [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1011214 (https://phabricator.wikimedia.org/T321347) (owner: 10Krinkle) [03:49:55] (03Merged) 10jenkins-bot: frontend: Fix excludeFiles mismatch between form read and URL write [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1011214 (https://phabricator.wikimedia.org/T321347) (owner: 10Krinkle) [03:50:22] 10VPS-project-Codesearch, 13Patch-For-Review, 07patch-welcome: Codesearch: "excludeFilePath" option should be preserved when switching - https://phabricator.wikimedia.org/T321347#9633212 (10Krinkle) p:05Triage→03Medium a:03Krinkle [03:54:14] 10VPS-project-Codesearch, 13Patch-For-Review, 07patch-welcome: 14Codesearch: "excludeFilePath" option should be preserved when switching - 14https://phabricator.wikimedia.org/T321347#9633216 (10Krinkle) 05Open→03Resolved [04:03:04] 10VPS-project-Codesearch: Allow browsing of full list of indexed repositories - https://phabricator.wikimedia.org/T346074#9633222 (10Krinkle) [04:03:35] 10VPS-project-Codesearch: Allow browsing of full list of indexed repositories - https://phabricator.wikimedia.org/T346074#9633223 (10Krinkle) I've renamed this task from an implied solution, to instead an observed use case. [04:03:38] 10VPS-project-Codesearch: Allow browsing of full list of indexed repositories - https://phabricator.wikimedia.org/T346074#9633221 (10Krinkle) This is intentionally not done, as it would make the menu slow to open (e.g. the browser freezes for a full second), and also autocompletion will take a full second for ev... [04:05:10] 10VPS-project-Codesearch, 07patch-welcome: Allow browsing of full list of indexed repositories - https://phabricator.wikimedia.org/T346074#9633224 (10Krinkle) p:05Triage→03Low [04:07:29] 10VPS-project-Codesearch: Codesearch should have a non-test filter - https://phabricator.wikimedia.org/T197802#9633227 (10Krinkle) Note that there's no requirement for the regex to match the full path. `"test"` suffices to exclude all directories and files pertaining to QUnit (tests/qunit/foo.test.js) or PHPUnit... [04:07:53] 10VPS-project-Codesearch: Codesearch should have a non-test filter - https://phabricator.wikimedia.org/T197802#9633229 (10Krinkle) p:05Triage→03Low [04:10:49] (03PS1) 10Krinkle: write_config: search codesearch using codesearch on codesearch [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1011215 [04:11:54] (03PS1) 10Krinkle: write_config: index labs/toollabs repo [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1011216 (https://phabricator.wikimedia.org/T358983) [04:12:29] (03CR) 10Krinkle: [C:03+2] write_config: search codesearch using codesearch on codesearch [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1011215 (owner: 10Krinkle) [04:12:34] (03CR) 10Krinkle: [C:03+2] write_config: index labs/toollabs repo [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1011216 (https://phabricator.wikimedia.org/T358983) (owner: 10Krinkle) [04:13:18] (03Merged) 10jenkins-bot: write_config: search codesearch using codesearch on codesearch [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1011215 (owner: 10Krinkle) [04:13:26] (03Merged) 10jenkins-bot: write_config: index labs/toollabs repo [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1011216 (https://phabricator.wikimedia.org/T358983) (owner: 10Krinkle) [04:14:20] 10VPS-project-Codesearch, 13Patch-For-Review: 14Index labs/toollabs - 14https://phabricator.wikimedia.org/T358983#9633232 (10Krinkle) p:05Triage→03Medium a:03Krinkle [04:15:38] 10VPS-project-Codesearch, 13Patch-For-Review: 14Index labs/toollabs - 14https://phabricator.wikimedia.org/T358983#9633235 (10Krinkle) 05Open→03Resolved 14It will be indexed within 24 hours. [05:46:56] (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:53:35] 10Toolforge: [envvars-api, envvars-cli] Create envvar name error message is not user friendly - https://phabricator.wikimedia.org/T360147#9633292 (10Slst2020) [07:12:32] 10Toolforge: [envvars-api, envvars-cli] Create envvar name error message is not user friendly - https://phabricator.wikimedia.org/T360147#9633308 (10Slst2020) The current error message probably comes from the envvars API that envvars-cli talks to. The repo is here: https://gitlab.wikimedia.org/repos/cloud/toolfo... [09:46:57] (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:49:09] 14Grid-Engine-to-K8s-Migration, 10Tools, 06All-and-every-Wikisource: Migrate phetools from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319965#9633459 (10Tpt) @Soda Amazing! Thank you! A user request from French Wikisource: Would it be possible to add to the website plots... [09:51:58] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Increase Object Storage quota for QRank - https://phabricator.wikimedia.org/T360162#9633479 (10aborrero) 05Open→03In progress p:05Triage→03Medium a:03aborrero [09:53:02] (03PS1) 10Majavah: Use deb.svc.toolforge.org [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1011276 (https://phabricator.wikimedia.org/T306039) [09:55:51] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:00:50] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:03:21] (03PS1) 10Majavah: Reformat with black [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1011277 [10:03:22] (03PS1) 10Majavah: Sort instances in instance selector [openstack/horizon/wmf-proxy-dashboard] - 10https://gerrit.wikimedia.org/r/1011278 [10:38:33] 06cloud-services-team, 10Cloud-VPS (Quota-requests): 14Increase Object Storage quota for QRank - 14https://phabricator.wikimedia.org/T360162#9633571 (10aborrero) 05In progress→03Resolved 14Before: `lang=shell-session aborrero@cloudcontrol1005:~ $ sudo radosgw-admin user info --uid qrank\$qrank {... [10:39:25] (03CR) 10Arturo Borrero Gonzalez: [C:03+1] "LGTM." [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1011276 (https://phabricator.wikimedia.org/T306039) (owner: 10Majavah) [10:42:05] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [jobs-api] introduce OpenAPI to jobs framework - https://phabricator.wikimedia.org/T356523#9633589 (10CodeReviewBot) aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/66 jobs-api: add /openapi.json endpoint [10:43:58] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [jobs-api] introduce OpenAPI to jobs framework - https://phabricator.wikimedia.org/T356523#9633590 (10CodeReviewBot) project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_req... [10:48:06] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [10:48:19] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [10:50:41] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [10:50:51] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [10:54:30] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [jobs-api] introduce OpenAPI to jobs framework - https://phabricator.wikimedia.org/T356523#9633608 (10CodeReviewBot) aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/222 jobs-api: bump to 0.0.269-2024... [10:55:46] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [jobs-api] introduce OpenAPI to jobs framework - https://phabricator.wikimedia.org/T356523#9633622 (10aborrero) The openapi JSON can now be fetched from the API: `lang=shell-session toolsbeta.test@toolsbeta-sgebastion-05:~$ curl --insecure --silent --c... [11:23:49] (PuppetDisabled) firing: Puppet disabled on cloudvirt2001-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [11:23:53] 06cloud-services-team: PuppetDisabled Puppet disabled on cloudvirt2001-dev:9100 - https://phabricator.wikimedia.org/T360183 (10phaultfinder) 03NEW [11:25:54] 06cloud-services-team: 14PuppetDisabled Puppet disabled on cloudvirt2001-dev:9100 - 14https://phabricator.wikimedia.org/T360183#9633659 (10taavi) 05Open→03Resolved a:03taavi [11:34:16] 10Toolforge (Toolforge iteration 07): 14[jobs-api] introduce OpenAPI to jobs framework - 14https://phabricator.wikimedia.org/T356523#9633708 (10aborrero) 05In progress→03Resolved [11:34:22] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [toolforge API] expose all backend APIs OpenAPI specs - https://phabricator.wikimedia.org/T358100#9633709 (10aborrero) [11:34:25] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [jobs-api,buildservice-api,envvars-api] Investigate ways to present our multiple Openapi definitions to a future consolidated CLI client - https://phabricator.wikimedia.org/T354745#9633710 (10aborrero) [11:40:58] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review: Improve cloudgw filter between VM instances and cloud-private - https://phabricator.wikimedia.org/T356986#9633728 (10aborrero) ping @cmooney [11:41:51] 10Toolforge: [jobs-cli] Allow exporting jobs list in YAML format - https://phabricator.wikimedia.org/T320575#9633731 (10aborrero) 05Open→03In progress [11:45:38] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [toolforge API] expose all backend APIs OpenAPI specs - https://phabricator.wikimedia.org/T358100#9633735 (10Slst2020) [11:46:17] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [jobs-api,buildservice-api,envvars-api] Investigate ways to present our multiple Openapi definitions to a future consolidated CLI client - https://phabricator.wikimedia.org/T354745#9633739 (10Slst2020) [11:46:36] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: 14[toolforge API] expose all backend APIs OpenAPI specs - 14https://phabricator.wikimedia.org/T358100#9633738 (10Slst2020) 05Open→03Resolved [11:55:29] 10Toolforge, 13Patch-For-Review: [jobs-cli] Allow exporting jobs list in YAML format - https://phabricator.wikimedia.org/T320575#9633743 (10CodeReviewBot) aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/17 tjf_cli: add option to list all jobs in YAML format [11:56:22] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [jobs-api,buildservice-api,envvars-api] Investigate ways to present our multiple Openapi definitions to a future consolidated CLI client - https://phabricator.wikimedia.org/T354745#9633745 (10dcaro) >>! In T354745#9631425, @Slst2020 wrote: > Do we want... [11:58:48] (PuppetDisabled) resolved: Puppet disabled on cloudvirt2001-dev:9100 - https://wikitech.wikimedia.org/wiki/Puppet/Runbooks#Puppet_Disabled - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet?var-cluster=wmcs&viewPanel=14 - https://alerts.wikimedia.org/?q=alertname%3DPuppetDisabled [12:19:30] 06cloud-services-team, 10Toolforge: [jobs-cli,jobs-api] make API and CLI key/values coherent - https://phabricator.wikimedia.org/T327280#9633778 (10aborrero) [12:29:43] (03CR) 10Majavah: [C:03+2] Use deb.svc.toolforge.org [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1011276 (https://phabricator.wikimedia.org/T306039) (owner: 10Majavah) [12:30:20] (03Merged) 10jenkins-bot: Use deb.svc.toolforge.org [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1011276 (https://phabricator.wikimedia.org/T306039) (owner: 10Majavah) [12:38:44] 10Toolforge, 07Documentation: Move content out of Kubernetes doc and into web and jobs framework docs - https://phabricator.wikimedia.org/T347888#9633808 (10taavi) That page should eventually all be merged to the jobs framework page, I think. [12:39:09] 10Toolforge: Provide metrics about build service and non-NFS adoption - https://phabricator.wikimedia.org/T360190 (10taavi) 03NEW [12:48:42] 14Grid-Engine-to-K8s-Migration: 14Migrate dibot from Toolforge GridEngine to Toolforge Kubernetes - 14https://phabricator.wikimedia.org/T319676#9633827 (10MBH) 14I added all of them to https://github.com/Saisengen/wikibots/tree/main/php-tools. If it's needed to add html/css/js files from Dmitry's folder - s... [13:02:50] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:07:50] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:17:20] 10Quarry: Deploy magnum cluster for quarry - https://phabricator.wikimedia.org/T349032#9633851 (10rook) @SD0001 @Audiodude could yinz take a look at quarry-test.wmcloud.org and see if there are any obvious problems? The data is all a duplicate of the production quarry, though now everything is running in k8s (ex... [13:34:05] 14Grid-Engine-to-K8s-Migration: 14Migrate dibot from Toolforge GridEngine to Toolforge Kubernetes - 14https://phabricator.wikimedia.org/T319676#9633879 (10dcaro) 14>>! In T319676#9633827, @MBH wrote: > I added all of them to https://github.com/Saisengen/wikibots/tree/main/php-tools. If it's needed to add ht... [13:37:00] 14Grid-Engine-to-K8s-Migration: 14Migrate dibot from Toolforge GridEngine to Toolforge Kubernetes - 14https://phabricator.wikimedia.org/T319676#9633903 (10MBH) 14Is there a way to move these files to the new repo, or I should re-create all of these files manually again? [13:38:57] 14Grid-Engine-to-K8s-Migration: 14Migrate dibot from Toolforge GridEngine to Toolforge Kubernetes - 14https://phabricator.wikimedia.org/T319676#9633907 (10dcaro) 14>>! In T319676#9633903, @MBH wrote: > Is there a way to move these files to the new repo, or I should re-create all of these files manually agai... [13:46:57] (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:23:38] 10PAWS: jupyterlab to 4.1.5 - https://phabricator.wikimedia.org/T360193 (10rook) 03NEW [14:24:44] 10PAWS: jupyterlab to 4.1.5 - https://phabricator.wikimedia.org/T360193#9633986 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/390 [14:24:56] vivian-rook opened https://github.com/toolforge/paws/pull/390 [16:15:06] vivian-rook closed https://github.com/toolforge/paws/pull/390 [16:15:45] 10PAWS: jupyterlab to 4.1.5 - https://phabricator.wikimedia.org/T360193#9634323 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/390 [16:16:34] 10PAWS: 14jupyterlab to 4.1.5 - 14https://phabricator.wikimedia.org/T360193#9634326 (10rook) 05Open→03Resolved [16:32:03] 14Grid-Engine-to-K8s-Migration: 14Migrate hunsbot from Toolforge GridEngine to Toolforge Kubernetes - 14https://phabricator.wikimedia.org/T319801#9634369 (10Hunsvotti) 05Declined→03Resolved [16:48:28] (PuppetAgentStaleLastRun) firing: (2) Last Puppet run was over 24 hours ago on instance toolsbeta-test-k8s-control-7 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [16:59:07] 14Grid-Engine-to-K8s-Migration: 14Migrate piglobot from Toolforge GridEngine to Toolforge Kubernetes - 14https://phabricator.wikimedia.org/T319968#9634473 (10Piglop) 05Declined→03Resolved 14Hello, Sorry, I didn't take care of this issue earlier. The tool was actually migrated automatically (it's only a... [17:26:50] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:31:50] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [17:46:57] (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:55:22] 10Quarry: Scrape prometheus metrics from Quarry - https://phabricator.wikimedia.org/T360220#9634600 (10taavi) They are supposed to be available on https://prometheus.wmcloud.org ([[ https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Monitoring | docs ]]) but seems like my code to merge the results of the... [17:59:23] 06cloud-services-team, 06Data-Platform-SRE: Review and fix any bugs found in the automated bootstrap process for a ceph mon/mgr server - https://phabricator.wikimedia.org/T332987#9634624 (10BTullis) a:05BTullis→03None [18:12:34] (DiskSpace) firing: Disk space cloudbackup1004:9100:/ 5.742% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [19:25:39] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 05Goal, 13Patch-For-Review, 10Puppet (Puppet 7.0): Migrate Cloud VPS puppet infrastructure to Puppet 7 - https://phabricator.wikimedia.org/T351450#9634826 (10Andrew) I have built a very large server (cloudinfra-cloudvps-puppetserver-1.cloudinfra.eqi... [19:48:28] (PuppetAgentStaleLastRun) firing: (2) Last Puppet run was over 24 hours ago on instance toolsbeta-test-k8s-control-7 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [20:44:54] 10Quarry: Deploy magnum cluster for quarry - https://phabricator.wikimedia.org/T349032#9634942 (10SD0001) Had a brief look and it looks good to me. Thanks! I'm no longer able to invoke `kubectl` from quarry-bastion, though. It says `Unable to connect to the server: dial tcp 172.16.4.237:6443: connect: no route... [20:55:44] 10Quarry: Deploy magnum cluster for quarry - https://phabricator.wikimedia.org/T349032#9634948 (10rook) Oh there's a new k8s cluster for it. I've put the updated config in /opt/quarry-123-2.config [20:57:46] 10Quarry: store quarry state in object storage - https://phabricator.wikimedia.org/T360233 (10rook) 03NEW [20:57:59] 10Quarry: store quarry state in object storage - https://phabricator.wikimedia.org/T360233#9634995 (10rook) [20:58:02] 10Quarry: Deploy magnum cluster for quarry - https://phabricator.wikimedia.org/T349032#9634994 (10rook) [20:59:18] 10Quarry: Deploy magnum cluster for quarry - https://phabricator.wikimedia.org/T349032#9634997 (10rook) I've opened T360233 to manage the tofu state in an object store. That way things shouldn't be stored locally, and tofu/deploy.sh can be run from wherever in the bastion. [21:42:34] (DiskSpace) resolved: Disk space cloudbackup1004:9100:/ 5.963% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace [21:46:57] (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:02:20] 10VPS-project-Codesearch: Can't search for multi-line regex any more - https://phabricator.wikimedia.org/T358786#9635106 (10Krinkle) There have been no logic changes in recent months to my knowledge, and upstream Hound has been in maintenance-mode for a few years now. You can try out the original Hound frontend... [22:48:28] (PuppetAgentStaleLastRun) firing: (2) Last Puppet run was over 24 hours ago on instance toolsbeta-test-k8s-control-7 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [23:31:34] (DiskSpace) firing: Disk space cloudbackup1004:9100:/ 5.968% free - https://wikitech.wikimedia.org/wiki/Monitoring/Disk_space - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&viewPanel=12&var-server=cloudbackup1004 - https://alerts.wikimedia.org/?q=alertname%3DDiskSpace