[00:11:28] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [00:16:08] (03CR) 10Krinkle: [C:03+2] "Before:" [labs/tools/coverme] - 10https://gerrit.wikimedia.org/r/940489 (owner: 10Krinkle) [00:16:28] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:16:28] (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [00:20:56] (CloudVPSDesignateLeaks) firing: (5) Detected 12 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:21:28] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:30:03] (03PS1) 10Krinkle: Frontend: change default from `all` to `index` [labs/tools/coverme] - 10https://gerrit.wikimedia.org/r/1014140 [00:30:23] (03CR) 10Krinkle: [C:03+2] Frontend: change default from `all` to `index` [labs/tools/coverme] - 10https://gerrit.wikimedia.org/r/1014140 (owner: 10Krinkle) [00:31:00] (03Merged) 10jenkins-bot: Frontend: change default from `all` to `index` [labs/tools/coverme] - 10https://gerrit.wikimedia.org/r/1014140 (owner: 10Krinkle) [00:36:40] (03PS1) 10Krinkle: Revert "Frontend: change default from `all` to `index`" [labs/tools/coverme] - 10https://gerrit.wikimedia.org/r/1014076 [00:36:43] (03CR) 10Krinkle: [C:03+2] Revert "Frontend: change default from `all` to `index`" [labs/tools/coverme] - 10https://gerrit.wikimedia.org/r/1014076 (owner: 10Krinkle) [00:37:20] (03Merged) 10jenkins-bot: Revert "Frontend: change default from `all` to `index`" [labs/tools/coverme] - 10https://gerrit.wikimedia.org/r/1014076 (owner: 10Krinkle) [02:01:19] 10Quarry, 10Toolforge, 10ChangeProp, 06collaboration-services, and 9 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9660226 (10tstarling) PhpRedis is getting behind KeyDB with [[https://github.com/phpredis/phpredis/issues/2466|#246... [02:33:58] 10Tools: 'hoiscript' tool uses an unreasonable amount of disk space - https://phabricator.wikimedia.org/T349913#9660243 (10Hoi) @bd808 I find that there are some repeated files in the folder. That was my attempt to deduplicate them in order to make the upload process more efficient. I will just let the upload sc... [03:04:26] 10Wikibugs, 07Software-Licensing: Relicense Wikibugs from MIT to GPL-3.0-or-later after approval by all substantive contributors - https://phabricator.wikimedia.org/T360718#9660283 (10yuvipanda) I approve! [03:34:33] 10Wikibugs, 07Software-Licensing: Relicense Wikibugs from MIT to GPL-3.0-or-later after approval by all substantive contributors - https://phabricator.wikimedia.org/T360718#9660312 (10bd808) 05Open→03In progress a:03bd808 [04:20:56] (CloudVPSDesignateLeaks) firing: (5) Detected 12 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:47:29] 10Wikibugs, 13Patch-For-Review, 07Software-Licensing: Relicense Wikibugs from MIT to GPL-3.0-or-later after approval by all substantive contributors - https://phabricator.wikimedia.org/T360718#9660329 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/22 R... [05:08:48] 10superset.wmcloud.org: Public viewing of superset - https://phabricator.wikimedia.org/T336522#9660336 (10KCVelaga_WMF) This would be great. As the [[ https://www.mediawiki.org/wiki/Wikimedia_Language_engineering | Language ]] team is considering to use Superset, as a successor to Special:ContentTranslationStats... [05:35:05] 10VPS-project-Codesearch, 06MediaWiki-Platform-Team, 13Patch-For-Review, 07patch-welcome: 14Allow browsing of full list of indexed repositories - 14https://phabricator.wikimedia.org/T346074#9660364 (10Krinkle) [05:37:03] 10VPS-project-Codesearch, 06MediaWiki-Platform-Team, 13Patch-For-Review, 07patch-welcome: 14Allow browsing of full list of indexed repositories - 14https://phabricator.wikimedia.org/T346074#9660362 (10Krinkle) 05Open→03Resolved a:03Krinkle [08:20:56] (CloudVPSDesignateLeaks) firing: (5) Detected 12 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:21:19] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [09:36:18] 10cloud-services-team (FY2023/2024-Q3-Q4), 06Infrastructure-Foundations, 10Spicerack, 10SRE-tools, 13Patch-For-Review: spicerack: tox fails to install PyYAML using python 3.11 on bookworm - https://phabricator.wikimedia.org/T345337#9660652 (10fnegri) [10:03:14] (03CR) 10Majavah: [C:03+2] vps: create_instance: Do not assume k8s-specific security group [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1013013 (owner: 10Majavah) [10:03:34] (03CR) 10Majavah: [C:03+2] vps: create_instance: Add flag to sign Puppet certs [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1013080 (owner: 10Majavah) [10:05:07] (03CR) 10Majavah: [C:03+2] wmcs_libs: openstack: Improve Neutron port handling [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1013081 (owner: 10Majavah) [10:06:51] (03Merged) 10jenkins-bot: vps: create_instance: Do not assume k8s-specific security group [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1013013 (owner: 10Majavah) [10:07:28] (03Merged) 10jenkins-bot: vps: create_instance: Add flag to sign Puppet certs [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1013080 (owner: 10Majavah) [10:08:20] (03Merged) 10jenkins-bot: wmcs_libs: openstack: Improve Neutron port handling [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1013081 (owner: 10Majavah) [10:15:04] (03CR) 10Majavah: [C:03+2] toolforge: Add cookbook to add new K8s HAProxy node [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1013082 (https://phabricator.wikimedia.org/T349206) (owner: 10Majavah) [10:15:08] (03CR) 10Majavah: [C:03+2] toolforge: Add cookbook to remove a K8s HAProxy node [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1013302 (https://phabricator.wikimedia.org/T349206) (owner: 10Majavah) [10:18:42] (03Merged) 10jenkins-bot: toolforge: Add cookbook to add new K8s HAProxy node [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1013082 (https://phabricator.wikimedia.org/T349206) (owner: 10Majavah) [10:18:46] (03Merged) 10jenkins-bot: toolforge: Add cookbook to remove a K8s HAProxy node [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1013302 (https://phabricator.wikimedia.org/T349206) (owner: 10Majavah) [10:21:59] (03PS1) 10Majavah: wmcs_libs: Generalize the batch runner pattern [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014470 [10:22:00] (03PS1) 10Majavah: openstack: cloudcontrol: Convert reboot cookbook to batch base [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014471 [10:25:26] (03CR) 10CI reject: [V:04-1] openstack: cloudcontrol: Convert reboot cookbook to batch base [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014471 (owner: 10Majavah) [10:25:34] 06cloud-services-team, 10Toolforge (Toolforge iteration 07), 07Kubernetes, 13Patch-For-Review: 14[infra] Upgrade Toolforge K8s haproxies to Bookworm - 14https://phabricator.wikimedia.org/T349206#9660751 (10taavi) 05Open→03Resolved [10:25:38] (03CR) 10CI reject: [V:04-1] wmcs_libs: Generalize the batch runner pattern [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014470 (owner: 10Majavah) [10:32:13] (03PS2) 10Majavah: wmcs_libs: Generalize the batch runner pattern [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014470 [10:32:13] (03PS2) 10Majavah: openstack: cloudcontrol: Convert reboot cookbook to batch base [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014471 [10:35:24] (03CR) 10CI reject: [V:04-1] openstack: cloudcontrol: Convert reboot cookbook to batch base [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014471 (owner: 10Majavah) [10:36:19] (03CR) 10CI reject: [V:04-1] wmcs_libs: Generalize the batch runner pattern [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014470 (owner: 10Majavah) [10:38:28] (03PS3) 10Majavah: wmcs_libs: Generalize the batch runner pattern [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014470 [10:38:28] (03PS3) 10Majavah: openstack: cloudcontrol: Convert reboot cookbook to batch base [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014471 [10:38:28] (03PS1) 10Majavah: tox: Add Python 3.12 support [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014479 [10:38:54] (03CR) 10CI reject: [V:04-1] wmcs_libs: Generalize the batch runner pattern [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014470 (owner: 10Majavah) [10:38:57] (03CR) 10CI reject: [V:04-1] tox: Add Python 3.12 support [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014479 (owner: 10Majavah) [10:38:57] (03CR) 10CI reject: [V:04-1] openstack: cloudcontrol: Convert reboot cookbook to batch base [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014471 (owner: 10Majavah) [10:40:15] (03PS2) 10Majavah: tox: Add Python 3.12 support [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014479 [10:40:15] (03PS4) 10Majavah: wmcs_libs: Generalize the batch runner pattern [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014470 [10:40:15] (03PS4) 10Majavah: openstack: cloudcontrol: Convert reboot cookbook to batch base [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014471 [11:13:54] 10Quarry, 10Toolforge, 10ChangeProp, 06collaboration-services, and 9 others: Figure out a plan to move forward with regarding Redis License changes - https://phabricator.wikimedia.org/T360596#9660965 (10larissagaulia) [12:20:57] (CloudVPSDesignateLeaks) firing: (5) Detected 12 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:20:59] 10VPS-project-Codesearch: Include dependencies of WMF deployed extensions in the WMF-deployed index - https://phabricator.wikimedia.org/T361002 (10taavi) 03NEW [12:21:52] (03CR) 10FNegri: [C:03+1] tox: Add Python 3.12 support [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014479 (owner: 10Majavah) [12:22:10] (03CR) 10Majavah: [C:03+2] tox: Add Python 3.12 support [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014479 (owner: 10Majavah) [12:25:28] (03Merged) 10jenkins-bot: tox: Add Python 3.12 support [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014479 (owner: 10Majavah) [12:43:53] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.remove_instance for instance tools-sgebastion-11 [12:44:45] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance tools-sgebastion-11 [12:45:42] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-bastion' [12:50:25] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-bastion' [12:51:28] (InstanceDown) firing: Project tools instance tools-sgebastion-11 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [12:54:19] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-bastion-13.tools.eqiad1.wikimedia.cloud [12:55:26] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-bastion-13.tools.eqiad1.wikimedia.cloud [12:56:28] (InstanceDown) resolved: Project tools instance tools-sgebastion-11 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [13:16:05] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.nfs.add_server [13:20:23] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99) [13:22:12] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.refresh_puppet_certs on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud [13:23:34] 10Toolforge, 07Software-Licensing: [builds-api] builds-api is missing a software license - https://phabricator.wikimedia.org/T361007 (10taavi) 03NEW [13:23:43] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on toolsbeta-nfs-3.toolsbeta.eqiad1.wikimedia.cloud [13:24:49] 10Toolforge, 07Software-Licensing: [builds-api] builds-api is missing a software license - https://phabricator.wikimedia.org/T361007#9661409 (10dcaro) +1 for agpl-3 [13:26:59] (03PS1) 10Majavah: nfs: add_server: Sign Puppet certs for new instance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014506 [13:27:09] 10Toolforge, 07Software-Licensing: [builds-api] builds-api is missing a software license - https://phabricator.wikimedia.org/T361007#9661486 (10aborrero) >>! In T361007#9661409, @dcaro wrote: > +1 for agpl-3 same! [13:31:17] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.nfs.migrate_service [13:31:24] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) [13:33:54] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.nfs.migrate_service [13:34:05] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) [13:45:15] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 13Patch-For-Review: 14[wmcs-backup] exclude_volumes is matching on IDs instead of names - 14https://phabricator.wikimedia.org/T359192#9661540 (10fnegri) 05In progress→03Resolved [13:50:24] (03PS1) 10Majavah: nfs: add_server: Use the ENC wmcs_libs library [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014513 [13:50:56] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3 [13:51:46] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3 [13:52:32] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [jobs-api] Split the API, business, and k8s models - https://phabricator.wikimedia.org/T359808#9661579 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/70 add runtime [13:53:32] (03CR) 10CI reject: [V:04-1] nfs: add_server: Use the ENC wmcs_libs library [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014513 (owner: 10Majavah) [13:54:46] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.nfs.add_server [13:55:11] !log taavi@cloudcumin1001 toolsbeta END (ERROR) - Cookbook wmcs.nfs.add_server (exit_code=97) [13:56:22] (03PS2) 10Majavah: nfs: add_server: Use the ENC wmcs_libs library [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014513 [13:56:35] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.nfs.add_server [13:59:41] (03CR) 10CI reject: [V:04-1] nfs: add_server: Use the ENC wmcs_libs library [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014513 (owner: 10Majavah) [14:00:50] (03PS3) 10Majavah: nfs: add_server: Use the ENC wmcs_libs library [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014513 [14:01:57] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.nfs.add_server (exit_code=99) [14:02:39] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 10Data-Services, 13Patch-For-Review: [cinder] Deleting snapshot does not work - https://phabricator.wikimedia.org/T356904#9661653 (10fnegri) [14:02:39] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 05Goal: [toolsdb] test creating a new replica host - https://phabricator.wikimedia.org/T344717#9661654 (10fnegri) [14:02:40] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 10Data-Services, 13Patch-For-Review: [cinder] Deleting snapshot does not work - https://phabricator.wikimedia.org/T356904#9661649 (10fnegri) 05Stalled→03In progress [14:02:41] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 10Data-Services, 13Patch-For-Review: [cinder] Deleting snapshot does not work - https://phabricator.wikimedia.org/T356904#9661655 (10fnegri) [14:02:49] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.vps.remove_instance for instance toolsbeta-nfs-3 [14:03:11] (03PS4) 10Majavah: nfs: add_server: Use the ENC wmcs_libs library [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014513 [14:03:38] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.vps.remove_instance (exit_code=0) for instance toolsbeta-nfs-3 [14:03:51] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.nfs.add_server [14:06:15] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 10Data-Services, 13Patch-For-Review: [cinder] Deleting snapshot does not work - https://phabricator.wikimedia.org/T356904#9661656 (10fnegri) p:05High→03Medium This is no longer a blocker for T344717, because the patch https://gerrit.wikimedia.org/... [14:09:28] (PuppetSyncFailure) firing: Failed to update Puppet repository /srv/git/operations/puppet on instance metricsinfra-puppetserver-1 in project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:10:13] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.nfs.add_server (exit_code=0) [14:10:44] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 10Data-Services, 13Patch-For-Review: [cinder] Deleting snapshot does not work - https://phabricator.wikimedia.org/T356904#9661665 (10fnegri) [14:10:46] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 05Goal: [toolsdb] test creating a new replica host - https://phabricator.wikimedia.org/T344717#9661666 (10fnegri) [14:10:58] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 10Data-Services, 13Patch-For-Review: [cinder] [toolsdb] Deleting snapshot does not work - https://phabricator.wikimedia.org/T356904#9661667 (10fnegri) [14:11:35] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.nfs.migrate_service [14:11:46] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) [14:14:47] (03PS1) 10Majavah: nfs: migrate_service: Use the ENC wmcs_libs library [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014520 [14:15:46] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 10Data-Services, 13Patch-For-Review: 14[cinder] [toolsdb] Deleting snapshot does not work - 14https://phabricator.wikimedia.org/T356904#9661671 (10fnegri) 05In progress→03Resolved 14Reading again the description, this bug is in effect complet... [14:16:05] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 05Goal: [toolsdb] test creating a new replica host - https://phabricator.wikimedia.org/T344717#9661679 (10fnegri) 05Stalled→03In progress [14:18:01] (03CR) 10CI reject: [V:04-1] nfs: migrate_service: Use the ENC wmcs_libs library [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014520 (owner: 10Majavah) [14:18:35] (03PS1) 10Andrew Bogott: migrate_service: remove overzealous validation of service fqdn [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014521 [14:18:42] (03CR) 10CI reject: [V:04-1] migrate_service: remove overzealous validation of service fqdn [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014521 (owner: 10Andrew Bogott) [14:20:07] (03PS2) 10Majavah: nfs: migrate_service: Use the ENC wmcs_libs library [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014520 [14:20:09] (03PS2) 10Andrew Bogott: migrate_service: remove overzealous validation of service fqdn [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014521 [14:20:45] (03PS3) 10Andrew Bogott: nfs: migrate_service: Use the ENC wmcs_libs library [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014520 (owner: 10Majavah) [14:20:46] (03PS3) 10Andrew Bogott: migrate_service: remove overzealous validation of service fqdn [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014521 [14:21:46] (03PS4) 10Majavah: nfs: migrate_service: remove overzealous validation of service fqdn [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014521 (owner: 10Andrew Bogott) [14:22:17] (03PS5) 10Majavah: nfs: migrate_service: remove overzealous validation of service fqdn [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014521 (owner: 10Andrew Bogott) [14:22:51] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.nfs.migrate_service [14:22:58] !log taavi@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.nfs.migrate_service (exit_code=99) [14:23:53] (03CR) 10CI reject: [V:04-1] nfs: migrate_service: remove overzealous validation of service fqdn [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014521 (owner: 10Andrew Bogott) [14:24:36] (03CR) 10CI reject: [V:04-1] nfs: migrate_service: Use the ENC wmcs_libs library [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014520 (owner: 10Majavah) [14:25:59] (03PS6) 10Majavah: nfs: migrate_service: Remove overzealous validation of service fqdn [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014521 (owner: 10Andrew Bogott) [14:26:37] (03PS4) 10Majavah: nfs: migrate_service: Use the ENC wmcs_libs library [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014520 [14:26:37] (03PS7) 10Majavah: nfs: migrate_service: Remove overzealous validation of service fqdn [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014521 (owner: 10Andrew Bogott) [14:28:20] (03PS5) 10Majavah: nfs: add_server: Use the ENC wmcs_libs library [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014513 [14:28:20] (03PS5) 10Majavah: nfs: migrate_service: Use the ENC wmcs_libs library [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014520 [14:28:20] (03PS8) 10Majavah: nfs: migrate_service: Remove overzealous validation of service fqdn [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014521 (owner: 10Andrew Bogott) [14:28:36] !log taavi@cloudcumin1001 toolsbeta START - Cookbook wmcs.nfs.migrate_service [14:30:13] !log taavi@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.nfs.migrate_service (exit_code=0) [14:32:37] (ProbeDown) firing: (2) Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:53:28] (WidespreadPuppetAgentFailure) firing: Widespread puppet agent failures in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [14:57:51] 06cloud-services-team, 10Observability-Alerting, 10SRE Observability (FY2023/2024-Q4): Karma UI shows duplicate alerts - https://phabricator.wikimedia.org/T353457#9661870 (10fgiunchedi) [14:58:15] 10Cloud-VPS, 10observability, 06SRE, 13Patch-For-Review, 10SRE Observability (FY2023/2024-Q4): ossl rsyslog errors post-migration - https://phabricator.wikimedia.org/T351710#9661872 (10fgiunchedi) [15:17:28] (PuppetAgentFailure) firing: Puppet agent failure detected on instance toolsbeta-test-k8s-worker-nfs-4 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:22:28] (PuppetAgentFailure) firing: (5) Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:27:28] (PuppetAgentFailure) firing: (6) Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:37:28] (PuppetAgentFailure) firing: (8) Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:52:28] (PuppetAgentFailure) firing: (8) Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:52:37] (ProbeDown) resolved: (2) Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [15:58:14] (03PS1) 10Majavah: nfs: add_server: Allow formatting newly created volumes [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014544 [15:58:41] (03CR) 10Andrew Bogott: [C:03+1] nfs: add_server: Allow formatting newly created volumes [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014544 (owner: 10Majavah) [16:02:28] (PuppetAgentFailure) firing: (8) Puppet agent failure detected on instance toolsbeta-bastion-6 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:03:28] (WidespreadPuppetAgentFailure) resolved: Widespread puppet agent failures in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DWidespreadPuppetAgentFailure [16:05:19] 10Toolforge: Upgrade toolsbeta-nfs to Debian Bullseye/Bookworm - https://phabricator.wikimedia.org/T360419#9662137 (10taavi) a:03taavi Done after a "fun" exercise with our volume backups. I will delete the old machine tomorrow. [16:07:28] (PuppetAgentFailure) firing: (7) Puppet agent failure detected on instance toolsbeta-mail-2 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:12:28] (PuppetAgentFailure) firing: (7) Puppet agent failure detected on instance toolsbeta-mail-2 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:14:30] (03CR) 10Andrew Bogott: [C:03+1] nfs: add_server: Sign Puppet certs for new instance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014506 (owner: 10Majavah) [16:16:31] 06cloud-services-team, 10Toolforge: Upgrade Toolforge apt repository (tools-services hosts) to Debian Bullseye or later - https://phabricator.wikimedia.org/T311914#9662195 (10aborrero) +1 to migrate to the main reprepro installation. [16:20:57] (CloudVPSDesignateLeaks) firing: (5) Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:22:28] (PuppetAgentFailure) resolved: (5) Puppet agent failure detected on instance toolsbeta-mail-2 in project toolsbeta - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [16:26:50] (03CR) 10Majavah: [C:03+2] nfs: add_server: Sign Puppet certs for new instance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014506 (owner: 10Majavah) [16:30:47] (03Merged) 10jenkins-bot: nfs: add_server: Sign Puppet certs for new instance [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1014506 (owner: 10Majavah) [16:30:52] 10Toolforge: Upgrade Toolforge Docker registry to bookworm - https://phabricator.wikimedia.org/T361030 (10taavi) 03NEW [16:33:13] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.create_instance_with_prefix with prefix 'tools-docker-registry' [16:35:52] 06cloud-services-team, 13Patch-For-Review: [wmcs][alerting] Allow volunteer admins silencing alerts from cloudvps/toolforge/paws/quarry - https://phabricator.wikimedia.org/T320973#9662341 (10andrea.denisse) Hi @dcaro , I've upgraded karma to 0.119 and it's now live in production. Please let me know if there's... [16:36:58] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.create_instance_with_prefix (exit_code=0) with prefix 'tools-docker-registry' [16:39:44] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud [16:41:24] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud [16:41:28] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tools-docker-registry-7 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [16:47:33] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud [16:50:53] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-docker-registry-7.tools.eqiad1.wikimedia.cloud [16:51:28] (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance tools-docker-registry-7 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [17:46:28] (InstanceDown) firing: Project cloudinfra instance cloudinfra-cloudvps-puppetserver-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:16:28] (InstanceDown) resolved: Project cloudinfra instance cloudinfra-cloudvps-puppetserver-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [18:21:44] 10Toolforge (Toolforge iteration 07): Upgrade Toolforge Docker registry to bookworm - https://phabricator.wikimedia.org/T361030#9662772 (10taavi) [18:22:22] 10Toolforge (Toolforge iteration 07): Upgrade Toolforge Docker registry to bookworm - https://phabricator.wikimedia.org/T361030#9662774 (10taavi) 05Open→03In progress [19:59:03] 06cloud-services-team, 10VPS-Projects, 06collaboration-services, 10Puppet (Puppet 7.0): Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9663285 (10Dzahn) Thank you for looking at this @brennen . It's appreciated. I made a separate ticket for the other buster machines in de... [20:03:38] 10Cloud-VPS (Debian Buster Deprecation), 06collaboration-services: replace buster machines in devtools project - https://phabricator.wikimedia.org/T360964#9663308 (10taavi) [20:03:39] 10Cloud-VPS (Debian Buster Deprecation), 06collaboration-services: replace buster machines in devtools project - https://phabricator.wikimedia.org/T360964#9663310 (10taavi) [20:03:40] 06cloud-services-team, 10VPS-Projects, 06collaboration-services, 10Puppet (Puppet 7.0): Update devtools project puppetmaster - https://phabricator.wikimedia.org/T360470#9663309 (10taavi) [20:20:57] (CloudVPSDesignateLeaks) firing: (5) Detected 12 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:24:48] 14cloud-services-team (Kanban): 14openstack: Failed to create DNS entry for new instance, designate error 'Managed records may not be updated' - 14https://phabricator.wikimedia.org/T280243#9663401 (10colewhite) 14I experienced this issue today. It appears the reverse IP pointer (in-addr.arpa) doesn't get c... [20:33:06] 14cloud-services-team (Kanban): 14openstack: Failed to create DNS entry for new instance, designate error 'Managed records may not be updated' - 14https://phabricator.wikimedia.org/T280243#9663443 (10taavi) 14>>! In T280243#9663401, @colewhite wrote: > I experienced this issue today. It appears the reverse... [21:25:42] (CloudVPSDesignateLeaks) resolved: (5) Detected 12 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:27:54] 10Tool-Global-user-contributions, 06Stewards-and-global-tools, 10Temporary accounts, 10XTools, and 2 others: 14[Design] Prototype and user testing plan - 14https://phabricator.wikimedia.org/T356099#9663978 (10KColeman-WMF) 05Open→03Resolved [22:28:16] 10Tool-Global-user-contributions, 06Stewards-and-global-tools, 10Temporary accounts, 10XTools, and 2 others: 14[Design] UX exploration and wireframes - 14https://phabricator.wikimedia.org/T354531#9663980 (10KColeman-WMF) 05Open→03Resolved [22:29:43] 10Tool-Global-user-contributions, 06Stewards-and-global-tools, 10Temporary accounts, 10XTools, and 2 others: [Design] Synthesise user testing results - https://phabricator.wikimedia.org/T358098#9663988 (10KColeman-WMF) 05Open→03In progress [23:10:04] 06cloud-services-team, 10Cloud-VPS (Quota-requests): Temporarily increase quota for dwl Buster migration - https://phabricator.wikimedia.org/T360788#9664050 (10Giftpflanze) 05Invalid→03Open I have deleted the last Buster instance. [23:41:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:46:41] (CloudVPSDesignateLeaks) firing: (5) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:49:44] 10Wikibugs, 07Software-Licensing: Relicense Wikibugs from MIT to GPL-3.0-or-later after approval by all substantive contributors - https://phabricator.wikimedia.org/T360718#9664095 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/22 Relicense under GPL-3.0... [23:52:47] 10Wikibugs, 07Software-Licensing: 14Relicense Wikibugs from MIT to GPL-3.0-or-later after approval by all substantive contributors - 14https://phabricator.wikimedia.org/T360718#9664104 (10bd808) 05In progress→03Resolved