[00:23:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance ntp-04 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [00:56:24] 10PAWS: Cleanup miners - https://phabricator.wikimedia.org/T379746#10320818 (10JJMC89) >>! In T379746#10319945, @rook wrote: > Someone's been busy. A few more running miners and DOS things: locked > And many more with such software in their home directories: locked all except Skywiks > Is there any connectio... [01:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:35:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:20:12] 10Tool-techcontribs: techcontribs: Add ways to show Phabricator contributions for non-creation/closure work - https://phabricator.wikimedia.org/T379882 (10Chlod) 03NEW [07:20:41] 10Tool-techcontribs: techcontribs: Add ways to show Phabricator contributions for non-creation/closure work - https://phabricator.wikimedia.org/T379882#10321281 (10Chlod) p:05Triage→03Medium [07:39:17] 10Tool-techcontribs: techcontribs: Year in Review - https://phabricator.wikimedia.org/T379884 (10Chlod) 03NEW [07:39:43] 10Tool-techcontribs: techcontribs: Year in Review - https://phabricator.wikimedia.org/T379884#10321345 (10Chlod) p:05Triage→03High [08:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:03:43] (03CR) 10David Caro: [C:03+2] create_project: add a bit nicer example [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1090889 (owner: 10David Caro) [09:03:50] (03CR) 10CI reject: [V:04-1] create_project: add a bit nicer example [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1090889 (owner: 10David Caro) [09:08:39] (03close) 10dcaro: toolforge-config: use external ip [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/212 [09:10:19] 10Toolforge (Toolforge iteration 16): [toolforge-weld] read setting from envvars too - https://phabricator.wikimedia.org/T379893 (10dcaro) 03NEW [09:10:22] 10Toolforge (Toolforge iteration 16): [toolforge-weld] read setting from envvars too - https://phabricator.wikimedia.org/T379893#10321614 (10dcaro) p:05Triage→03Medium [09:11:27] 10Toolforge (Toolforge iteration 16): [toolforge-weld] read setting from envvars too - https://phabricator.wikimedia.org/T379893#10321617 (10dcaro) We can start only with the toolforge api url, so we unblock the move out of the old bastion, and then move the whole config to use pydantic-settings [09:20:50] !log dcaro@urcuchillay donotmergeme START - Cookbook wmcs.vps.create_project for project donotmergeme in codfw1dev [09:20:52] wmbot~dcaro@urcuchillay: Unknown project "donotmergeme" [09:20:59] !log dcaro@urcuchillay donotmergeme END (FAIL) - Cookbook wmcs.vps.create_project (exit_code=99) for project donotmergeme in codfw1dev [09:21:00] wmbot~dcaro@urcuchillay: Unknown project "donotmergeme" [09:22:30] (03close) 10dcaro: projects: added project donotmergeme [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/126 (owner: 10group_199_bot_333a6c67971a471aeb1cf0b14ccf9f49) [09:22:56] (03close) 10dcaro: DONOTMERGE: silly test [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/29 [09:23:38] 06cloud-services-team, 10Cloud-VPS, 07IPv6: IPv6 support in cloud-private - https://phabricator.wikimedia.org/T379283#10321652 (10aborrero) Thanks for working on this @cmooney. I agree with the allocations. I would appreciate if you can add the netbox objects yourself. Otherwise I will try to do it myself n... [09:24:36] (03open) 10dcaro: config: load api url from env too [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/66 (https://phabricator.wikimedia.org/T379893) [09:47:06] (03PS1) 10Stevemunene: Add airflow oidc clients for pcc [labs/private] - 10https://gerrit.wikimedia.org/r/1091193 (https://phabricator.wikimedia.org/T378440) [09:48:19] !log aborrero@cloudcumin2001 admin START - Cookbook wmcs.openstack.restart_openstack [09:50:38] (03CR) 10Brouberol: [C:03+1] Add airflow oidc clients for pcc [labs/private] - 10https://gerrit.wikimedia.org/r/1091193 (https://phabricator.wikimedia.org/T378440) (owner: 10Stevemunene) [09:52:17] (03CR) 10Stevemunene: [V:03+2 C:03+2] Add airflow oidc clients for pcc [labs/private] - 10https://gerrit.wikimedia.org/r/1091193 (https://phabricator.wikimedia.org/T378440) (owner: 10Stevemunene) [09:52:58] !log aborrero@cloudcumin2001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [09:53:28] (03update) 10hartman: Fix error in [A-z] range [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/115 [09:57:32] 06cloud-services-team, 10Cloud-VPS, 10Striker, 13Patch-For-Review: openstack: wmfkeystonehooks: project ids rather than names are being used in LDAP group creation - https://phabricator.wikimedia.org/T379030#10321796 (10Physikerwelt) From an non-wmf perspective, the most important aspect is documentation.... [10:51:17] (03update) 10aborrero: verify_ip_address: refresh function [repos/cloud/cloud-vps/nova_fullstack_test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test/-/merge_requests/3 (https://phabricator.wikimedia.org/T379356) [10:55:46] (03merge) 10aborrero: verify_ip_address: refresh function [repos/cloud/cloud-vps/nova_fullstack_test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test/-/merge_requests/3 (https://phabricator.wikimedia.org/T379356) [10:56:40] (03update) 10dcaro: config: load api url from env too [repos/cloud/toolforge/toolforge-weld] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-weld/-/merge_requests/66 (https://phabricator.wikimedia.org/T379893) [10:57:51] (03update) 10aborrero: nova_fullstack_test: factorize dig code [repos/cloud/cloud-vps/nova_fullstack_test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test/-/merge_requests/2 (https://phabricator.wikimedia.org/T379356) [10:57:55] (03approved) 10dcaro: Fix error in [A-z] range [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/115 (owner: 10hartman) [11:05:16] (03open) 10dcaro: toolforge_get_versions: avoid failing when package is not installed [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/595 [11:08:41] 06cloud-services-team, 10Cloud-VPS, 10Striker, 13Patch-For-Review: openstack: wmfkeystonehooks: project ids rather than names are being used in LDAP group creation - https://phabricator.wikimedia.org/T379030#10321922 (10aborrero) I think my opinion is that I'm happy if openstack internally has uuids for pr... [11:09:00] (03update) 10dcaro: toolforge_get_versions: avoid failing when package is not installed [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/595 [11:09:05] (03update) 10dcaro: toolforge_get_versions: avoid failing when package is not installed [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/595 [11:09:56] (03approved) 10fnegri: toolforge_get_versions: avoid failing when package is not installed [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/595 (owner: 10dcaro) [11:10:14] (03approved) 10dcaro: toolforge_get_versions: avoid failing when package is not installed [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/595 [11:10:22] (03update) 10dcaro: toolforge_get_versions: avoid failing when package is not installed [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/595 [11:10:22] (03merge) 10dcaro: toolforge_get_versions: avoid failing when package is not installed [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/595 [11:14:19] 06cloud-services-team, 10Toolforge: webservice restart suddenly stopped working - https://phabricator.wikimedia.org/T379903 (10Urbanecm) 03NEW [11:18:25] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): webservice restart suddenly stopped working - https://phabricator.wikimedia.org/T379903#10321961 (10dcaro) p:05Triage→03High a:03dcaro [11:18:28] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): webservice restart suddenly stopped working - https://phabricator.wikimedia.org/T379903#10321964 (10Urbanecm) p:05High→03Triage a:05dcaro→03None [11:20:41] FIRING: CloudVPSDesignateLeaks: Detected 3 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:20:53] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): webservice restart suddenly stopped working - https://phabricator.wikimedia.org/T379903#10321970 (10Urbanecm) p:05Triage→03High a:03dcaro ...edit conflicts, sorry about that. [11:32:29] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): webservice restart suddenly stopped working - https://phabricator.wikimedia.org/T379903#10322009 (10dcaro) I can reproduce locally, looking [11:37:03] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): webservice restart suddenly stopped working - https://phabricator.wikimedia.org/T379903#10322045 (10dcaro) I think I found the issue, the 'type' must be the last thing to append to the args: https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-w... [11:37:32] 06cloud-services-team, 10Cloud-VPS, 07Epic: Enable self-service Prometheus configuration management for project administrators - https://phabricator.wikimedia.org/T284993#10322040 (10fnegri) [11:45:18] (03open) 10dcaro: service.template: parse type always the last [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/62 (https://phabricator.wikimedia.org/T379903) [11:47:56] (03open) 10dcaro: functional: test a service.template with the type not the last [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/596 (https://phabricator.wikimedia.org/T379903) [11:48:06] (03update) 10aborrero: nova_fullstack_test: factorize dig code [repos/cloud/cloud-vps/nova_fullstack_test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test/-/merge_requests/2 (https://phabricator.wikimedia.org/T379356) [11:50:51] (03approved) 10fnegri: service.template: parse type always the last [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/62 (https://phabricator.wikimedia.org/T379903) (owner: 10dcaro) [11:51:23] (03update) 10aborrero: nova_fullstack_test: factorize dig code [repos/cloud/cloud-vps/nova_fullstack_test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test/-/merge_requests/2 (https://phabricator.wikimedia.org/T379356) [11:53:07] (03merge) 10dcaro: service.template: parse type always the last [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/62 (https://phabricator.wikimedia.org/T379903) [11:54:02] (03open) 10dcaro: d/changelog: bump to 0.103.14 [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/63 (https://phabricator.wikimedia.org/T379903) [11:58:38] (03merge) 10aborrero: nova_fullstack_test: factorize dig code [repos/cloud/cloud-vps/nova_fullstack_test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test/-/merge_requests/2 (https://phabricator.wikimedia.org/T379356) [12:01:09] (03approved) 10fnegri: functional: test a service.template with the type not the last [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/596 (https://phabricator.wikimedia.org/T379903) (owner: 10dcaro) [12:01:28] (03approved) 10fnegri: d/changelog: bump to 0.103.14 [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/63 (https://phabricator.wikimedia.org/T379903) (owner: 10dcaro) [12:07:25] (03open) 10aborrero: verify_ssh: fix function call to admit argparse timeout [repos/cloud/cloud-vps/nova_fullstack_test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test/-/merge_requests/4 [12:09:41] (03merge) 10aborrero: verify_ssh: fix function call to admit argparse timeout [repos/cloud/cloud-vps/nova_fullstack_test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test/-/merge_requests/4 [12:50:00] (03merge) 10dcaro: functional: test a service.template with the type not the last [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/596 (https://phabricator.wikimedia.org/T379903) [12:50:44] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [12:56:44] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice [12:59:55] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [13:02:10] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice [13:02:50] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [13:04:46] !log dcaro@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice [13:07:17] (03open) 10aborrero: nova_fullstack_tests: add support to verify IPv6 DNS entries [repos/cloud/cloud-vps/nova_fullstack_test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test/-/merge_requests/5 (https://phabricator.wikimedia.org/T379356) [13:10:29] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [13:10:34] (03update) 10aborrero: nova_fullstack_tests: add support to verify IPv6 DNS entries [repos/cloud/cloud-vps/nova_fullstack_test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test/-/merge_requests/5 (https://phabricator.wikimedia.org/T379356) [13:16:35] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component tools-webservice [13:16:52] (03approved) 10dcaro: d/changelog: bump to 0.103.14 [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/63 (https://phabricator.wikimedia.org/T379903) [13:16:56] (03update) 10dcaro: d/changelog: bump to 0.103.14 [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/63 (https://phabricator.wikimedia.org/T379903) [13:16:57] (03merge) 10dcaro: d/changelog: bump to 0.103.14 [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/63 (https://phabricator.wikimedia.org/T379903) [13:19:18] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: webservice restart suddenly stopped working - https://phabricator.wikimedia.org/T379903#10322355 (10dcaro) The fix has been released :) I'll close this, you can move back the service.template so you don't lose the options t... [13:19:42] (03update) 10aborrero: nova_fullstack_tests: add support to verify IPv6 DNS entries [repos/cloud/cloud-vps/nova_fullstack_test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test/-/merge_requests/5 (https://phabricator.wikimedia.org/T379356) [13:21:05] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [toolforge-weld] read setting from envvars too - https://phabricator.wikimedia.org/T379893#10322360 (10dcaro) 05Open→03In progress [13:23:03] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: webservice restart suddenly stopped working - https://phabricator.wikimedia.org/T379903#10322357 (10dcaro) 05Open→03Resolved [13:23:17] (03update) 10aborrero: nova_fullstack_tests: add support to verify IPv6 DNS entries [repos/cloud/cloud-vps/nova_fullstack_test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test/-/merge_requests/5 (https://phabricator.wikimedia.org/T379356) [13:28:05] (03update) 10aborrero: nova_fullstack_tests: add support to verify IPv6 DNS entries [repos/cloud/cloud-vps/nova_fullstack_test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test/-/merge_requests/5 (https://phabricator.wikimedia.org/T379356) [13:35:31] (03merge) 10aborrero: nova_fullstack_tests: add support to verify IPv6 DNS entries [repos/cloud/cloud-vps/nova_fullstack_test] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/nova_fullstack_test/-/merge_requests/5 (https://phabricator.wikimedia.org/T379356) [13:37:10] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: openstack: nova-fullstack: add support for IPv6 - https://phabricator.wikimedia.org/T379356#10322434 (10aborrero) This is now mostly ready to deploy. There is a new `--ipv6` option that the daemon accepts that will trigger IPv6 DNS checks. It... [13:37:23] 06cloud-services-team, 10Cloud-VPS, 07IPv6, 13Patch-For-Review: openstack: nova-fullstack: add support for IPv6 - https://phabricator.wikimedia.org/T379356#10322435 (10aborrero) 05Open→03In progress [13:42:45] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 06Content-Transform-Team-WIP: Rebuild or delete deployment-docker-proton01 - https://phabricator.wikimedia.org/T369916#10322441 (10hashar) 05Resolved→03Open [[ https://openstack-browser.toolforge.org/server/d... [13:54:13] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): [jobs-api,jobs-emailer] Prometheus monitoring toolforge-jobs server side components - https://phabricator.wikimedia.org/T320284#10322468 (10dcaro) a:03dcaro [13:54:29] 06cloud-services-team, 10Toolforge (Toolforge iteration 16): [jobs-api,jobs-emailer] Prometheus monitoring toolforge-jobs server side components - https://phabricator.wikimedia.org/T320284#10322467 (10dcaro) [13:57:03] (03open) 10dcaro: webserver: add a minimal metrics endpoint [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/7 (https://phabricator.wikimedia.org/T320284) [13:57:07] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [jobs-api,jobs-emailer] Prometheus monitoring toolforge-jobs server side components - https://phabricator.wikimedia.org/T320284#10322484 (10dcaro) 05Open→03In progress [13:58:11] 10Toolforge (Toolforge iteration 16): [jobs-emailer] deploy on lima-kilo - https://phabricator.wikimedia.org/T379917 (10dcaro) 03NEW [13:58:33] 06cloud-services-team, 10Toolforge (Toolforge iteration 16), 07Kubernetes: [jobs-api] Allow Toolforge scheduled jobs to have a maximum runtime - https://phabricator.wikimedia.org/T306391#10322473 (10dcaro) 05Stalled→03In progress a:03dcaro [13:59:45] 10Toolforge (Toolforge iteration 16): [jobs-emailer] deploy on lima-kilo - https://phabricator.wikimedia.org/T379917#10322499 (10dcaro) p:05Triage→03Medium a:03dcaro [14:07:31] 10Toolforge (Toolforge iteration 16): [jobs-emailer] deploy on lima-kilo - https://phabricator.wikimedia.org/T379917#10322560 (10dcaro) [14:13:51] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:18:50] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:19:05] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 10Beta-Cluster-Infrastructure, 06Content-Transform-Team-WIP: Rebuild or delete deployment-docker-proton01 - https://phabricator.wikimedia.org/T369916#10322602 (10fnegri) p:05Triage→03Low If it's not deleted by tomorrow, it will be deleted... [14:31:54] (03open) 10dcaro: toolforge: deploy jobs-emailer too [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/213 (https://phabricator.wikimedia.org/T379917) [14:33:36] (03open) 10dcaro: jobs-emailer: make explicit the config on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/597 (https://phabricator.wikimedia.org/T379917) [14:36:37] (03open) 10dcaro: add jobs emailer [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/598 (https://phabricator.wikimedia.org/T379917) [14:37:36] (03open) 10dcaro: toolforge_get_versions: don't fail on missing charts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/599 [14:41:43] (03update) 10dcaro: toolforge_get_versions: don't fail on missing charts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/599 [14:42:24] (03update) 10dcaro: toolforge_get_versions: don't fail on missing charts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/599 [14:43:56] (03update) 10dcaro: toolforge_get_versions: don't fail on missing charts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/599 [14:44:58] (03update) 10dcaro: toolforge_get_versions: don't fail on missing charts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/599 [14:45:57] (03update) 10dcaro: toolforge_get_versions: don't fail on missing charts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/599 [14:48:59] (03update) 10dcaro: add jobs emailer [repos/cloud/toolforge/toolforge-deploy] (add_jobs_emailer_config_local) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/598 (https://phabricator.wikimedia.org/T379917) [14:49:07] (03update) 10dcaro: add jobs emailer [repos/cloud/toolforge/toolforge-deploy] (add_jobs_emailer_config_local) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/598 (https://phabricator.wikimedia.org/T379917) [14:54:03] (03approved) 10fnegri: add jobs emailer [repos/cloud/toolforge/toolforge-deploy] (add_jobs_emailer_config_local) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/598 (https://phabricator.wikimedia.org/T379917) (owner: 10dcaro) [14:54:19] (03approved) 10fnegri: jobs-emailer: make explicit the config on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/597 (https://phabricator.wikimedia.org/T379917) (owner: 10dcaro) [14:54:43] (03approved) 10fnegri: toolforge: deploy jobs-emailer too [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/213 (https://phabricator.wikimedia.org/T379917) (owner: 10dcaro) [14:55:18] (03update) 10dcaro: webserver: add a minimal metrics endpoint [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/7 (https://phabricator.wikimedia.org/T320284) [14:55:39] (03merge) 10dcaro: jobs-emailer: make explicit the config on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/597 (https://phabricator.wikimedia.org/T379917) [14:55:40] (03update) 10dcaro: add jobs emailer [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/598 (https://phabricator.wikimedia.org/T379917) [14:56:28] (03merge) 10dcaro: add jobs emailer [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/598 (https://phabricator.wikimedia.org/T379917) [14:57:15] (03merge) 10dcaro: toolforge: deploy jobs-emailer too [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/213 (https://phabricator.wikimedia.org/T379917) [14:58:32] (03update) 10dcaro: toolforge_deploy_mr: allow downloading packages from forks [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/211 [14:58:33] 06cloud-services-team, 10Cloud-VPS, 10Striker, 13Patch-For-Review: openstack: wmfkeystonehooks: project ids rather than names are being used in LDAP group creation - https://phabricator.wikimedia.org/T379030#10322740 (10Andrew) It's definitely the case that we enforce unique project names in eqiad1. So the... [14:58:35] (03update) 10dcaro: toolforge_deploy_mr: allow downloading packages from forks [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/211 [15:01:22] (03update) 10dcaro: webserver: add a minimal metrics endpoint [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/7 (https://phabricator.wikimedia.org/T320284) [15:01:54] (03update) 10dcaro: toolforge_get_versions: don't fail on missing charts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/599 [15:03:37] (03approved) 10fnegri: toolforge_get_versions: don't fail on missing charts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/599 (owner: 10dcaro) [15:04:00] (03approved) 10fnegri: toolforge_deploy_mr: allow downloading packages from forks [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/211 (owner: 10dcaro) [15:06:08] 10Toolforge (Toolforge iteration 16): [jobs-emailer] http requests are blocked by the loops - https://phabricator.wikimedia.org/T379924 (10dcaro) 03NEW [15:06:12] 10Toolforge (Toolforge iteration 16): [jobs-emailer] http requests are blocked by the loops - https://phabricator.wikimedia.org/T379924#10322762 (10dcaro) p:05Triage→03Medium [15:07:26] 10Toolforge (Toolforge iteration 16), 13Patch-For-Review: [jobs-emailer] deploy on lima-kilo - https://phabricator.wikimedia.org/T379917#10322748 (10dcaro) 05Open→03Resolved [15:08:01] (03approved) 10dcaro: toolforge_deploy_mr: allow downloading packages from forks [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/211 [15:08:39] (03merge) 10dcaro: toolforge_deploy_mr: allow downloading packages from forks [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/211 [15:10:02] (03merge) 10dcaro: toolforge_get_versions: don't fail on missing charts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/599 [15:10:03] (03update) 10dcaro: toolforge_get_versions: don't fail on missing charts [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/599 [15:20:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:28:14] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 07Puppet: Puppet removed "nameserver" line from /etc/resolv.conf - https://phabricator.wikimedia.org/T379927 (10fnegri) 03NEW [15:29:05] 06cloud-services-team, 10Cloud-VPS, 06Infrastructure-Foundations, 07Puppet: Puppet removed "nameserver" line from /etc/resolv.conf - https://phabricator.wikimedia.org/T379927#10322866 (10fnegri) 05Open→03Resolved a:03fnegri The issue is resolved, I created this task to track it in case it happens... [15:30:29] FIRING: PuppetAgentFailure: Puppet agent failure detected on instance tools-db-4 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [15:38:28] RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance ntp-04 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [16:00:08] 10cloud-services-team (FY2024/2025-Q1-Q2): [cloudcumin] After the upgrade to spicerack 8.15.2, the wmcs.openstack.cloudvirt.vm_console cookbook stopped working - https://phabricator.wikimedia.org/T379570#10323001 (10fnegri) I can reproduce even without Python: ` root@cloudcumin2001:~# SSH_AUTH_SOCK=/run/keyhold... [16:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:28:42] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component tools-webservice [16:28:44] !log dcaro@cloudcumin1001 toolsbeta END (FAIL) - Cookbook wmcs.toolforge.component.deploy (exit_code=99) for component tools-webservice [16:29:19] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.component.deploy for component builds-cli [16:33:57] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.component.deploy (exit_code=0) for component builds-cli [16:36:12] 10Toolforge (Toolforge iteration 16): [jobs-emailer] http requests are blocked by the loops - https://phabricator.wikimedia.org/T379924#10323182 (10aborrero) The easier solution is to run the webserver task in a different thread. Running it in the same thread as the other emailer tasks was a conscious decision... [16:36:53] 10Toolforge (Toolforge iteration 16): [jobs-emailer] http requests are blocked by the loops - https://phabricator.wikimedia.org/T379924#10323187 (10aborrero) [16:45:16] 10Toolforge (Toolforge iteration 16): [jobs-emailer] http requests are blocked by the loops - https://phabricator.wikimedia.org/T379924#10323232 (10dcaro) >>! In T379924#10323182, @aborrero wrote: > The easier solution is to run the webserver task in a different thread. > > Running it in the same thread as the... [16:49:32] 10cloud-services-team (FY2024/2025-Q1-Q2): [cloudcumin] After the upgrade to spicerack 8.15.2, the wmcs.openstack.cloudvirt.vm_console cookbook stopped working - https://phabricator.wikimedia.org/T379570#10323271 (10dcaro) >>! In T379570#10323001, @fnegri wrote: > I can reproduce even without Python: > > ` > ro... [16:51:12] 10Toolforge (Toolforge iteration 16): [jobs-emailer] http requests are blocked by the loops - https://phabricator.wikimedia.org/T379924#10323276 (10dcaro) >>! In T379924#10323232, @dcaro wrote: >>>! In T379924#10323182, @aborrero wrote: >> The easier solution is to run the webserver task in a different thread. >... [16:55:32] (03merge) 10fluq: Fix range error of [A-z] [toolforge-repos/svbot2] - 10https://gitlab.wikimedia.org/toolforge-repos/svbot2/-/merge_requests/1 (owner: 10hartman) [17:19:09] 10Tools: In Glamorous the square brackets [ ] in the link cause these links to break in wikitext - https://phabricator.wikimedia.org/T379721#10323425 (10Prototyperspective) [17:19:59] 10Tools: In Glamorous enable limiting the number of uses shown by default - https://phabricator.wikimedia.org/T379725#10323431 (10Prototyperspective) [17:21:17] 10Tools: In Glamorous the square brackets [ ] in the link cause these links to break in wikitext - https://phabricator.wikimedia.org/T379721#10323442 (10Prototyperspective) [17:29:31] 10cloud-services-team (FY2024/2025-Q1-Q2), 10Toolforge (Toolforge iteration 16), 05Goal: [toolsdb] Upgrade to MariaDB 10.6 - https://phabricator.wikimedia.org/T352206#10323471 (10fnegri) I'm creating a new replica **tools-db-4** following the procedure at https://wikitech.wikimedia.org/wiki/Portal:Toolforge/... [17:32:58] RESOLVED: PuppetAgentFailure: Puppet agent failure detected on instance tools-db-4 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentFailure [17:45:35] (03open) 10dcaro: package: adopted the common setup [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/8 [17:49:21] (03update) 10dcaro: package: adopted the common setup [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/8 [17:50:31] (03update) 10dcaro: package: adopted the common setup [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/8 [17:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:52:59] (03update) 10dcaro: package: adopted the common setup [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/8 [17:55:02] (03update) 10dcaro: package: adopted the common setup [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/8 [17:56:16] (03update) 10dcaro: package: adopted the common setup [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/8 [18:00:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:02:15] (03open) 10dcaro: jobs-emailer: pull always on lima-kilo [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/600 [18:02:28] (03update) 10dcaro: package: adopted the common setup [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/8 [18:03:33] (03update) 10dcaro: webserver: add a minimal metrics endpoint [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/7 (https://phabricator.wikimedia.org/T320284) [18:08:32] (03update) 10dcaro: package: adopted the common setup [repos/cloud/toolforge/jobs-emailer] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-emailer/-/merge_requests/8 [18:54:03] 06cloud-services-team, 10Toolforge: Toolforge: consider introducing a command line for creating reverse proxies - https://phabricator.wikimedia.org/T337191#10323873 (10GPSLeo) I just noticed that this method is currently producing 421 errors as proxy for tile.openstreetmap.org. Other services work. I had t... [19:50:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:30:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks