[00:02:48] 10PAWS: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T366226#9844695 (10LibUp-bot) [00:02:53] 10Toolforge: New upstream release for Pywikibot - https://phabricator.wikimedia.org/T363631#9844697 (10LibUp-bot) A new upstream version of Pywikibot is now available: 9.1.3. * https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Pywikibot_image * https://gerrit.wikimedia.org/g/pywikibot/core/+/refs/tags/9... [00:09:49] FIRING: TfInfraTestApplyFailed: Terraform failed to apply/create the resources on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [02:11:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:15:32] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack [02:18:49] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) [02:21:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:43:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:53:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:04:50] FIRING: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:09:50] RESOLVED: ProbeDown: Service tools-static-15:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-15:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [05:13:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:20:02] 10Tool-toolwatch, 06Indic-MediaWiki-Developers: Enhance Tool Watch to Track Tool Liveliness and Display Graphs - https://phabricator.wikimedia.org/T365857#9845031 (10Gopavasanth) 05Open→03Resolved a:03Hrideshmg Thank you @Hrideshmg, @Rihaan180 for your amazing contributions to Tool-Watch! Now with... [05:23:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:43:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:53:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:40:03] 10Toolforge (Toolforge iteration 10): [jobs-api] api endpoint that returns all the default values of a job from the backend - https://phabricator.wikimedia.org/T366209#9845213 (10dcaro) This would go away if we move that logic to the API right? Once this is in {T362075} the jobs.yaml can become 'toolforge deplo... [08:19:45] 06cloud-services-team, 10Cloud-VPS, 05Cloud-Services-Origin-Alert: [cloudcephosd1031] re-introduce sda to the OS raid - https://phabricator.wikimedia.org/T366240 (10dcaro) 03NEW [08:31:25] 14Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9845358 (10dcaro) >>! In T320164#9839361, @MBH wrote: > @dcaro A script `sandbox.py` generates an error `Skipped '/workspace/user-config.py': owned by someone... [08:41:43] 14Grid-Engine-to-K8s-Migration, 10Wiki-Loves-Monuments-Database: Migrate heritage from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319787#9845392 (10dcaro) >>! In T319787#9828796, @JeanFred wrote: > Thanks for the answer!: > > Here’s the toolhub record: https://toolhub.wi... [08:45:23] (03update) 10sstefanova: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T363808) [09:01:14] 10Cloud-VPS (Quota-requests): Increase Object Storage quota for QRank - https://phabricator.wikimedia.org/T366244#9845459 (10Aklapper) @Sascha: Hi, please follow the steps in https://phabricator.wikimedia.org/project/view/2880/ - thanks. [09:05:35] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [09:24:39] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [09:30:29] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [09:41:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:46:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:51:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:56:28] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [09:56:40] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [09:56:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:57:11] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [10:05:09] 10Toolforge (Toolforge iteration 10): [toolforge-cli,jobs-cli,builds-cli,envvars-cli] Explore OpenAPI SDK tooling for client consolidation - https://phabricator.wikimedia.org/T356261#9845772 (10Slst2020) Trying with just the builds-api openapi.yaml and openapi-generator, it doesn't like `.common` section ` (ope... [10:41:10] 10Toolforge: toolforge: ldap: disabling an account creates 2 entries in the LDAP tree - https://phabricator.wikimedia.org/T366263 (10aborrero) 03NEW [10:51:19] 10Toolforge: toolforge: ldap: disabling an account creates 2 entries in the LDAP tree - https://phabricator.wikimedia.org/T366263#9845956 (10dcaro) the dns are different :/, one has `cn=toolsbeta.test8` the other `uid=toolsbeta.test8`, other account don't have the `cn=toolsbeta.test8`, maybe it was created by mi... [10:52:52] 10Toolforge: toolforge: ldap: disabling an account creates 2 entries in the LDAP tree - https://phabricator.wikimedia.org/T366263#9845961 (10dcaro) oh, wait, some use uid=toolsbeta.*, some use cn=toolsbeta.* :/ ` root@mwmaint1002:~# ldapsearch -x uid=toolsbeta.test4 + | grep dn: dn: cn=toolsbeta.test4,ou=people... [11:02:40] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [11:09:43] 10Toolforge: toolforge: ldap: disabling an account creates 2 entries in the LDAP tree - https://phabricator.wikimedia.org/T366263#9846002 (10dcaro) hmm, the docs for toolsbeta creation say `cn=` https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolsbeta#create_a_tool_account_in_toolsbeta striker user u... [11:54:09] 10Toolforge: toolforge: ldap: disabling an account creates 2 entries in the LDAP tree - https://phabricator.wikimedia.org/T366263#9846110 (10aborrero) Before extending the mess, I would like to confirm. It seems one is the posixgroup (dn: cn=) and the other is the actual possixAccount (dn: uid=). [11:56:48] 10Toolforge: toolforge: ldap: disabling an account creates 2 entries in the LDAP tree - https://phabricator.wikimedia.org/T366263#9846117 (10aborrero) >>! In T366263#9846110, @aborrero wrote: > Before extending the mess, I would like to confirm. > > It seems one is the posixgroup (dn: cn=) and the other is the... [12:04:47] 10Toolforge: toolforge: ldap: disabling an account creates 2 entries in the LDAP tree - https://phabricator.wikimedia.org/T366263#9846139 (10aborrero) I believe the wrong entry for the `test8` account was created by myself, by mistake, yesterday when trying to disable the account. The fix I would like to execut... [12:04:59] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [12:05:09] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [12:13:51] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [12:23:57] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [12:31:16] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [12:31:50] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [12:32:11] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/9 [12:35:40] (03approved) 10dcaro: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T363808) (owner: 10sstefanova) [12:36:22] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [12:36:24] (03update) 10dcaro: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T363808) (owner: 10sstefanova) [12:39:46] 10Toolforge (Toolforge iteration 10): [envvars-api] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363809#9846201 (10Slst2020) 05Open→03In progress [12:40:42] 10Toolforge (Toolforge iteration 10): [envvars-api] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363809#9846199 (10Slst2020) a:03Slst2020 [12:44:50] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [12:47:54] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [12:58:25] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [12:59:19] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [13:03:56] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [13:05:13] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [13:08:35] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [13:11:16] 14Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9846380 (10MBH) > You can replace the `logging.info` with bare `print` statements, that will not add the timestamps, but will make them go to stdout (simplest).... [13:11:35] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [13:11:55] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [13:19:41] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [13:26:28] (03update) 10dcaro: functional_tests: add script to run them [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/133 [13:29:51] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [13:31:29] 06cloud-services-team, 10Horizon, 10Fiwiki-Wikidata-Commons: Adding new members to Cloud VPS project fails - https://phabricator.wikimedia.org/T365096#9846528 (10Zache) Ok, now i tested the adding user to the project. So shell name is case sensitive and it worked when i filled only one of the values. Failed... [13:34:12] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [13:36:30] 14Grid-Engine-to-K8s-Migration: Migrate wikisaurusbot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320164#9846551 (10MBH) You mostly fixed these bots in March, but two bots, `sandbox` and `autopurge` (and three tasks, because second bot is runned as two tasks, `autop... [13:39:17] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [13:39:33] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [13:41:10] (03update) 10dcaro: functional_tests: add script to run them [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/133 [13:47:40] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[39-41] - https://phabricator.wikimedia.org/T363341#9846603 (10Jclark-ctr) [13:48:24] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install cloudcephosd10[35-38] - https://phabricator.wikimedia.org/T363344#9846613 (10Jclark-ctr) [13:56:20] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [13:56:40] (03update) 10dcaro: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) [14:15:55] (03open) 10sstefanova: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/31 (https://phabricator.wikimedia.org/T363809) [14:36:26] 10Toolforge (Toolforge iteration 10): [jobs-api] api endpoint that returns all the default values of a job from the backend - https://phabricator.wikimedia.org/T366209#9846888 (10Raymond_Ndibe) then we rename this task to `move jobs load logic to the jobs-api`? that way it will still be valid for `component-api`... [14:36:36] 10Toolforge (Toolforge iteration 10): [jobs-api] api endpoint that returns all the default values of a job from the backend - https://phabricator.wikimedia.org/T366209#9846890 (10Raymond_Ndibe) also I am working on unifying the parameters [14:52:28] FIRING: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance cloudinfra-cloudvps-puppetserver-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [14:54:40] (03approved) 10aborrero: functional_tests: add script to run them [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/133 (owner: 10dcaro) [14:57:28] RESOLVED: PuppetSyncFailure: Failed to update Puppet repository /srv/git/operations/puppet on instance cloudinfra-cloudvps-puppetserver-1 in project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetSyncFailure [15:07:54] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: cloudvirt1041: can't boot after reimage - https://phabricator.wikimedia.org/T364984#9847000 (10aborrero) I guess I never got to refresh the interfaces in netbox. [15:22:39] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: cloudvirt1041: can't boot after reimage - https://phabricator.wikimedia.org/T364984#9847060 (10aborrero) a:05aborrero→03Andrew Just run the cookbook: ` aborrero@cumin1002:~ $ sudo cookbook sre.network.configure-switch-interfaces cloudvirt1041 Acquir... [15:40:33] 10PAWS: ingress-nginx and prometheus not idempotent - https://phabricator.wikimedia.org/T366121#9847156 (10rook) https://github.com/toolforge/paws/pull/429 seems to work [15:43:19] 10PAWS: ingress-nginx and prometheus not idempotent - https://phabricator.wikimedia.org/T366121#9847189 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/429 [15:43:29] vivian-rook closed https://github.com/toolforge/paws/pull/429 [15:52:30] vivian-rook closed https://github.com/toolforge/paws/pull/425 [15:53:01] 10PAWS: Update chart version on PR? - https://phabricator.wikimedia.org/T365725#9847272 (10rook) 05Open→03Resolved [15:53:05] 10PAWS: Update chart version on PR? - https://phabricator.wikimedia.org/T365725#9847266 (10rook) changes in T366121 seem to have mostly resolved this. Further discovery in T366183 [15:53:09] 10PAWS: Update chart version on PR? - https://phabricator.wikimedia.org/T365725#9847276 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/425 [15:55:24] (03update) 10aborrero: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) (owner: 10dcaro) [15:55:31] (03approved) 10aborrero: jobs-api: add some basic functional tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/288 (https://phabricator.wikimedia.org/T357977) (owner: 10dcaro) [15:56:22] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [15:58:02] 06cloud-services-team: Document and enforce restrictions on openstack project names - https://phabricator.wikimedia.org/T366301 (10Andrew) 03NEW [15:58:20] 10Toolforge (Toolforge iteration 10): [toolforge-cli,jobs-cli,builds-cli,envvars-cli] Explore OpenAPI SDK tooling for client consolidation - https://phabricator.wikimedia.org/T356261#9847339 (10dcaro) I think those changes are mostly because the api changed? We wrap now all the responses with a messages response... [16:00:07] 10Tools, 10MediaViewer, 10Thumbor, 13Patch-For-Review: Explore moving the Panoviewer gadget/Toolforge tool into production - https://phabricator.wikimedia.org/T138933#9847347 (10Sdkb) @tstarling I don't understand the gerritbot comment above. Would you be able to give an update on where this task is at? [16:15:56] 10Cloud-VPS (Project-requests): TfInfraTest project - https://phabricator.wikimedia.org/T365822#9847438 (10rook) Changing project name to lowercase only. ` root@cloudcontrol1005:~# openstack project delete TfInfraTest root@cloudcontrol1005:~# openstack project create --description 'tofu infra tests' tofuinfr... [16:27:51] !log andrew@cloudcumin1001 cloudvirt-canary START - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary on eqiad1, with recreate False, for hosts list: ['cloudvirt1041'] [16:28:04] !log andrew@cloudcumin1001 cloudvirt-canary END (PASS) - Cookbook wmcs.openstack.cloudvirt.lib.ensure_canary (exit_code=0) on eqiad1, with recreate False, for hosts list: ['cloudvirt1041'] [16:42:14] 06cloud-services-team, 06DC-Ops, 10ops-eqiad, 06SRE: cloudvirt1041: can't boot after reimage - https://phabricator.wikimedia.org/T364984#9847571 (10Andrew) 05Open→03Resolved This host is now pooled and working properly. [17:02:54] (03update) 10dcaro: [jobs-cli] add messages to all responses [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/32 (https://phabricator.wikimedia.org/T356974) (owner: 10raymond-ndibe) [17:14:23] (03update) 10dcaro: [jobs-api] add messages to all responses [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/85 (https://phabricator.wikimedia.org/T356974) (owner: 10raymond-ndibe) [17:32:10] 06cloud-services-team, 10Horizon, 10Fiwiki-Wikidata-Commons: Inconsistency between r/w and r/o ldap - https://phabricator.wikimedia.org/T366310 (10Andrew) 03NEW [17:35:17] 06cloud-services-team, 10Horizon, 10Fiwiki-Wikidata-Commons: Adding new members to Cloud VPS project fails - https://phabricator.wikimedia.org/T365096#9847852 (10Andrew) [17:35:18] 06cloud-services-team, 10Horizon, 10Fiwiki-Wikidata-Commons: Inconsistency between r/w and r/o ldap - https://phabricator.wikimedia.org/T366310#9847850 (10Andrew) [17:35:19] 10Toolforge: toolforge: ldap: disabling an account creates 2 entries in the LDAP tree - https://phabricator.wikimedia.org/T366263#9847851 (10Andrew) [17:36:02] 06cloud-services-team, 10Horizon, 10Fiwiki-Wikidata-Commons, 07LDAP: Inconsistency between r/w and r/o ldap - https://phabricator.wikimedia.org/T366310#9847856 (10bd808) [17:36:07] 06cloud-services-team, 10Horizon, 10Fiwiki-Wikidata-Commons, 07LDAP: Inconsistency between r/w and r/o ldap - https://phabricator.wikimedia.org/T366310#9847853 (10Andrew) [17:39:33] 06cloud-services-team, 10Horizon, 10Fiwiki-Wikidata-Commons, 07LDAP: Inconsistency between r/w and r/o ldap - https://phabricator.wikimedia.org/T366310#9847876 (10bd808) {T354916} was a similar problem [17:39:48] 10Toolforge: toolforge: ldap: disabling an account creates 2 entries in the LDAP tree - https://phabricator.wikimedia.org/T366263#9847884 (10Andrew) On serpens (the r/w host) there are 5 of these miscopied records: ` root@cloudcontrol1006:~# ldapsearch -x -E pr=5000/noprompt -H ldap://serpens.wikimedia.org:389... [17:41:29] 10Toolforge, 07LDAP: toolforge: ldap: disabling an account creates 2 entries in the LDAP tree - https://phabricator.wikimedia.org/T366263#9847901 (10bd808) [17:41:35] 10Toolforge, 07LDAP: toolforge: ldap: disabling an account creates 2 entries in the LDAP tree - https://phabricator.wikimedia.org/T366263#9847903 (10Andrew) ...actually there are two different mistakes present. dn: cn=toolsbeta.fourohfour,ou=people,ou=servicegroups,dc=wikimedia,dc=org should be changed to dn:... [17:46:52] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/9 (owner: 10l10n-bot) [17:46:55] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/9 (owner: 10l10n-bot) [19:11:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:26:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:44:15] vivian-rook opened https://github.com/toolforge/paws/pull/430 [19:45:39] vivian-rook closed https://github.com/toolforge/paws/pull/430 [20:05:30] 06cloud-services-team, 10Horizon, 10Fiwiki-Wikidata-Commons, 13Patch-For-Review: Adding new members to Cloud VPS project fails - https://phabricator.wikimedia.org/T365096#9848611 (10Andrew) a:03Andrew [21:56:25] 10Toolforge: ptwikis can't create right URLs for three coded languages - https://phabricator.wikimedia.org/T366327 (10Aram) 03NEW [22:04:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:06:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance cvn-app10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:06:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance extdist-06 on project extdist - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:07:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance tf-bastion on project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:07:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance etcd-discovery-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:10:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-puppetserver-1 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:10:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:11:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:15:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance project-proxy-puppetserver-1 on project project-proxy - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:17:28] FIRING: PuppetAgentNoResources: No Puppet resources found on instance clouddb-services-puppetserver-1 on project clouddb-services - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:20:28] FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:22:28] FIRING: [3x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:26:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:27:28] FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [22:30:11] 10Toolforge: ptwikis can't create right URLs for three coded languages - https://phabricator.wikimedia.org/T366327#9848916 (10Danilo) 05Open→03Resolved a:03Danilo Fixed Note that in the tool side menu there is a "Coordination/bug report" link pointing to a talk page in pt.wikipedia, that is the best p... [22:40:31] (03open) 10bd808: bacc: use `` rather than `
` in alert [toolforge-repos/deploy-commands] - 10https://gitlab.wikimedia.org/toolforge-repos/deploy-commands/-/merge_requests/3
[22:40:47] 	 (03update) 10bd808: bacc: use `` rather than `
` in alert [toolforge-repos/deploy-commands] - 10https://gitlab.wikimedia.org/toolforge-repos/deploy-commands/-/merge_requests/3
[22:51:28] 	 FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[22:51:28] 	 RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance extdist-06 on project extdist   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[22:52:28] 	 RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance tf-bastion on project tf-infra-test   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[22:54:28] 	 RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance gitlab-runners-puppetserver-01 on project gitlab-runners   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[22:55:28] 	 RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance metricsinfra-puppetserver-1 on project metricsinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[22:55:28] 	 FIRING: [2x] PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[22:56:28] 	 FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[22:57:28] 	 FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[23:02:28] 	 RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance clouddb-services-puppetserver-1 on project clouddb-services   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[23:02:34] 	 10Tools: ptwikis can't create right URLs for three coded languages - https://phabricator.wikimedia.org/T366327#9848989 (10JJMC89)
[23:05:28] 	 RESOLVED: [2x] PuppetAgentNoResources: No Puppet resources found on instance bastion on project paws   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[23:05:28] 	 RESOLVED: PuppetAgentNoResources: No Puppet resources found on instance project-proxy-puppetserver-1 on project project-proxy   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[23:07:28] 	 FIRING: [4x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[23:11:28] 	 RESOLVED: [3x] PuppetAgentNoResources: No Puppet resources found on instance cvn-apache10 on project cvn   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources
[23:12:28] 	 RESOLVED: [3x] PuppetAgentNoResources: No Puppet resources found on instance cloudinfra-internal-puppetserver-1 on project cloudinfra   - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources