[00:20:28] (03PS1) 10JHathaway: dummy ssl key [labs/private] - 10https://gerrit.wikimedia.org/r/1038920 [00:22:38] (03CR) 10JHathaway: [C:03+2] dummy ssl key [labs/private] - 10https://gerrit.wikimedia.org/r/1038920 (owner: 10JHathaway) [00:22:41] (03CR) 10JHathaway: [V:03+2 C:03+2] dummy ssl key [labs/private] - 10https://gerrit.wikimedia.org/r/1038920 (owner: 10JHathaway) [00:38:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [00:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:37:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:47:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:09:59] (03update) 10dcaro: functional-tests: add smoke tests for envvars [repos/cloud/toolforge/toolforge-deploy] (add_builds_functional_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/306 (https://phabricator.wikimedia.org/T357977) [07:11:21] (03update) 10sstefanova: openapi: refactor yaml [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/90 [07:13:04] (03update) 10dcaro: toolforge_deploy_mr:add option to restore a component/cli [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/134 [07:14:38] 10wikitech.wikimedia.org, 06DBA, 10MediaModeration, 06Trust and Safety Product Team, 07Wikimedia-production-error: extension1 database for wikitech is always overloaded - https://phabricator.wikimedia.org/T366574#9862436 (10ABran-WMF) p:05Triage→03Medium [07:19:44] (03approved) 10sstefanova: toolforge_deploy_mr:add option to restore a component/cli [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/134 (owner: 10dcaro) [07:20:16] (03update) 10sstefanova: toolforge_deploy_mr:add option to restore a component/cli [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/134 (owner: 10dcaro) [07:21:55] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T366651#9862484 (10Curb_Safe_Charmer) 05Open→03Invalid Working fine right now. [07:24:39] (03update) 10dcaro: [jobs-api] add messages to all responses [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/85 (https://phabricator.wikimedia.org/T356974) (owner: 10raymond-ndibe) [07:31:07] (03merge) 10dcaro: toolforge_deploy_mr:add option to restore a component/cli [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/134 [07:33:17] 10Toolforge (Toolforge iteration 11): [builds-api, envvars-api] Refactor openapi yaml to comply with OAPI spec 3.1.0 - https://phabricator.wikimedia.org/T366668 (10Slst2020) 03NEW [07:36:05] (03update) 10sstefanova: openapi: refactor yaml [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/90 [07:36:05] 10Toolforge (Toolforge iteration 11): [jobs-api, builds-api, envvars-api] Refactor openapi yaml to comply with OAPI spec 3.1.0 - https://phabricator.wikimedia.org/T366668#9862520 (10Slst2020) a:03Slst2020 [07:36:15] (03update) 10sstefanova: openapi: refactor yaml [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/90 [07:38:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:43:35] 10Toolforge (Toolforge iteration 11): [jobs-api, builds-api, envvars-api] Refactor openapi yaml to comply with OAPI spec 3.1.0 - https://phabricator.wikimedia.org/T366668#9862532 (10Slst2020) 05Open→03In progress [07:48:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:53:40] 10Toolforge (Toolforge iteration 11): [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#9862549 (10dcaro) [08:00:34] (03merge) 10aborrero: alerts: add maintain-kubeusers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/12 (https://phabricator.wikimedia.org/T366598) [08:04:45] (03approved) 10dcaro: openapi: refactor yaml [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/90 (owner: 10sstefanova) [08:04:49] (03update) 10dcaro: openapi: refactor yaml [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/90 (owner: 10sstefanova) [08:04:54] (03update) 10sstefanova: openapi: refactor yaml [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/90 [08:13:41] (03update) 10sstefanova: openapi: refactor yaml [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/90 [08:14:19] (03update) 10dcaro: [jobs-api] add messages to all responses [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/85 (https://phabricator.wikimedia.org/T356974) (owner: 10raymond-ndibe) [08:14:40] (03update) 10sstefanova: openapi: refactor yaml [repos/cloud/toolforge/jobs-api] (add_messages_to_all_responses) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/90 [08:14:48] (03update) 10dcaro: [jobs-cli] add messages to all responses [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/32 (https://phabricator.wikimedia.org/T356974) (owner: 10raymond-ndibe) [08:17:11] (03update) 10sstefanova: openapi: refactor yaml [repos/cloud/toolforge/jobs-api] (add_messages_to_all_responses) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/90 [08:18:02] 10Toolforge (Toolforge iteration 11): [jobs-api, builds-api, envvars-api] Make openapi yaml comply with OAPI spec 3.1.0 - https://phabricator.wikimedia.org/T366668#9862585 (10taavi) [08:19:19] (03update) 10dcaro: [jobs-cli] add messages to all responses [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/32 (https://phabricator.wikimedia.org/T356974) (owner: 10raymond-ndibe) [08:19:19] (03approved) 10dcaro: [jobs-cli] add messages to all responses [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/32 (https://phabricator.wikimedia.org/T356974) (owner: 10raymond-ndibe) [08:19:32] (03merge) 10dcaro: [jobs-cli] add messages to all responses [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/32 (https://phabricator.wikimedia.org/T356974) (owner: 10raymond-ndibe) [08:19:42] (03merge) 10sstefanova: openapi: refactor yaml [repos/cloud/toolforge/jobs-api] (add_messages_to_all_responses) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/90 [08:19:46] (03update) 10sstefanova: [jobs-api] add messages to all responses [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/85 (owner: 10raymond-ndibe) [08:24:43] (03open) 10dcaro: d/changelog: bump to 0.103.6 [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/40 (https://phabricator.wikimedia.org/T361437) [08:29:39] (03open) 10dcaro: functional-tests: allow running from any directory [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/309 [08:29:41] (03update) 10sstefanova: cli: use prefixed endpoints [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/43 [08:30:14] (03merge) 10sstefanova: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/31 (https://phabricator.wikimedia.org/T363809) [08:31:27] (03approved) 10aborrero: functional-tests: allow running from any directory [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/309 (owner: 10dcaro) [08:32:33] (03approved) 10dcaro: functional-tests: allow running from any directory [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/309 [08:32:33] (03open) 10dcaro: d/changelog: bump to 16.0.10 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/35 (https://phabricator.wikimedia.org/T356974) [08:32:35] (03update) 10dcaro: functional-tests: allow running from any directory [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/309 [08:32:36] (03merge) 10dcaro: functional-tests: allow running from any directory [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/309 [08:34:31] (03PS1) 10Muehlenhoff: Remove obsolete stub certs [labs/private] - 10https://gerrit.wikimedia.org/r/1039173 (https://phabricator.wikimedia.org/T360414) [08:35:37] (03merge) 10sstefanova: cli: use prefixed endpoints [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/71 [08:35:56] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: envvars-api: bump to 0.0.46-20240605083025-b81e0804 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/310 (https://phabricator.wikimedia.org/T363809) [08:37:22] (03PS1) 10Majavah: migrations: Re-format with latest black [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1039174 [08:37:22] (03PS1) 10Majavah: Fix support for keystone projects where name is not equal to ID [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1039175 [08:38:22] (03CR) 10Majavah: [C:03+2] migrations: Re-format with latest black [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1039174 (owner: 10Majavah) [08:38:50] (03Merged) 10jenkins-bot: migrations: Re-format with latest black [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1039174 (owner: 10Majavah) [08:39:43] 10Tool-schedule-deployment, 10Gerrit, 13Patch-For-Review: Link to https://schedule-deployment.toolforge.org/backport/{change-id} from changes eligable for deployment in a backport window - https://phabricator.wikimedia.org/T366512#9862626 (10hashar) I have deployed Bryan plugin and a currently open change of... [08:43:46] (03CR) 10David Caro: [C:03+1] "LGTM" [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1039175 (owner: 10Majavah) [08:44:46] (03CR) 10Majavah: [C:03+2] Fix support for keystone projects where name is not equal to ID (031 comment) [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1039175 (owner: 10Majavah) [08:45:11] (03Merged) 10jenkins-bot: Fix support for keystone projects where name is not equal to ID [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1039175 (owner: 10Majavah) [08:48:08] (03update) 10dcaro: d/changelog: bump to 16.0.10 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/35 (https://phabricator.wikimedia.org/T356974) [08:48:09] (03approved) 10dcaro: d/changelog: bump to 16.0.10 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/35 (https://phabricator.wikimedia.org/T356974) [08:48:14] (03merge) 10dcaro: d/changelog: bump to 16.0.10 [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/35 (https://phabricator.wikimedia.org/T356974) [08:49:10] (03PS1) 10Majavah: maintain-projects: Fix typo in variable name [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1039181 [08:50:03] (03CR) 10Majavah: [C:03+2] maintain-projects: Fix typo in variable name [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1039181 (owner: 10Majavah) [08:50:25] (03Merged) 10jenkins-bot: maintain-projects: Fix typo in variable name [cloud/metricsinfra/prometheus-manager] - 10https://gerrit.wikimedia.org/r/1039181 (owner: 10Majavah) [08:56:35] (03approved) 10aborrero: functional-tests: add smoke build tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/307 (https://phabricator.wikimedia.org/T357977) (owner: 10dcaro) [08:57:08] (03approved) 10aborrero: functional-tests: add smoke tests for envvars [repos/cloud/toolforge/toolforge-deploy] (add_builds_functional_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/306 (https://phabricator.wikimedia.org/T357977) (owner: 10dcaro) [08:57:38] (03update) 10dcaro: functional-tests: add smoke build tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/307 (https://phabricator.wikimedia.org/T357977) [08:58:34] (03update) 10dcaro: functional-tests: add smoke tests for envvars [repos/cloud/toolforge/toolforge-deploy] (add_builds_functional_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/306 (https://phabricator.wikimedia.org/T357977) [08:58:53] (03update) 10dcaro: functional-tests: add smoke build tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/307 (https://phabricator.wikimedia.org/T357977) [08:58:58] (03merge) 10dcaro: functional-tests: add smoke build tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/307 (https://phabricator.wikimedia.org/T357977) [08:58:59] (03update) 10dcaro: functional-tests: add smoke tests for envvars [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/306 (https://phabricator.wikimedia.org/T357977) [08:59:11] (03update) 10dcaro: functional-tests: add smoke tests for envvars [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/306 (https://phabricator.wikimedia.org/T357977) [08:59:17] (03merge) 10dcaro: functional-tests: add smoke tests for envvars [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/306 (https://phabricator.wikimedia.org/T357977) [08:59:18] 06cloud-services-team, 10Toolforge: lima-kilo: container image caching - https://phabricator.wikimedia.org/T362967#9862710 (10aborrero) with the introduction of functional tests via toolforge-deploy, this is even more relevant today. [09:02:00] (03open) 10sstefanova: utils: update bump script [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/73 [09:14:29] 10Toolforge (Toolforge iteration 11): [jobs-cli,envvars-cli,webservice] update package/version scripts to use bookworm - https://phabricator.wikimedia.org/T366674 (10dcaro) 03NEW [09:14:31] 10Toolforge (Toolforge iteration 11): [jobs-cli,envvars-cli,webservice] update package/version scripts to use bookworm - https://phabricator.wikimedia.org/T366674#9862747 (10dcaro) p:05Triage→03High [09:16:41] 10Toolforge (Toolforge iteration 11): [jobs-cli,envvars-cli,webservice] update package/version scripts to use bookworm - https://phabricator.wikimedia.org/T366674#9862760 (10Slst2020) a:03Slst2020 [09:18:08] 10Toolforge (Toolforge iteration 11): [jobs-cli,envvars-cli,webservice] update package/version scripts to use bookworm - https://phabricator.wikimedia.org/T366674#9862762 (10Slst2020) 05Open→03In progress [09:19:30] (03open) 10dcaro: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) [09:20:05] (03update) 10dcaro: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) [09:21:12] (03update) 10dcaro: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) [09:25:08] (03update) 10dcaro: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) [09:28:44] 10wikitech.wikimedia.org, 06DBA, 10MediaModeration, 06Trust and Safety Product Team, 07Wikimedia-production-error: wikitech/cloudweb hosts cannot access x1 - https://phabricator.wikimedia.org/T366574#9862782 (10taavi) [09:29:47] (03update) 10dcaro: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) [09:29:49] (03update) 10dcaro: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) [09:32:19] (03CR) 10Andrea Denisse: [C:03+1] "LGTM, thank you!" [labs/private] - 10https://gerrit.wikimedia.org/r/1039173 (https://phabricator.wikimedia.org/T360414) (owner: 10Muehlenhoff) [09:33:22] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove obsolete stub certs [labs/private] - 10https://gerrit.wikimedia.org/r/1039173 (https://phabricator.wikimedia.org/T360414) (owner: 10Muehlenhoff) [09:37:56] (03PS1) 10Muehlenhoff: Remove obsolete LDAP stub secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1039184 [09:45:19] (03update) 10ladsgroup: Add support for switchover of ES hosts [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/3 (https://phabricator.wikimedia.org/T365098) [09:48:40] (03update) 10dcaro: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) [09:50:21] 10Tool-openstack-browser: openstack-browser support for projects where id != name - https://phabricator.wikimedia.org/T366679 (10taavi) 03NEW [09:51:06] (03update) 10ladsgroup: Add support for switchover of ES hosts [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/3 (https://phabricator.wikimedia.org/T365098) [09:51:40] (03approved) 10marostegui: Add support for switchover of ES hosts [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/3 (https://phabricator.wikimedia.org/T365098) (owner: 10ladsgroup) [09:51:41] (03update) 10ladsgroup: Add support for switchover of ES hosts [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/3 (https://phabricator.wikimedia.org/T365098) [09:51:44] (03PS1) 10Muehlenhoff: Remove tendril stub cert [labs/private] - 10https://gerrit.wikimedia.org/r/1039189 [09:54:39] (03merge) 10ladsgroup: Add support for switchover of ES hosts [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/3 (https://phabricator.wikimedia.org/T365098) [09:55:20] 06cloud-services-team, 10Toolforge (Toolforge iteration 11): maintain-kubeusers: metrics, monitoring and alerting - https://phabricator.wikimedia.org/T366598#9862892 (10aborrero) 05Open→03Resolved a:03aborrero we have basic alerts and a basic dashboard https://grafana-rw.wmcloud.org/d/E9HyRBySk/maint... [09:58:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [10:07:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:12:05] PROBLEM - Host cloudcephosd1031 is DOWN: PING CRITICAL - Packet loss = 100% [10:12:47] FIRING: NodeDown: The node cloudcephosd1031 is unreachable. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/NodeDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1031 - https://alerts.wikimedia.org/?q=alertname%3DNodeDown [10:15:02] (03PS1) 10Muehlenhoff: Remove kartotherian stub cert [labs/private] - 10https://gerrit.wikimedia.org/r/1039191 (https://phabricator.wikimedia.org/T360778) [10:17:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:31:56] FIRING: [2x] SystemdUnitDown: The service unit dumps-fetch-pagecounts-ez.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [10:33:06] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove kartotherian stub cert [labs/private] - 10https://gerrit.wikimedia.org/r/1039191 (https://phabricator.wikimedia.org/T360778) (owner: 10Muehlenhoff) [10:38:01] (03update) 10dcaro: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) [10:41:56] FIRING: [4x] SystemdUnitDown: The service unit dumps-fetch-media-contestwinners.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [10:42:53] (03open) 10aborrero: homedir: use less filesystem checks [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/34 (https://phabricator.wikimedia.org/T366564) [10:45:39] RESOLVED: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [10:57:03] 10Toolforge: Investigate AdvertiseDetector Outage - https://phabricator.wikimedia.org/T366690 (10SrikanthLunavath03) 03NEW [10:58:18] 10Tools: Investigate AdvertiseDetector Outage - https://phabricator.wikimedia.org/T366690#9863205 (10taavi) [11:02:56] (03update) 10sstefanova: utils: update bump script [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/73 [11:03:58] (03update) 10sstefanova: utils: update deb build and bump setup [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/73 [11:04:00] 06cloud-services-team, 10Cloud-VPS, 05Cloud-Services-Origin-Alert: [cloudcephosd1031] re-introduce sda to the OS raid - https://phabricator.wikimedia.org/T366240#9863216 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by jiji@cumin1002 for host cloudcephosd1031.eqiad.wmnet with OS bul... [11:06:46] (03update) 10sstefanova: utils: update deb build and bump setup [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/73 [11:09:11] RECOVERY - Host cloudcephosd1031 is UP: PING OK - Packet loss = 0%, RTA = 0.27 ms [11:15:00] (03update) 10aborrero: components: add kyverno [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/238 (https://phabricator.wikimedia.org/T279110) [11:21:21] FIRING: MaintainKubeusersHang: maintain-kubeusers last finished run is very old - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersHang [11:21:56] FIRING: [6x] SystemdUnitDown: The service unit dumps-fetch-media-contestwinners.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [11:26:21] RESOLVED: MaintainKubeusersHang: maintain-kubeusers last finished run is very old - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersHang [11:26:56] FIRING: [6x] SystemdUnitDown: The service unit dumps-fetch-media-contestwinners.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [11:31:56] FIRING: [6x] SystemdUnitDown: The service unit dumps-fetch-media-contestwinners.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [11:37:36] 06cloud-services-team, 10Cloud-VPS, 05Cloud-Services-Origin-Alert: [cloudcephosd1031] re-introduce sda to the OS raid - https://phabricator.wikimedia.org/T366240#9863314 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by jiji@cumin1002 for host cloudcephosd1031.eqiad.wmnet with OS bullsey... [11:41:33] !log dcaro@urcuchillay admin START - Cookbook wmcs.ceph.osd.bootstrap_and_add [11:41:37] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [11:48:09] FIRING: CephClusterInWarning: Ceph cluster in eqiad is in warning status - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephClusterInWarning - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephClusterInWarning [11:48:58] 10Tool-openstack-browser: openstack-browser support for projects where id != name - https://phabricator.wikimedia.org/T366679#9863364 (10taavi) [11:49:00] 06cloud-services-team, 10Cloud-VPS, 07Epic: Wind down use of project ID and project name equivalency in OpenStack - https://phabricator.wikimedia.org/T274268#9863365 (10taavi) [11:50:38] (03open) 10aborrero: maintain-kubeusers: print the time in the hang alert [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/13 (https://phabricator.wikimedia.org/T366598) [11:53:22] 10Tool-phab-ban, 07Technical-Debt: Deploy more recent PhabBanBot code from repository into production - https://phabricator.wikimedia.org/T366587#9863391 (10Agabi10) I just realized that the Deprecated Calls Log shows also `user.enable`, which I don't see even in old versions of the code of the tool. Is it pos... [11:54:29] (03approved) 10dcaro: maintain-kubeusers: print the time in the hang alert [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/13 (https://phabricator.wikimedia.org/T366598) (owner: 10aborrero) [11:54:31] (03update) 10dcaro: maintain-kubeusers: print the time in the hang alert [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/13 (https://phabricator.wikimedia.org/T366598) (owner: 10aborrero) [11:57:41] !log dcaro@urcuchillay admin END (FAIL) - Cookbook wmcs.ceph.osd.bootstrap_and_add (exit_code=99) [11:57:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Admin/SAL [12:01:08] (03open) 10aborrero: cli: initialize maintain_kubeusers_run_finished prometheus metrics [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/35 (https://phabricator.wikimedia.org/T366598) [12:27:12] (03update) 10dcaro: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) [12:30:51] (03update) 10dcaro: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) [12:31:11] (03update) 10dcaro: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) [12:33:27] (03update) 10sstefanova: utils: update deb build and bump setup [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/73 [12:35:34] (03update) 10dcaro: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) [12:42:14] (03update) 10sstefanova: envvars-api: bump to 0.0.46-20240605083025-b81e0804 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/310 (https://phabricator.wikimedia.org/T363809) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [12:43:23] (03update) 10dcaro: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) [12:44:04] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api [12:44:15] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api [12:46:25] (03update) 10dcaro: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) [12:46:26] (03update) 10sstefanova: utils: update deb build and bump setup [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/73 (https://phabricator.wikimedia.org/T366674) [12:46:38] (03update) 10dcaro: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) [12:49:45] (03update) 10aborrero: cli: initialize maintain_kubeusers_run_finished prometheus metrics [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/35 (https://phabricator.wikimedia.org/T366598) [12:51:11] (03update) 10aborrero: cli: initialize maintain_kubeusers_run_finished prometheus metrics [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/35 (https://phabricator.wikimedia.org/T366598) [12:55:39] (03merge) 10aborrero: maintain-kubeusers: print the time in the hang alert [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/13 (https://phabricator.wikimedia.org/T366598) [12:58:40] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api [12:58:53] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api [13:04:50] (03update) 10sstefanova: envvars-api: bump to 0.0.46-20240605083025-b81e0804 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/310 (https://phabricator.wikimedia.org/T363809) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [13:05:53] (03approved) 10sstefanova: envvars-api: bump to 0.0.46-20240605083025-b81e0804 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/310 (https://phabricator.wikimedia.org/T363809) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [13:05:58] (03merge) 10sstefanova: envvars-api: bump to 0.0.46-20240605083025-b81e0804 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/310 (https://phabricator.wikimedia.org/T363809) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [13:16:56] FIRING: [2x] SystemdUnitDown: The systemd unit dumps-fetch-wikistats_1.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [13:17:01] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T366704 (10phaultfinder) 03NEW [13:19:38] 06cloud-services-team: SystemdUnitDown - https://phabricator.wikimedia.org/T366704#9863698 (10taavi) →14Duplicate dup:03T366694 [13:21:56] RESOLVED: [6x] SystemdUnitDown: The service unit dumps-fetch-media-contestwinners.service is in failed status on host clouddumps1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [13:21:56] RESOLVED: [2x] SystemdUnitDown: The systemd unit dumps-fetch-wikistats_1.service on node clouddumps1001 has been failing for more than two hours. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [13:28:58] (03approved) 10dcaro: d/changelog: bump to 0.103.6 [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/40 (https://phabricator.wikimedia.org/T361437) [13:29:00] (03update) 10dcaro: d/changelog: bump to 0.103.6 [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/40 (https://phabricator.wikimedia.org/T361437) [13:29:09] (03merge) 10dcaro: d/changelog: bump to 0.103.6 [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/40 (https://phabricator.wikimedia.org/T361437) [13:31:12] 10Toolforge: tools-static webserver shows directory listing instead of the index file - https://phabricator.wikimedia.org/T366482#9863746 (10taavi) p:05Triage→03Low I'm curious about the use case here - tools-static is mostly intended for hosting static assets/images/etc, not full websites accessed by users. [13:33:18] 10Toolforge (Toolforge iteration 11), 13Patch-For-Review: [webservice-cli] `webservice logs -f` should expect KeyboardInterrupt - https://phabricator.wikimedia.org/T361437#9863768 (10dcaro) Released! {F54952499} [13:33:25] 10Toolforge (Toolforge iteration 11), 13Patch-For-Review: [webservice-cli] `webservice logs -f` should expect KeyboardInterrupt - https://phabricator.wikimedia.org/T361437#9863772 (10dcaro) Thanks a lot for the fix! :) [13:33:38] 10Toolforge (Toolforge iteration 11), 13Patch-For-Review: [webservice-cli] `webservice logs -f` should expect KeyboardInterrupt - https://phabricator.wikimedia.org/T361437#9863771 (10dcaro) 05In progress→03Resolved [13:35:49] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: decommission cloudcontrol2001-dev.codfw.wmnet - https://phabricator.wikimedia.org/T364577#9863782 (10Jhancock.wm) 05Open→03Resolved [13:36:36] (03update) 10sstefanova: utils: update deb build and bump setup [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/73 (https://phabricator.wikimedia.org/T366674) [13:37:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:40:43] !log taavi@cloudcumin1001 taavitestwikitech START - Cookbook wmcs.vps.create_project for project taavitestwikitech in eqiad1 [13:40:44] taavi@cloudcumin1001: Unknown project "taavitestwikitech" [13:40:46] !log taavi@cloudcumin1001 taavitestwikitech END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project taavitestwikitech in eqiad1 [13:40:46] taavi@cloudcumin1001: Unknown project "taavitestwikitech" [13:42:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:43:54] (03update) 10dcaro: create: add nicer error when quota is reached [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/29 [13:47:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:49:21] !log taavi@cloudcumin1001 taavitestwikitech START - Cookbook wmcs.vps.create_project for project taavitestwikitech in eqiad1 [13:49:22] taavi@cloudcumin1001: Unknown project "taavitestwikitech" [13:49:23] !log taavi@cloudcumin1001 taavitestwikitech END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project taavitestwikitech in eqiad1 [13:49:23] taavi@cloudcumin1001: Unknown project "taavitestwikitech" [13:52:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:54:12] !log taavi@cloudcumin1001 taavitestwikitech START - Cookbook wmcs.vps.create_project for project taavitestwikitech in eqiad1 [13:54:13] taavi@cloudcumin1001: Unknown project "taavitestwikitech" [13:54:15] !log taavi@cloudcumin1001 taavitestwikitech END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project taavitestwikitech in eqiad1 [13:54:15] taavi@cloudcumin1001: Unknown project "taavitestwikitech" [13:54:53] (03update) 10dcaro: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) [14:00:58] !log taavi@cloudcumin1001 taavitestwikitech START - Cookbook wmcs.vps.create_project for project taavitestwikitech in eqiad1 [14:00:59] taavi@cloudcumin1001: Unknown project "taavitestwikitech" [14:01:32] !log taavi@cloudcumin1001 taavitestwikitech END (PASS) - Cookbook wmcs.vps.create_project (exit_code=0) for project taavitestwikitech in eqiad1 [14:01:32] taavi@cloudcumin1001: Unknown project "taavitestwikitech" [14:18:20] (03update) 10dcaro: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) [14:22:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:24:23] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9863937 (10dcaro) > Each tool domain (tools, toolsbeta) will have a service user with inherited project-wide domain membership: tools-admin and toolsbeta-admin... [14:24:32] (03PS1) 10Muehlenhoff: Remove obsolete virt-star stub secret [labs/private] - 10https://gerrit.wikimedia.org/r/1039227 [14:24:37] (03update) 10sstefanova: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) (owner: 10dcaro) [14:24:42] (03approved) 10sstefanova: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) (owner: 10dcaro) [14:24:49] (03update) 10sstefanova: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) (owner: 10dcaro) [14:25:06] (03merge) 10dcaro: functional_tests: add webservice smoke tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/311 (https://phabricator.wikimedia.org/T357977) [14:29:12] (03approved) 10dcaro: homedir: use less filesystem checks [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/34 (https://phabricator.wikimedia.org/T366564) (owner: 10aborrero) [14:29:13] (03update) 10dcaro: homedir: use less filesystem checks [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/34 (https://phabricator.wikimedia.org/T366564) (owner: 10aborrero) [14:33:18] (03open) 10aborrero: maintain-kubeusers: MaintainKubeusersHang: adjust alert 'for' value [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/14 (https://phabricator.wikimedia.org/T366598) [14:33:33] 10Cloud-Services, 06serviceops, 06SRE, 13Patch-For-Review: Modernise memcached systemd unit / sync, and make it presentable - https://phabricator.wikimedia.org/T273950#9863991 (10MoritzMuehlenhoff) [14:43:17] (03update) 10aborrero: homedir: use less filesystem checks [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/34 (https://phabricator.wikimedia.org/T366564) [14:49:02] (03update) 10aborrero: homedir: use less filesystem checks [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/34 (https://phabricator.wikimedia.org/T366564) [14:49:34] (03merge) 10aborrero: homedir: use less filesystem checks [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/34 (https://phabricator.wikimedia.org/T366564) [14:52:01] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.142-20240605144945-15057389 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/312 (https://phabricator.wikimedia.org/T366564) [14:52:07] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.142-20240605144945-15057389 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/312 (https://phabricator.wikimedia.org/T366564) [14:54:42] (03open) 10sstefanova: Draft: openapi: refactor yaml [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/96 (https://phabricator.wikimedia.org/T366668) [15:13:26] (03update) 10raymond-ndibe: [jobs-api] add messages to all responses [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/85 (https://phabricator.wikimedia.org/T356974) [15:13:58] (03update) 10raymond-ndibe: [jobs-api] add messages to all responses [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/85 (https://phabricator.wikimedia.org/T356974) [15:26:57] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [15:27:07] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [15:29:27] 06cloud-services-team, 10Toolforge (Toolforge iteration 11), 13Patch-For-Review: toolforge: new maintain-kubeusers takes long time to loop over all the accounts to reconcile them - https://phabricator.wikimedia.org/T366564#9864168 (10aborrero) investigating error in the last patch in toolsbeta: ` starting... [15:31:27] 10Tool-schedule-deployment, 10Gerrit: Link to https://schedule-deployment.toolforge.org/backport/{change-id} from changes eligable for deployment in a backport window - https://phabricator.wikimedia.org/T366512#9864181 (10bd808) 05In progress→03Resolved Thanks for your help with this @hashar :) [15:32:54] 10Tool-schedule-deployment: Add placeholder text to form elements - https://phabricator.wikimedia.org/T366714 (10bd808) 03NEW [15:34:03] 10Tool-schedule-deployment: Use [[Template:Deploy]] when updating page - https://phabricator.wikimedia.org/T366715 (10bd808) 03NEW [15:39:21] FIRING: MaintainKubeusersDown: maintain-kubeusers is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersDown [15:44:21] RESOLVED: MaintainKubeusersDown: maintain-kubeusers is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersDown [15:47:51] FIRING: [2x] MaintainKubeusersDown: maintain-kubeusers is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersDown [15:58:56] (03open) 10aborrero: Revert "homedir: use less filesystem checks" [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/36 [15:59:13] (03approved) 10dcaro: Revert "homedir: use less filesystem checks" [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/36 (owner: 10aborrero) [16:00:36] (03merge) 10aborrero: Revert "homedir: use less filesystem checks" [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/36 [16:02:21] (03PS1) 10Majavah: vps: create_project: Allow using dashes again [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1039238 [16:02:48] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.142-20240605144945-15057389 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/312 (https://phabricator.wikimedia.org/T366564) [16:03:51] (03close) 10aborrero: maintain-kubeusers: bump to 0.0.142-20240605144945-15057389 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/312 (https://phabricator.wikimedia.org/T366564) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:04:03] (03reopen) 10aborrero: maintain-kubeusers: bump to 0.0.142-20240605144945-15057389 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/312 (https://phabricator.wikimedia.org/T366564) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:04:41] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [16:04:52] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [16:05:16] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [16:05:27] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [16:06:26] (03merge) 10aborrero: maintain-kubeusers: bump to 0.0.142-20240605144945-15057389 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/312 (https://phabricator.wikimedia.org/T366564) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:06:44] (03approved) 10dcaro: maintain-kubeusers: MaintainKubeusersHang: adjust alert 'for' value [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/14 (https://phabricator.wikimedia.org/T366598) (owner: 10aborrero) [16:06:45] (03update) 10dcaro: maintain-kubeusers: MaintainKubeusersHang: adjust alert 'for' value [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/14 (https://phabricator.wikimedia.org/T366598) (owner: 10aborrero) [16:07:51] RESOLVED: MaintainKubeusersDown: maintain-kubeusers is down - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersDown [16:08:21] FIRING: MaintainKubeusersHang: maintain-kubeusers last finished run is 28.63M minutes old - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersHang [16:08:41] (03update) 10raymond-ndibe: [jobs-api] move simple job validations to pydantic [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/89 (https://phabricator.wikimedia.org/T366209) [16:09:20] (03update) 10raymond-ndibe: [jobs-api] move simple job validations to pydantic [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/89 (https://phabricator.wikimedia.org/T366209) [16:13:21] RESOLVED: MaintainKubeusersHang: maintain-kubeusers last finished run is 28.63M minutes old - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainKubeusersDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DMaintainKubeusersHang [16:22:40] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9864459 (10Andrew) >>! In T358496#9863937, @dcaro wrote: >> Each tool domain (tools, toolsbeta) will have a service user with inherited project-wide domain mem... [16:40:25] (03open) 10samtar: Draft: Use {{Deploy}} template [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/2 (https://phabricator.wikimedia.org/T366715) [16:43:03] (03update) 10samtar: Use {{Deploy}} template [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/2 (https://phabricator.wikimedia.org/T366715) [16:43:50] 10Tool-schedule-deployment, 13Patch-For-Review: Use [[Template:Deploy]] when updating page - https://phabricator.wikimedia.org/T366715#9864635 (10TheresNoTime) a:03TheresNoTime nb. I guess be mindful of the template wrapping content: - `
` around each instance of the template.... [16:45:39] 10Tool-extloc, 10Release-Engineering-Team (Priority Backlog 📥): extloc: Save historical data - https://phabricator.wikimedia.org/T365664#9864663 (10brennen) [17:10:04] 06cloud-services-team, 10Cloud-VPS, 05Cloud-Services-Origin-Alert: [cloudcephosd1031] re-introduce sda to the OS raid - https://phabricator.wikimedia.org/T366240#9864861 (10dcaro) 05Open→03Resolved Host back up and running, as I did not remove the osds before the reboot and forced reimage (because it... [17:37:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:42:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:48:42] 10Tool-phab-ban, 07Technical-Debt: Deploy more recent PhabBanBot code from repository into production - https://phabricator.wikimedia.org/T366587#9865039 (10bd808) >>! In T366587#9863391, @Agabi10 wrote: > I just realized that the Deprecated Calls Log shows also `user.enable`, which I don't see even in old ver... [18:32:31] (03CR) 10BCornwall: [C:03+2] Move ncmonitor credentials to its own profile [labs/private] - 10https://gerrit.wikimedia.org/r/1037857 (owner: 10BCornwall) [18:32:33] (03CR) 10BCornwall: [V:03+2 C:03+2] Move ncmonitor credentials to its own profile [labs/private] - 10https://gerrit.wikimedia.org/r/1037857 (owner: 10BCornwall) [18:51:05] (03PS1) 10BCornwall: ncmonitor: Reformat credentials [labs/private] - 10https://gerrit.wikimedia.org/r/1039275 [18:51:17] (03CR) 10BCornwall: [V:03+2 C:03+2] ncmonitor: Reformat credentials [labs/private] - 10https://gerrit.wikimedia.org/r/1039275 (owner: 10BCornwall) [19:08:07] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#9865335 (10wiki_willy) Hi @dcaro - just following up on this. Can you provide the racking information for us, to start this install? Thanks, Willy [19:14:24] 10Tool-ducttape, 06Abstract Wikipedia team, 10function-evaluator, 10function-orchestrator, 03Abstract Wikipedia Fix-It tasks: Improve CI for Wikifunctions services to better test like reality - https://phabricator.wikimedia.org/T350284#9865422 (10Jdforrester-WMF) a:05Jdforrester-WMF→03None [19:29:06] (03PS1) 10BCornwall: ncmonitor: Move ssh key block to end of the file [labs/private] - 10https://gerrit.wikimedia.org/r/1039281 [19:29:21] (03CR) 10BCornwall: [V:03+2 C:03+2] ncmonitor: Move ssh key block to end of the file [labs/private] - 10https://gerrit.wikimedia.org/r/1039281 (owner: 10BCornwall) [19:38:44] (03open) 10raymond-ndibe: [jobs-api] create seperate api.py and move flask things there [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/91 (https://phabricator.wikimedia.org/T359804) [19:58:10] (03PS1) 10BCornwall: ncmonitor: Add test key to solve pcc error [labs/private] - 10https://gerrit.wikimedia.org/r/1039288 [19:59:07] (03CR) 10BCornwall: [V:03+2 C:03+2] ncmonitor: Add test key to solve pcc error [labs/private] - 10https://gerrit.wikimedia.org/r/1039288 (owner: 10BCornwall) [20:04:39] (03update) 10bd808: Use {{Deploy}} template [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/2 (https://phabricator.wikimedia.org/T366715) (owner: 10samtar) [20:06:31] (03merge) 10bd808: Use {{Deploy}} template [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/2 (https://phabricator.wikimedia.org/T366715) (owner: 10samtar) [20:06:41] (03PS1) 10BCornwall: ncmonitor: Fix namespacing of keys [labs/private] - 10https://gerrit.wikimedia.org/r/1039293 [20:07:19] (03PS2) 10BCornwall: ncmonitor: Fix namespacing of keys [labs/private] - 10https://gerrit.wikimedia.org/r/1039293 [20:07:48] (03CR) 10BCornwall: [V:03+2 C:03+2] ncmonitor: Fix namespacing of keys [labs/private] - 10https://gerrit.wikimedia.org/r/1039293 (owner: 10BCornwall) [20:10:25] 10Tool-schedule-deployment: Add placeholder text to form elements - https://phabricator.wikimedia.org/T366714#9865607 (10bd808) 05Open→03In progress p:05Triage→03Medium a:03bd808 [20:25:20] (03open) 10bd808: form: Add placeholder text for gerrit_id and name [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/3 (https://phabricator.wikimedia.org/T366714) [20:28:03] (03merge) 10bd808: form: Add placeholder text for gerrit_id and name [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/3 (https://phabricator.wikimedia.org/T366714) [20:32:21] 10Tool-schedule-deployment: Add placeholder text to form elements - https://phabricator.wikimedia.org/T366714#9865700 (10bd808) 05In progress→03Resolved [20:36:56] 10Tool-schedule-deployment: Use [[Template:Deploy]] when updating page - https://phabricator.wikimedia.org/T366715#9865706 (10bd808) 05Open→03Resolved I did double check before merging that this doesn't blow up the #jouncebot parser which relies on css selectors to find things. I also made a few tweaks t... [21:11:53] 10Tool-schedule-deployment: Keep irc_nick and name cookies longer than the browser session - https://phabricator.wikimedia.org/T366754 (10bd808) 03NEW [21:19:25] (03open) 10bd808: Keep irc_nick and name cookies for 32 days [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/4 (https://phabricator.wikimedia.org/T366754) [21:25:51] 10Tool-phab-ban, 07Technical-Debt: Deploy more recent PhabBanBot code from repository into production - https://phabricator.wikimedia.org/T366587#9865776 (10taavi) The [[ https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/b9a736d7d6264fdb89300e764e273b08dc356606/wmf-config/wikitech.php#159 | Wikitech... [21:27:10] (03merge) 10bd808: Keep irc_nick and name cookies for 32 days [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/4 (https://phabricator.wikimedia.org/T366754) [21:30:38] 10Tool-schedule-deployment: Prevent WIP changes from being scheduled for a backport - https://phabricator.wikimedia.org/T366757 (10bd808) 03NEW [21:30:39] 10Tool-schedule-deployment: Keep irc_nick and name cookies longer than the browser session - https://phabricator.wikimedia.org/T366754#9865789 (10bd808) 05Open→03In progress p:05Triage→03Medium a:03bd808 [21:31:51] 10Tool-schedule-deployment: Prevent WIP changes from being scheduled for a backport - https://phabricator.wikimedia.org/T366757#9865808 (10bd808) 05Open→03In progress p:05Triage→03Medium a:03bd808 [21:35:05] (03open) 10bd808: Block WIP changes from being scheduled [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/5 (https://phabricator.wikimedia.org/T366757) [21:37:26] (03update) 10bd808: Block WIP changes from being scheduled [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/5 (https://phabricator.wikimedia.org/T366757) [21:39:47] (03update) 10bd808: Block WIP changes from being scheduled [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/5 (https://phabricator.wikimedia.org/T366757) [21:41:09] (03merge) 10bd808: Block WIP changes from being scheduled [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/5 (https://phabricator.wikimedia.org/T366757) [21:42:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:45:10] 10Tool-schedule-deployment: Keep irc_nick and name cookies longer than the browser session - https://phabricator.wikimedia.org/T366754#9865852 (10bd808) 05In progress→03Resolved [21:46:18] 10Tool-schedule-deployment, 13Patch-For-Review: Prevent WIP changes from being scheduled for a backport - https://phabricator.wikimedia.org/T366757#9865855 (10bd808) 05In progress→03Resolved [21:52:31] 10Tool-phab-ban, 07Technical-Debt: Deploy more recent PhabBanBot code from repository into production - https://phabricator.wikimedia.org/T366587#9865892 (10bd808) >>! In T366587#9865776, @taavi wrote: > The [[ https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/b9a736d7d6264fdb89300e764e273b08dc35660... [21:54:48] 10wikitech.wikimedia.org, 07Technical-Debt: Update Phabricator BlockIpComplete hook to use "user.edit" Conduit API - https://phabricator.wikimedia.org/T366587#9865910 (10bd808) p:05Low→03Medium [22:40:52] 10Tools: Investigate AdvertiseDetector Outage - https://phabricator.wikimedia.org/T366690#9865997 (10Vahurzpu) The project directory on Toolforge doesn't appear to have any code: ` vahurzpu@tools-bastion-13:/data/project/advertisedetector$ find . ./.toolskube ./.toolskube/client.crt ./.toolskube/client.key ./re... [22:42:54] 10wikitech.wikimedia.org, 13Patch-For-Review, 07Technical-Debt: Update Phabricator BlockIpComplete hook to use "user.edit" Conduit API - https://phabricator.wikimedia.org/T366587#9865998 (10bd808) @PhabBanBot's profile even has (emphasis added) "Bot account for performing account bans via https://phab-ban.to... [22:54:55] 10wikitech.wikimedia.org, 13Patch-For-Review, 07Technical-Debt: Update Phabricator BlockIpComplete hook to use "user.edit" Conduit API - https://phabricator.wikimedia.org/T366587#9866006 (10bd808) 05Open→03In progress a:03bd808 [22:59:41] 10wikitech.wikimedia.org, 13Patch-For-Review, 07Technical-Debt: Update Phabricator BlockIpComplete hook to use "user.edit" Conduit API - https://phabricator.wikimedia.org/T366587#9866014 (10bd808) >>! In T366587#9865892, @bd808 wrote: > That also means I broke phab disable/enables when I rotated the API toke... [23:26:46] 10Tool-schedule-deployment: Leave a comment on the Gerrit change when it is scheduled for a backport - https://phabricator.wikimedia.org/T366763 (10bd808) 03NEW [23:35:38] 10Tool-schedule-deployment: Leave a comment on the Gerrit change when it is scheduled for a backport - https://phabricator.wikimedia.org/T366763#9866102 (10bd808) The magic for figuring out the anchor tag is to take the UTC time of the window and format it as `%Y%m%dT%H%M`. https://wikitech.wikimedia.org/wiki/Mo...