[00:09:28] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [00:14:28] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [01:13:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [01:23:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:36:51] 10Tool-schedule-deployment, 10Gerrit: Link to https://schedule-deployment.toolforge.org/backport/{change-id} from changes eligable for deployment in a backport window - https://phabricator.wikimedia.org/T366512#9857829 (10bd808) p:05Triage→03Medium a:03bd808 [02:42:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:57:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:00:00] 10Tool-toolwatch, 06Indic-MediaWiki-Developers: Sort tools based on tool Title - https://phabricator.wikimedia.org/T353579#9857882 (10Hks3333) @marrivs Yes, I'm still working on this [04:11:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:16:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:21:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:26:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:41:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:46:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:51:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:56:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:04:09] 10Toolforge: Need some help with php - https://phabricator.wikimedia.org/T366543#9857920 (10Wurgl) 05Open→03Resolved a:03Wurgl after about two hours, some output came. [06:40:16] (03update) 10sstefanova: [envvars-cli] remove unused code [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/41 (owner: 10raymond-ndibe) [06:41:09] (03open) 10sstefanova: cli: centralize context management [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/44 [06:41:46] (03update) 10sstefanova: cli: use prefixed endpoints [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/43 [06:50:47] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove obsolete stub cert [labs/private] - 10https://gerrit.wikimedia.org/r/1038380 (owner: 10Muehlenhoff) [07:26:51] (03merge) 10dcaro: nginx: increase read timeout for the logs endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/87 (https://phabricator.wikimedia.org/T359953) [07:29:37] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.305-20240604072701-b6dec32d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/300 (https://phabricator.wikimedia.org/T359953) [08:04:56] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [08:05:06] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [08:12:11] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [08:12:22] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [08:17:28] 06cloud-services-team, 10Toolforge (Toolforge iteration 10): toolforge: new maintain-kubeusers takes long time to loop over all the accounts to reconcile them - https://phabricator.wikimedia.org/T366564 (10aborrero) 03NEW [08:18:27] 10Toolforge (Toolforge iteration 10): [jobs-api, jobs-cli] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363346#9858336 (10Slst2020) a:03Slst2020 [08:19:21] 06cloud-services-team, 10Toolforge (Toolforge iteration 10): toolforge: new maintain-kubeusers takes long time to loop over all the accounts to reconcile them - https://phabricator.wikimedia.org/T366564#9858337 (10aborrero) [08:19:32] 10Toolforge (Toolforge iteration 10): [jobs-api, jobs-cli] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363346#9858334 (10Slst2020) 05Open→03In progress [08:20:28] (03approved) 10dcaro: jobs-api: bump to 0.0.305-20240604072701-b6dec32d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/300 (https://phabricator.wikimedia.org/T359953) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:20:32] (03merge) 10dcaro: jobs-api: bump to 0.0.305-20240604072701-b6dec32d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/300 (https://phabricator.wikimedia.org/T359953) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:20:32] (03update) 10dcaro: jobs-api: bump to 0.0.305-20240604072701-b6dec32d [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/300 (https://phabricator.wikimedia.org/T359953) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:35:47] 06cloud-services-team, 10Toolforge (Toolforge iteration 10): toolforge: new maintain-kubeusers takes long time to loop over all the accounts to reconcile them - https://phabricator.wikimedia.org/T366564#9858389 (10aborrero) [08:44:07] 10Cloud-Services, 06serviceops, 06SRE: Modernise memcached systemd unit / sync, and make it presentable - https://phabricator.wikimedia.org/T273950#9858437 (10jijiki) 05In progress→03Open The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikim... [08:45:29] 10Cloud-Services, 06serviceops, 06SRE: Modernise memcached systemd unit / sync, and make it presentable - https://phabricator.wikimedia.org/T273950#9858443 (10jijiki) [08:51:52] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Data-Services, 06Data-Persistence: Upgrade clouddb* hosts to Bookworm - https://phabricator.wikimedia.org/T365424#9858459 (10fnegri) p:05Triage→03Medium [08:54:05] 10Cloud-VPS (Project-requests), 10Performance-Device-Lab, 06Quality-and-Test-Engineering-Team, 10Synthetic-Performance-Testing: Request creation of web performance test VPS project - https://phabricator.wikimedia.org/T366569 (10Peter) 03NEW [09:01:43] 06cloud-services-team, 10Toolforge: [toolforge] [redis] Prometheus exporter logging errors - https://phabricator.wikimedia.org/T366471#9858510 (10fnegri) CLIENT is indeed hashed. That doesn't explain the "error: kex_exchange_identification" though. [09:03:12] 06cloud-services-team, 10Toolforge (Toolforge iteration 10): toolforge: new maintain-kubeusers takes long time to loop over all the accounts to reconcile them - https://phabricator.wikimedia.org/T366564#9858513 (10aborrero) [09:03:34] 10Cloud-Services, 06serviceops, 06SRE: Modernise memcached systemd unit / sync, and make it presentable - https://phabricator.wikimedia.org/T273950#9858514 (10jijiki) [09:04:54] 06cloud-services-team, 10Toolforge: [toolforge] [redis] Prometheus exporter logging errors - https://phabricator.wikimedia.org/T366471#9858515 (10dcaro) that is sshd no? (not prometheus exporter) [09:05:48] 10Cloud-Services, 06serviceops, 06SRE: Modernise memcached systemd unit / sync, and make it presentable - https://phabricator.wikimedia.org/T273950#9858516 (10jijiki) [09:06:51] (03open) 10aborrero: maintain-kubeusers: handle livenessprobe checks more often [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/28 (https://phabricator.wikimedia.org/T366564) [09:08:03] (03approved) 10dcaro: maintain-kubeusers: handle livenessprobe checks more often [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/28 (https://phabricator.wikimedia.org/T366564) (owner: 10aborrero) [09:08:05] (03update) 10dcaro: maintain-kubeusers: handle livenessprobe checks more often [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/28 (https://phabricator.wikimedia.org/T366564) (owner: 10aborrero) [09:08:34] 06cloud-services-team, 10Toolforge (Toolforge iteration 10), 13Patch-For-Review: toolforge: new maintain-kubeusers takes long time to loop over all the accounts to reconcile them - https://phabricator.wikimedia.org/T366564#9858524 (10aborrero) 05Open→03In progress p:05Triage→03High [09:11:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:13:36] (03merge) 10sstefanova: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/93 [09:15:00] 06cloud-services-team, 10Toolforge: [toolforge] [redis] Prometheus exporter logging errors - https://phabricator.wikimedia.org/T366471#9858545 (10fnegri) It is, but the connection comes from the Prometheus hosts, and at the same time of the other error, so it looks somewhat related? [09:15:45] 06cloud-services-team, 10Toolforge: [toolforge] [redis] Prometheus exporter logging errors - https://phabricator.wikimedia.org/T366471#9858558 (10fnegri) To clarify, the 4 lines in the description are always logged together. [09:18:15] 10Cloud-Services, 06serviceops, 06SRE: Modernise memcached systemd unit / sync, and make it presentable - https://phabricator.wikimedia.org/T273950#9858577 (10jijiki) [09:19:04] 06cloud-services-team, 10Toolforge: [toolforge] [redis] Prometheus exporter logging errors - https://phabricator.wikimedia.org/T366471#9858579 (10dcaro) Maybe an "is ssh open?" probe? (and prometheus doing all the checks together) [09:20:02] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-api: bump to 0.0.154-20240604091345-fa0904fb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/301 [09:20:45] 06cloud-services-team, 10Toolforge: [toolforge] [redis] Prometheus exporter logging errors - https://phabricator.wikimedia.org/T366471#9858580 (10fnegri) That sounds possible, I will double check later [09:21:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:22:24] (03merge) 10aborrero: maintain-kubeusers: handle livenessprobe checks more often [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/28 (https://phabricator.wikimedia.org/T366564) [09:24:49] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.136-20240604092234-e8d7cdd4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/302 (https://phabricator.wikimedia.org/T366564) [09:25:37] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [09:25:47] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [09:26:40] 06cloud-services-team, 10Toolforge: [toolforge] [redis] Prometheus exporter logging errors - https://phabricator.wikimedia.org/T366471#9858590 (10fnegri) For the CLIENT command, there's a `REDIS_EXPORTER_SET_CLIENT_NAME` [[ https://github.com/oliver006/redis_exporter?tab=readme-ov-file#command-line-flags | opt... [09:26:46] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [09:26:57] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [09:27:53] (03merge) 10aborrero: maintain-kubeusers: bump to 0.0.136-20240604092234-e8d7cdd4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/302 (https://phabricator.wikimedia.org/T366564) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:06:56] FIRING: SystemdUnitDown: The service unit networking.service is in failed status on host cloudlb1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudlb1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [10:13:13] (03open) 10aborrero: kubecerts: don't check if file exists in needs_create() [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/29 (https://phabricator.wikimedia.org/T366564) [10:16:40] 10wikitech.wikimedia.org, 06DBA, 10MediaModeration, 06Trust and Safety Product Team, 07Wikimedia-production-error: extension1 database for wikitech is always overloaded - https://phabricator.wikimedia.org/T366574#9858703 (10taavi) This is most likely the special hosts that run Wikitech not having the nec... [10:16:56] FIRING: [2x] SystemdUnitDown: The service unit networking.service is in failed status on host cloudlb1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [10:17:05] 10wikitech.wikimedia.org, 06DBA, 10MediaModeration, 06Trust and Safety Product Team, 07Wikimedia-production-error: extension1 database for wikitech is always overloaded - https://phabricator.wikimedia.org/T366574#9858708 (10Ladsgroup) I'm sure this is a bug. If extension1 was actually overloaded, all wik... [10:17:41] 10wikitech.wikimedia.org, 06DBA, 10MediaModeration, 06Trust and Safety Product Team, 07Wikimedia-production-error: extension1 database for wikitech is always overloaded - https://phabricator.wikimedia.org/T366574#9858734 (10Dreamy_Jazz) Thanks both. [10:18:34] (03approved) 10dcaro: kubecerts: don't check if file exists in needs_create() [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/29 (https://phabricator.wikimedia.org/T366564) (owner: 10aborrero) [10:18:35] (03update) 10dcaro: kubecerts: don't check if file exists in needs_create() [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/29 (https://phabricator.wikimedia.org/T366564) (owner: 10aborrero) [10:27:26] 06cloud-services-team, 10Toolforge (Toolforge iteration 10), 13Patch-For-Review: toolforge: new maintain-kubeusers takes long time to loop over all the accounts to reconcile them - https://phabricator.wikimedia.org/T366564#9858763 (10aborrero) [10:28:56] (03merge) 10aborrero: kubecerts: don't check if file exists in needs_create() [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/29 (https://phabricator.wikimedia.org/T366564) [10:30:47] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.137-20240604102906-d1b2d380 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/303 (https://phabricator.wikimedia.org/T366564) [10:32:16] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [10:32:27] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [10:32:45] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [10:32:55] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [10:34:17] (03merge) 10aborrero: maintain-kubeusers: bump to 0.0.137-20240604102906-d1b2d380 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/303 (https://phabricator.wikimedia.org/T366564) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:41:50] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:43:05] 10Cloud-VPS (Project-requests), 10Performance-Device-Lab, 06Quality-and-Test-Engineering-Team, 10Synthetic-Performance-Testing: Request creation of web performance test VPS project - https://phabricator.wikimedia.org/T366569#9858807 (10dcaro) +1 For the reports back, if you use https you can use the regul... [10:43:15] 10Toolforge (Toolforge iteration 10): [builds-api] Fix issue with log streaming timing out - https://phabricator.wikimedia.org/T366147#9858813 (10dcaro) a:03Slst2020 [10:43:49] 10Toolforge (Toolforge iteration 10): [builds-api] Fix issue with log streaming timing out - https://phabricator.wikimedia.org/T366147#9858815 (10dcaro) 05Open→03Resolved [10:48:49] 06cloud-services-team, 10Cloud-VPS, 10Puppet (Puppet 7.0): cloud-vps puppetservers filling up / with puppetserver reports - https://phabricator.wikimedia.org/T366357#9858847 (10taavi) @Andrew Doesn't the PCC updater rely on the facts report directory? Also there should be a systemd timer cleaning up old... [10:50:23] 06cloud-services-team, 10Toolforge (Toolforge iteration 10): toolforge: Refresh certs that are not controlled by kubeadm (mid 2024 edition) - https://phabricator.wikimedia.org/T309782#9858871 (10dcaro) a:03dcaro [10:50:24] 10Toolforge (Toolforge iteration 10): [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#9858867 (10dcaro) 05Open→03In progress [10:51:20] 06cloud-services-team, 10Toolforge (Toolforge iteration 10): [infra,k8s,monitoring] Add an alert to warn when the prometheus k8s cert is about to expire - https://phabricator.wikimedia.org/T366579 (10dcaro) 03NEW [10:51:22] 06cloud-services-team, 10Toolforge (Toolforge iteration 10): [infra,k8s,monitoring] Add an alert to warn when the prometheus k8s cert is about to expire - https://phabricator.wikimedia.org/T366579#9858886 (10dcaro) p:05Triage→03High [10:51:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:51:56] RESOLVED: SystemdUnitDown: The service unit networking.service is in failed status on host cloudlb1001. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/SystemdUnitDown - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudlb1001 - https://alerts.wikimedia.org/?q=alertname%3DSystemdUnitDown [10:53:22] 06cloud-services-team, 10Cloud-VPS, 10Puppet (Puppet 7.0): cloud-vps puppetservers filling up / with puppetserver reports - https://phabricator.wikimedia.org/T366357#9858896 (10dcaro) >>! In T366357#9858847, @taavi wrote: > @Andrew Doesn't the PCC updater rely on the facts report directory? Also there sh... [10:54:26] 06cloud-services-team, 10Cloud-VPS, 10Puppet (Puppet 7.0): cloud-vps puppetservers filling up / with puppetserver reports - https://phabricator.wikimedia.org/T366357#9858898 (10taavi) And https://gerrit.wikimedia.org/r/c/operations/puppet/+/1037812 was merged yesterday disabling the reports entirely? [10:57:08] (03open) 10aborrero: kubeconfig: store some state about kubeconfig in the configmap to save NFS hits [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/30 (https://phabricator.wikimedia.org/T366564) [10:58:03] 06cloud-services-team, 10Cloud-VPS, 10Puppet (Puppet 7.0): cloud-vps puppetservers filling up / with puppetserver reports - https://phabricator.wikimedia.org/T366357#9858916 (10dcaro) >>! In T366357#9858898, @taavi wrote: > And https://gerrit.wikimedia.org/r/c/operations/puppet/+/1037812 was merged yeste... [11:01:44] 10wikitech.wikimedia.org, 06DBA, 10MediaModeration, 06Trust and Safety Product Team, 07Wikimedia-production-error: extension1 database for wikitech is always overloaded - https://phabricator.wikimedia.org/T366574#9858926 (10Ladsgroup) Can we wait until wikitech host is moved to production network? That'd... [11:04:04] 10Data-Services, 06Data-Persistence, 10Data-Platform-SRE (2024.05.27 - 2024.06.16): Upgrade clouddb1021 to bookworm - https://phabricator.wikimedia.org/T365450#9858950 (10BTullis) a:03BTullis [11:04:16] (03update) 10aborrero: kubeconfig: store some state about kubeconfig in the configmap to save NFS hits [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/30 (https://phabricator.wikimedia.org/T366564) [11:07:51] 10wikitech.wikimedia.org, 06DBA, 10MediaModeration, 06Trust and Safety Product Team, 07Wikimedia-production-error: extension1 database for wikitech is always overloaded - https://phabricator.wikimedia.org/T366574#9858971 (10Dreamy_Jazz) That should be okay as long as this is planned within the next few m... [11:20:29] (03PS1) 10Muehlenhoff: Remove obsolete Icinga stub secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1038748 [11:27:55] (03update) 10aborrero: kubeconfig: store some state about kubeconfig in the configmap to save NFS hits [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/30 (https://phabricator.wikimedia.org/T366564) [11:35:57] 10Tools: supercount @ toolforge is timing out - https://phabricator.wikimedia.org/T366584 (10Titore) 03NEW [11:56:01] (03open) 10aborrero: tests: properly cleanup k8s namespaces [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/31 [11:59:38] (03approved) 10dcaro: kubeconfig: store some state about kubeconfig in the configmap to save NFS hits [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/30 (https://phabricator.wikimedia.org/T366564) (owner: 10aborrero) [11:59:41] (03update) 10dcaro: kubeconfig: store some state about kubeconfig in the configmap to save NFS hits [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/30 (https://phabricator.wikimedia.org/T366564) (owner: 10aborrero) [12:01:03] (03open) 10aborrero: maintain_kubeusers: don't sleep 1 minute between runs [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/32 [12:01:37] (03merge) 10aborrero: kubeconfig: store some state about kubeconfig in the configmap to save NFS hits [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/30 (https://phabricator.wikimedia.org/T366564) [12:03:35] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.138-20240604120147-29f8d0f2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/304 (https://phabricator.wikimedia.org/T366564) [12:04:22] (03update) 10dcaro: maintain_kubeusers: don't sleep 1 minute between runs [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/32 (owner: 10aborrero) [12:04:58] 10Cloud-VPS: Frequent radosgw 500 errors with OpenTofu - https://phabricator.wikimedia.org/T360626#9859091 (10taavi) [12:05:37] 10Cloud-VPS, 10PAWS: periodic error from tofu when state in object storage - https://phabricator.wikimedia.org/T366124#9859089 (10taavi) →14Duplicate dup:03T360626 [12:06:16] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [12:06:26] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [12:06:58] (03approved) 10dcaro: maintain_kubeusers: don't sleep 1 minute between runs [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/32 (owner: 10aborrero) [12:07:01] (03update) 10dcaro: maintain_kubeusers: don't sleep 1 minute between runs [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/32 (owner: 10aborrero) [12:08:42] 10Toolforge (Toolforge iteration 10): [jobs-api,jobs-cli] Support services in jobs - https://phabricator.wikimedia.org/T348758#9859097 (10dcaro) What's missing here, the docs? [12:08:52] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [12:09:04] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [12:09:36] (03update) 10dcaro: [jobs-api] add messages to all responses [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/85 (https://phabricator.wikimedia.org/T356974) (owner: 10raymond-ndibe) [12:10:40] (03merge) 10aborrero: maintain-kubeusers: bump to 0.0.138-20240604120147-29f8d0f2 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/304 (https://phabricator.wikimedia.org/T366564) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [12:11:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:14:00] (03update) 10aborrero: tests: properly cleanup k8s namespaces [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/31 [12:14:46] 10Tool-phab-ban, 07Technical-Debt: Port PhabBanBot code from calling "user.disable"/"user.enable" to "user.edit" Conduit API - https://phabricator.wikimedia.org/T366587 (10Aklapper) 03NEW p:05Triage→03Low [12:15:56] (03merge) 10aborrero: maintain_kubeusers: don't sleep 1 minute between runs [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/32 [12:16:14] (03update) 10aborrero: tests: properly cleanup k8s namespaces [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/31 [12:16:20] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T366588 (10Redalert2fan) 03NEW [12:17:34] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [12:17:45] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [12:18:21] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.139-20240604121608-84e492a3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/305 [12:18:36] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [12:18:46] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [12:18:50] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [12:18:58] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [12:19:29] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [12:19:39] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [12:20:48] (03merge) 10aborrero: maintain-kubeusers: bump to 0.0.139-20240604121608-84e492a3 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/305 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [12:21:18] (03update) 10sstefanova: builds-api: bump to 0.0.154-20240604091345-fa0904fb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/301 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [12:21:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:21:44] (03CR) 10Filippo Giunchedi: [C:03+1] Remove obsolete Icinga stub secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1038748 (owner: 10Muehlenhoff) [12:22:19] (03open) 10dcaro: functional-tests: add smoke tests for envvars [repos/cloud/toolforge/toolforge-deploy] (add_builds_functional_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/306 (https://phabricator.wikimedia.org/T357977) [12:22:20] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove obsolete Icinga stub secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1038748 (owner: 10Muehlenhoff) [12:22:35] (03open) 10dcaro: functional-tests: add smoke build tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/307 (https://phabricator.wikimedia.org/T357977) [12:23:08] 10Tool-phab-ban, 07Technical-Debt: Port PhabBanBot code from calling "user.disable"/"user.enable" to "user.edit" Conduit API - https://phabricator.wikimedia.org/T366587#9859147 (10Agabi10) The code has already been changed based on [[https://gitlab.wikimedia.org/toolforge-repos/phab-ban/-/commit/7956e6c47d276e... [12:24:32] 10Tool-phab-ban, 07Technical-Debt: Port PhabBanBot code from calling "user.disable"/"user.enable" to "user.edit" Conduit API - https://phabricator.wikimedia.org/T366587#9859151 (10Aklapper) Heh, thanks. Yes, that is what I also just realized. :D The code actually already uses `user.edit` in https://gitlab.wi... [12:25:44] 10Tool-phab-ban, 07Technical-Debt: Deploy more recent PhabBanBot code from repository into production - https://phabricator.wikimedia.org/T366587#9859152 (10Aklapper) [12:26:13] 10Tool-phab-ban, 07Technical-Debt: Deploy more recent PhabBanBot code from repository into production - https://phabricator.wikimedia.org/T366587#9859155 (10Aklapper) @bd808: Would you have an idea how/who to achieve that, maybe? :) [12:29:55] 06cloud-services-team, 10Data-Services, 06DBA: Prepare and check storage layer for dtpwiki - https://phabricator.wikimedia.org/T365229#9859166 (10taavi) 05Open→03Resolved a:03taavi [12:30:56] 10Tool-gitlab-account-approval: Add error logging - https://phabricator.wikimedia.org/T361079#9859186 (10Aklapper) On a related node, `@glaab` sometimes throws `ERR-INVALID-PARAMETER` when calling Phabricator's `user.ldapquery` or `user.mediawikiquery` Conduit APIs according to https://phabricator.wikimedia.org/... [12:34:21] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T366588#9859193 (10Curb_Safe_Charmer) a:03Curb_Safe_Charmer [12:38:56] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T366588#9859207 (10Curb_Safe_Charmer) Ran ` become refill-api webservice restart ` No difference. [12:44:39] (03open) 10dcaro: toolforge_deploy_mr:add option to restore a component/cli [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/134 [12:45:26] (03update) 10dcaro: functional-tests: add smoke tests for envvars [repos/cloud/toolforge/toolforge-deploy] (add_builds_functional_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/306 (https://phabricator.wikimedia.org/T357977) [12:45:44] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T366588#9859242 (10Curb_Safe_Charmer) Ran ` ./restart.sh kubectl get pods ` No difference. [12:45:53] (03update) 10dcaro: functional-tests: add smoke tests for envvars [repos/cloud/toolforge/toolforge-deploy] (add_builds_functional_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/306 (https://phabricator.wikimedia.org/T357977) [12:46:16] (03update) 10dcaro: functional-tests: add smoke build tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/307 (https://phabricator.wikimedia.org/T357977) [12:46:26] (03update) 10dcaro: functional-tests: add smoke build tests [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/307 (https://phabricator.wikimedia.org/T357977) [12:46:44] (03update) 10dcaro: functional-tests: add smoke tests for envvars [repos/cloud/toolforge/toolforge-deploy] (add_builds_functional_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/306 (https://phabricator.wikimedia.org/T357977) [12:47:22] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [12:47:33] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [12:53:38] (03update) 10dcaro: functional-tests: add smoke tests for envvars [repos/cloud/toolforge/toolforge-deploy] (add_builds_functional_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/306 (https://phabricator.wikimedia.org/T357977) [12:56:04] (03update) 10dcaro: functional-tests: add smoke tests for envvars [repos/cloud/toolforge/toolforge-deploy] (add_builds_functional_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/306 (https://phabricator.wikimedia.org/T357977) [12:56:46] (03approved) 10sstefanova: builds-api: bump to 0.0.154-20240604091345-fa0904fb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/301 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [12:56:49] (03merge) 10sstefanova: builds-api: bump to 0.0.154-20240604091345-fa0904fb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/301 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [12:57:25] (03update) 10dcaro: functional-tests: add smoke tests for envvars [repos/cloud/toolforge/toolforge-deploy] (add_builds_functional_tests) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/306 (https://phabricator.wikimedia.org/T357977) [12:58:07] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 10): Intermittent redis connection timeouts in Toolforge - https://phabricator.wikimedia.org/T318479#9859330 (10fnegri) I looked again at the errors mentioned in the description of this task, related to `spi-tools`. They are still happ... [12:58:35] (03PS1) 10Muehlenhoff: Remove obsolete swift stub secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1038769 [13:04:00] (03CR) 10MVernon: [C:03+1] "Thanks for doing the tidy-up, this looks good to me." [labs/private] - 10https://gerrit.wikimedia.org/r/1038769 (owner: 10Muehlenhoff) [13:05:46] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T366588#9859377 (10Curb_Safe_Charmer) a:05Curb_Safe_Charmer→03TheresNoTime Assigning to Sammy as not sure what else to try. [13:09:14] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove obsolete swift stub secrets [labs/private] - 10https://gerrit.wikimedia.org/r/1038769 (owner: 10Muehlenhoff) [13:25:04] 06cloud-services-team, 10Toolforge (Toolforge iteration 10), 13Patch-For-Review: [maintain-kubeusers,infra,k8s]: introduce some logic to backfill maintain-kubeuser resources (like per-tool kyverno policies) - https://phabricator.wikimedia.org/T364312#9859497 (10aborrero) 05In progress→03Resolved this... [13:26:04] (03update) 10aborrero: tests: properly cleanup k8s namespaces [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/31 [13:26:17] 06cloud-services-team, 10Toolforge (Toolforge iteration 10): [infra,k8s,monitoring] Add an alert to warn when the prometheus k8s cert is about to expire - https://phabricator.wikimedia.org/T366579#9859520 (10dcaro) 05Open→03In progress [13:31:36] 10Toolforge (Toolforge iteration 10): [maintain-kubeusers,infra] deal with tools with invalid names - https://phabricator.wikimedia.org/T366477#9859533 (10dcaro) p:05Triage→03Medium a:03taavi [13:33:21] 10Toolforge (Toolforge iteration 10): [jobs-api,builds-api,envvars-api] consolidate api paths - https://phabricator.wikimedia.org/T365014#9859566 (10Slst2020) a:03Slst2020 [13:33:42] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 10): [toolforge] webservice logs crashes with some unicode chars - https://phabricator.wikimedia.org/T364609#9859567 (10dcaro) p:05High→03Medium [13:35:03] 10Toolforge (Toolforge iteration 10): [builds-api, envvars-api] add oapi-codegen installation to makefile - https://phabricator.wikimedia.org/T362290#9859569 (10Raymond_Ndibe) 05In progress→03Resolved [13:35:03] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge: [toolforge] webservice logs crashes with some unicode chars - https://phabricator.wikimedia.org/T364609#9859570 (10dcaro) [13:35:09] 06cloud-services-team, 10Toolforge: lima-kilo: container image caching - https://phabricator.wikimedia.org/T362967#9859572 (10dcaro) [13:36:14] 10Toolforge (Toolforge iteration 10): [maintain-kubeusers] Increment default services quota - https://phabricator.wikimedia.org/T362520#9859574 (10dcaro) 05Open→03Resolved [13:36:20] 06cloud-services-team, 10Toolforge: lima-kilo: container image caching - https://phabricator.wikimedia.org/T362967#9859581 (10fnegri) 05In progress→03Open [13:38:04] 06cloud-services-team, 10Toolforge (Toolforge iteration 10): maintain-kubeusers: metrics, monitoring and alerting - https://phabricator.wikimedia.org/T366598 (10aborrero) 03NEW [13:39:40] 10Toolforge: Upgrade golang buildpack to 1.22 - https://phabricator.wikimedia.org/T363854#9859610 (10taavi) [13:39:41] 10Toolforge: Can't pip install mysqlclient on Toolforge - https://phabricator.wikimedia.org/T349341#9859611 (10taavi) [13:39:45] 10Toolforge: Support jdk21 on toolforge - https://phabricator.wikimedia.org/T346477#9859612 (10taavi) [13:39:47] 10Toolforge: python3.9 image (and maybe others?) contains `pip` in default search path - https://phabricator.wikimedia.org/T337145#9859613 (10taavi) [13:39:49] 10Toolforge: Install licensecheck on Toolforge login or dev hosts - https://phabricator.wikimedia.org/T325958#9859614 (10taavi) [13:39:50] 10Toolforge: Install msbuild - https://phabricator.wikimedia.org/T311454#9859615 (10taavi) [13:39:53] 06cloud-services-team, 10Toolforge: Missing Perl packages on dev.toolforge.org for anomiebot workflows - https://phabricator.wikimedia.org/T360488#9859616 (10taavi) [13:39:56] 06cloud-services-team, 10Toolforge: Consider adding `kubectl`, `webservice`, and `toolforge` binaries to shell container images - https://phabricator.wikimedia.org/T360818#9859617 (10taavi) [13:40:04] 10Toolforge, 13Patch-For-Review: Provide a Redis container for use within a tool's namespace - https://phabricator.wikimedia.org/T360378#9859618 (10taavi) [13:40:08] 10Toolforge: [builds-builder] Request for supporting Deno on Toolforge - https://phabricator.wikimedia.org/T253470#9859620 (10taavi) [13:41:44] 10Toolforge, 13Patch-For-Review: Provide a Redis container for use within a tool's namespace - https://phabricator.wikimedia.org/T360378#9859637 (10dcaro) Now that we have services in jobs (docs coming soon), @bd808 do you want to continue your work on this or do you want for someone else to finish it up? [13:42:11] 10Toolforge: Upgrade golang buildpack to 1.22 - https://phabricator.wikimedia.org/T363854#9859639 (10dcaro) p:05Triage→03Low [13:42:48] 10Toolforge: [toolforge] [redis] Improve Puppet config - https://phabricator.wikimedia.org/T366365#9859652 (10fnegri) p:05Triage→03Medium [13:47:53] 10Toolforge: python3.9 image (and maybe others?) contains `pip` in default search path - https://phabricator.wikimedia.org/T337145#9859696 (10dcaro) p:05Triage→03Low [13:50:29] 06cloud-services-team, 10Toolforge: Consider adding `kubectl`, `webservice`, and `toolforge` binaries to shell container images - https://phabricator.wikimedia.org/T360818#9859700 (10taavi) At least for the main (non-shell) containers I'd prefer not to include the `toolforge`/`webservice` since I don't want to... [13:50:53] 06cloud-services-team, 10Toolforge (Toolforge iteration 10): maintain-kubeusers: metrics, monitoring and alerting - https://phabricator.wikimedia.org/T366598#9859705 (10dcaro) p:05Triage→03High [13:51:33] 10Toolforge (Toolforge iteration 10): Toolforge Aptfile not producing working copy of `ffmpeg` - https://phabricator.wikimedia.org/T365633#9859701 (10dcaro) p:05Triage→03Medium a:03dcaro [13:51:35] 10Toolforge (Toolforge iteration 10): Toolforge Aptfile not producing working copy of `ffmpeg` - https://phabricator.wikimedia.org/T365633#9859704 (10dcaro) [13:54:16] 10Toolforge: Deleting an envvar breaks ReplicaSet driven automatic restarts of a Pod (CreateContainerConfigError) - https://phabricator.wikimedia.org/T365048#9859728 (10dcaro) p:05Triage→03Low [13:58:14] (03open) 10dcaro: add envvars admission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/135 [13:58:40] (03update) 10dcaro: add envvars admission [repos/cloud/toolforge/lima-kilo] (deploy_add_restore) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/135 [14:00:21] 10Toolforge (Toolforge iteration 10): [maintain-kubeusers,infra] deal with tools with invalid names - https://phabricator.wikimedia.org/T366477#9859768 (10taavi) I think we can just archive those tools, as after the Grid Engine was shut down there's not much they could be used for. [14:01:09] 10Toolforge (Toolforge iteration 10): Toolforge Aptfile not producing working copy of `ffmpeg` - https://phabricator.wikimedia.org/T365633#9859767 (10aborrero) In debian systems the `libpulsecommon.so` file is usually distributed with the `libpulse0` .deb package: ` ± dpkg -L libpulse0 /. /etc /etc/pulse /etc/p... [14:01:15] 10Toolforge, 07Upstream: Python buildpack does not detect requirements from pyproject.toml - https://phabricator.wikimedia.org/T353762#9859777 (10fnegri) p:05Triage→03Low [14:01:26] 10Toolforge, 07Documentation: [docs] update READMEs - https://phabricator.wikimedia.org/T362390#9859764 (10fnegri) p:05Triage→03Medium [14:05:51] 10Toolforge (Toolforge iteration 10): Toolforge Aptfile not producing working copy of `ffmpeg` - https://phabricator.wikimedia.org/T365633#9859813 (10taavi) No, it should be pulled as a dependency of a dependency. I tested it on a fresh sid container and it works as expected: ` $ sudo apt install ffmpeg $ aptitu... [14:08:36] 06cloud-services-team, 10Toolforge (Toolforge iteration 10): toolforge: new maintain-kubeusers takes long time to loop over all the accounts to reconcile them - https://phabricator.wikimedia.org/T366564#9859821 (10aborrero) with the latest changes we are down to ~3 minutes per noop loop. [14:10:14] (03update) 10aborrero: tests: properly cleanup k8s namespaces [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/31 [14:22:16] (03open) 10aborrero: kubeconfig: remove resource changes in the update routine [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/33 [14:23:56] (03approved) 10aborrero: tests: properly cleanup k8s namespaces [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/31 [14:24:04] (03merge) 10aborrero: tests: properly cleanup k8s namespaces [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/31 [14:26:01] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.140-20240604142418-04781a71 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/308 [14:31:17] (03update) 10aborrero: kubeconfig: remove resource changes in the update routine [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/33 [14:38:34] (03update) 10aborrero: kubeconfig: remove resource changes in the update routine [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/33 [14:41:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:46:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:51:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:52:44] (03update) 10dcaro: add envvars admission [repos/cloud/toolforge/lima-kilo] (deploy_add_restore) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/135 [14:52:53] 10Tool-schedule-deployment, 10Gerrit, 13Patch-For-Review: Link to https://schedule-deployment.toolforge.org/backport/{change-id} from changes eligable for deployment in a backport window - https://phabricator.wikimedia.org/T366512#9860023 (10bd808) 05Open→03In progress [14:53:02] (03update) 10dcaro: add envvars admission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/135 [14:55:55] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 11), 05Goal: [infra] Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#9860034 (10dcaro) [14:55:58] 10Toolforge (Toolforge iteration 11): [toolforge] simplify calling the different toolforge apis from within the containers - https://phabricator.wikimedia.org/T356377#9860038 (10dcaro) [14:56:38] 10Toolforge (Toolforge iteration 11), 07Upstream: [builds-builder] golang based images get infinite nested loops for procfile entries - https://phabricator.wikimedia.org/T363417#9860040 (10dcaro) [14:56:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:56:45] 10Toolforge (Toolforge iteration 11), 13Patch-For-Review, 07Upstream: [maintain-harbor] Manage project quotas via maintain-harbor - https://phabricator.wikimedia.org/T352417#9860042 (10dcaro) [14:57:06] 10Toolforge (Toolforge iteration 11), 07Upstream: [builds-builder,jobs-api,upstream] Calling nontrivial Procfile commands with arguments results in confusing error (“no such file or directory”) - https://phabricator.wikimedia.org/T356016#9860036 (10dcaro) [14:57:07] 10Toolforge (Toolforge iteration 11): [jobs-api,jobs-cli] Support services in jobs - https://phabricator.wikimedia.org/T348758#9860044 (10dcaro) [14:57:14] 06cloud-services-team, 10Toolforge (Toolforge iteration 11): [infra,k8s,monitoring] Add an alert to warn when the prometheus k8s cert is about to expire - https://phabricator.wikimedia.org/T366579#9860046 (10dcaro) [14:57:18] 06cloud-services-team, 10Toolforge (Toolforge iteration 11): toolforge: new maintain-kubeusers takes long time to loop over all the accounts to reconcile them - https://phabricator.wikimedia.org/T366564#9860048 (10dcaro) [14:57:31] 06cloud-services-team, 10Toolforge (Toolforge iteration 11), 13Patch-For-Review: Toolforge: Replace all bastion with grid-less bookworm based bastion hosts - https://phabricator.wikimedia.org/T314665#9860032 (10dcaro) [14:58:18] 10Toolforge (Toolforge iteration 11), 13Patch-For-Review: [envvars-api, envvars-cli] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363809#9860030 (10dcaro) [14:58:34] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 11): Intermittent redis connection timeouts in Toolforge - https://phabricator.wikimedia.org/T318479#9860052 (10dcaro) [14:58:45] 06cloud-services-team, 10Toolforge (Toolforge iteration 11): toolforge: Refresh certs that are not controlled by kubeadm (mid 2024 edition) - https://phabricator.wikimedia.org/T309782#9860050 (10dcaro) [14:59:00] 10Toolforge (Toolforge iteration 11): [toolforge] Investigate authentication - https://phabricator.wikimedia.org/T363983#9860058 (10dcaro) [14:59:18] 10Toolforge (Toolforge iteration 11), 13Patch-For-Review: [jobs-api] Split the API, business, and k8s models - https://phabricator.wikimedia.org/T359808#9860062 (10dcaro) [14:59:18] 10Toolforge (Toolforge iteration 11): [toolforge-cli,jobs-cli,builds-cli,envvars-cli] Explore OpenAPI SDK tooling for client consolidation - https://phabricator.wikimedia.org/T356261#9860056 (10dcaro) [15:00:14] 10Toolforge (Toolforge iteration 11): [jobs-api, jobs-cli] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363346#9860060 (10dcaro) [15:00:15] 10Toolforge (Toolforge iteration 11), 13Patch-For-Review: [builds-api, builds-cli] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363808#9860054 (10dcaro) [15:00:15] 06cloud-services-team, 10Toolforge (Toolforge iteration 11): toolforge: Refresh certs that are not controlled by kubeadm (mid 2024 edition) - https://phabricator.wikimedia.org/T309782#9860072 (10dcaro) 05Open→03In progress [15:00:32] 10Toolforge (Toolforge iteration 11): Toolforge Aptfile not producing working copy of `ffmpeg` - https://phabricator.wikimedia.org/T365633#9860075 (10dcaro) [15:00:37] 06cloud-services-team, 10Toolforge (Toolforge iteration 11): maintain-kubeusers: metrics, monitoring and alerting - https://phabricator.wikimedia.org/T366598#9860076 (10dcaro) [15:00:42] 10Toolforge (Toolforge iteration 11): [maintain-kubeusers,infra] deal with tools with invalid names - https://phabricator.wikimedia.org/T366477#9860077 (10dcaro) [15:00:47] 10Toolforge (Toolforge iteration 11): [jobs-cli] enforce proper validation for load jobs before calculate_changes - https://phabricator.wikimedia.org/T366211#9860078 (10dcaro) [15:00:51] 10Toolforge (Toolforge iteration 11): [jobs-api] move jobs load feature to the backend - https://phabricator.wikimedia.org/T366209#9860079 (10dcaro) [15:00:53] 06cloud-services-team, 10Toolforge (Toolforge iteration 11), 13Patch-For-Review: toolforge: review pod templates for PSP replacement - https://phabricator.wikimedia.org/T362050#9860064 (10dcaro) [15:00:55] 10Toolforge (Toolforge iteration 11): [jobs-api] Save business models in a DB - https://phabricator.wikimedia.org/T359650#9860080 (10dcaro) [15:00:59] 10Toolforge (Toolforge iteration 11): [webservice-cli] `webservice logs -f` should expect KeyboardInterrupt - https://phabricator.wikimedia.org/T361437#9860081 (10dcaro) [15:01:03] 10Toolforge (Toolforge iteration 11): [jobs-api,builds-api,envvars-api] consolidate api paths - https://phabricator.wikimedia.org/T365014#9860082 (10dcaro) [15:01:07] 10Toolforge (Toolforge iteration 11): [builds-api,envvars-api] bump the version in the openapi definition when bumping the package version - https://phabricator.wikimedia.org/T356972#9860083 (10dcaro) [15:01:11] 10Toolforge (Toolforge iteration 11), 07Epic: [jobs-cli,builds-cli,toolforge-cli,webservice] Consolidate the Toolforge CLIs - https://phabricator.wikimedia.org/T356262#9860084 (10dcaro) [15:01:15] 10Toolforge (Toolforge iteration 11), 13Patch-For-Review: [builds-api,jobs-api,envvars-api,api-gateway] Figure out and document how to do non-backwards compatible changes - https://phabricator.wikimedia.org/T356974#9860068 (10dcaro) [15:01:19] 10Toolforge (Toolforge iteration 11), 13Patch-For-Review: [k8s] Add node anti-affinity topologySpreadConstraints to infrastructure components where relevant - https://phabricator.wikimedia.org/T358203#9860066 (10dcaro) [15:01:28] 10Toolforge (Toolforge iteration 11): [builds-cli,builds-api] `build quota` fails if tool has no builds - https://phabricator.wikimedia.org/T353701#9860085 (10dcaro) [15:01:32] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 11): [docs] Create a tutorial on how to deploy a Node.js app using Build Service - https://phabricator.wikimedia.org/T353313#9860086 (10dcaro) [15:01:36] 06cloud-services-team, 10Toolforge (Toolforge iteration 11): [api-gateway] add alert for uptime - https://phabricator.wikimedia.org/T348633#9860088 (10dcaro) [15:01:40] 10Toolforge (Toolforge iteration 11), 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project: [maintain-harbor,docs] Document current setup and admin procedures - https://phabricator.wikimedia.org/T329176#9860089 (10dcaro) [15:01:44] 10Toolforge (Toolforge iteration 11), 07Documentation: [harbor,docs] Improve Harbor quota handling and docs - https://phabricator.wikimedia.org/T351092#9860087 (10dcaro) [15:02:04] 06cloud-services-team, 10Toolforge (Toolforge iteration 11): [infra,k8s,monitoring] Add an alert to warn when the prometheus k8s cert is about to expire - https://phabricator.wikimedia.org/T366579#9860099 (10taavi) https://samber.github.io/awesome-prometheus-alerts/rules.html#rule-kubernetes-1-32 [15:13:09] 10Cloud-Services, 06serviceops, 06SRE: Modernise memcached systemd unit / sync, and make it presentable - https://phabricator.wikimedia.org/T273950#9860167 (10jijiki) [15:13:14] 10Cloud-Services, 06serviceops, 06SRE: Modernise memcached systemd unit / sync, and make it presentable - https://phabricator.wikimedia.org/T273950#9860169 (10jijiki) 05Open→03In progress [15:19:16] 10Tool-Global-user-contributions, 06Stewards-and-global-tools, 10XTools, 07Epic, 10Temporary accounts (Create/update essential tools/anti-abuse management): Investigate: How to make the GUC query performant - https://phabricator.wikimedia.org/T355672#9860194 (10Tchanders) [15:21:25] 10Tool-Global-user-contributions, 06Stewards-and-global-tools, 07Design, 10Temporary accounts (Create/update essential tools/anti-abuse management): [Design EPIC] Global User Contributions - https://phabricator.wikimedia.org/T349901#9860210 (10Tchanders) [15:23:24] (03approved) 10fnegri: add envvars admission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/135 (owner: 10dcaro) [15:25:01] 06cloud-services-team, 10Toolforge (Toolforge iteration 11): [infra,k8s,monitoring] Add an alert to warn when the prometheus k8s cert is about to expire - https://phabricator.wikimedia.org/T366579#9860261 (10dcaro) >>! In T366579#9860099, @taavi wrote: > https://samber.github.io/awesome-prometheus-alerts/rules... [15:25:04] 10Toolforge, 13Patch-For-Review: Provide a Redis container for use within a tool's namespace - https://phabricator.wikimedia.org/T360378#9860262 (10bd808) >>! In T360378#9859637, @dcaro wrote: > Now that we have services in jobs (docs coming soon), @bd808 do you want to continue your work on this or do you wan... [15:26:51] 10wikitech.wikimedia.org, 06DBA, 10MediaModeration, 06Trust and Safety Product Team, 07Wikimedia-production-error: extension1 database for wikitech is always overloaded - https://phabricator.wikimedia.org/T366574#9860264 (10Ladsgroup) I hope we get there before that. The thing is that every time we add a... [15:27:38] (03update) 10dcaro: add envvars admission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/135 [15:27:39] (03approved) 10dcaro: add envvars admission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/135 [15:27:43] (03merge) 10dcaro: add envvars admission [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/135 [15:29:01] (03open) 10aborrero: alerts: add maintain-kubeusers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/12 (https://phabricator.wikimedia.org/T366598) [15:37:44] (03open) 10raymond-ndibe: [jobs-api] move simple job validations to pydantic [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/89 (https://phabricator.wikimedia.org/T366209) [15:40:01] (03update) 10raymond-ndibe: [jobs-api] move simple job validations to pydantic [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/89 (https://phabricator.wikimedia.org/T366209) [15:42:43] (03CR) 10Arturo Borrero Gonzalez: [C:03+1] "LGTM." [docker-images/toollabs-images] - 10https://gerrit.wikimedia.org/r/1012797 (https://phabricator.wikimedia.org/T360378) (owner: 10BryanDavis) [15:46:18] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 11): Intermittent redis connection timeouts in Toolforge - https://phabricator.wikimedia.org/T318479#9860388 (10RoySmith) > This shows the request remains stuck for 15 minutes before failing with the Redis ConnectionError. I haven't fo... [15:47:08] (03approved) 10dcaro: kubeconfig: remove resource changes in the update routine [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/33 (owner: 10aborrero) [15:47:09] (03update) 10dcaro: kubeconfig: remove resource changes in the update routine [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/33 (owner: 10aborrero) [15:52:30] 10Cloud-VPS: Drop 68.10.in-addr.arpa. from Designate - https://phabricator.wikimedia.org/T361220#9860403 (10taavi) 05Open→03Resolved a:03taavi `lang=shell-session taavi@cloudcontrol1005 ~ $ os zone delete 8d114f3c-815b-466c-bdd4-9b91f704ea60 --sudo-project-id noauth-project ` [15:52:38] (03approved) 10aborrero: kubeconfig: remove resource changes in the update routine [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/33 [15:52:42] (03merge) 10aborrero: kubeconfig: remove resource changes in the update routine [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/33 [15:53:37] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.140-20240604142418-04781a71 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/308 [15:55:00] 10Data-Services: [cloud-vps] Deprecate clouddb-services project - https://phabricator.wikimedia.org/T365975#9860416 (10taavi) [15:58:33] 10PAWS, 10Quarry: update github action - https://phabricator.wikimedia.org/T348873#9860452 (10taavi) [15:59:08] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 10Data-Services: cloudcumin: allow wmcs-admin to run wikireplicas cookbooks and scripts - https://phabricator.wikimedia.org/T347977#9860465 (10taavi) [16:00:02] (03approved) 10dcaro: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/31 (https://phabricator.wikimedia.org/T363809) (owner: 10sstefanova) [16:00:04] (03update) 10dcaro: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/31 (https://phabricator.wikimedia.org/T363809) (owner: 10sstefanova) [16:01:03] 10Cloud-VPS: Cannot set up standalone puppetmaster due to stray ruby process at port 8140 - https://phabricator.wikimedia.org/T343628#9860487 (10taavi) 05Open→03Invalid Please re-open if this is an issue with the new jdk-based puppetserver too. [16:01:08] 10Cloud-VPS, 07Documentation: Deprecate Help:Resize_root_partition_of_an_OpenStack_hosted_virtual_machine - https://phabricator.wikimedia.org/T347890#9860479 (10taavi) 05Open→03Resolved a:03taavi That page is not linked from anywhere and it has a "do not do this" banner so I'm calling this done. [16:02:17] (03update) 10dcaro: cli: use prefixed endpoints [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/43 (owner: 10sstefanova) [16:02:36] 06cloud-services-team, 10Cloud-VPS, 10Observability-Metrics: Current status of cloudmetrics and its components - https://phabricator.wikimedia.org/T336774#9860495 (10taavi) 05Open→03Resolved a:03taavi [16:02:40] 10VPS-Projects: Cleanup memberships of maps project - https://phabricator.wikimedia.org/T323412#9860505 (10taavi) [16:03:16] 10Cloud-VPS: Bullseye Cloud VPS hosts show "Your environment specifies an invalid locale." error - https://phabricator.wikimedia.org/T354044#9860514 (10taavi) [16:03:47] 10Toolforge, 13Patch-For-Review: Provide a Redis container for use within a tool's namespace - https://phabricator.wikimedia.org/T360378#9860492 (10bd808) A new discussion thread that has happened on IRC is related to the idea of making this container using the build service (in the style of https://gitlab.wik... [16:03:48] 10Cloud-VPS: Cloud VPS Bullseye box uses invalid locale - https://phabricator.wikimedia.org/T321994#9860512 (10taavi) →14Duplicate dup:03T354044 [16:03:58] 06cloud-services-team, 10Data-Services, 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Unplanned: [cloudvps] Find and cleanup any mounts to labstore1006/1007 - https://phabricator.wikimedia.org/T320425#9860516 (10taavi) [16:04:44] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 11): Intermittent redis connection timeouts in Toolforge - https://phabricator.wikimedia.org/T318479#9860532 (10RoySmith) What makes this even weirder is I've got my cache configured with a 300 second timeout: ` REDIS_CACHE = {... [16:06:55] 06cloud-services-team, 10Cloud-VPS: Recommended solution for Terraform state backend - https://phabricator.wikimedia.org/T318360#9860553 (10taavi) 05Open→03Resolved a:03taavi Calling this done with https://wikitech.wikimedia.org/wiki/Help:Using_OpenTofu_on_Cloud_VPS#State_management. [16:08:38] 06cloud-services-team, 10Horizon: Horizon User Management: page ldap queries - https://phabricator.wikimedia.org/T298531#9860577 (10taavi) [16:09:16] (03update) 10aborrero: alerts: add maintain-kubeusers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/12 (https://phabricator.wikimedia.org/T366598) [16:10:52] (03update) 10aborrero: alerts: add maintain-kubeusers [repos/cloud/toolforge/alerts] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/alerts/-/merge_requests/12 (https://phabricator.wikimedia.org/T366598) [16:11:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:11:52] 06cloud-services-team, 10Cloud-VPS: graphite.wmflabs.org no longer purges data for deleted instances - https://phabricator.wikimedia.org/T93861#9860595 (10taavi) 05Open→03Invalid Graphite is gone. [16:12:02] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [16:12:13] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [16:12:37] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [16:12:48] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [16:12:55] 10Tool-gitlab-account-approval: Add error logging - https://phabricator.wikimedia.org/T361079#9860616 (10bd808) >>! In T361079#9859186, @Aklapper wrote: > On a related node, `@glaab` sometimes throws `ERR-INVALID-PARAMETER` when calling Phabricator's `user.ldapquery` or `user.mediawikiquery` Conduit APIs accordi... [16:13:00] (03merge) 10aborrero: maintain-kubeusers: bump to 0.0.140-20240604142418-04781a71 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/308 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:13:55] 06cloud-services-team, 10Cloud-VPS: Make an evacuation plan for labs instances - https://phabricator.wikimedia.org/T106144#9860614 (10taavi) [16:14:51] 06cloud-services-team, 10Cloud-VPS: Get instance block-migration working reliably; script and document - https://phabricator.wikimedia.org/T106146#9860611 (10taavi) 05Open→03Declined AIUI this is irrelevant on Ceph. [16:17:58] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:22:50] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:27:53] (03approved) 10dcaro: cli: use prefixed endpoints [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/43 (owner: 10sstefanova) [16:32:24] 06cloud-services-team, 10Cloud-VPS: Get instance block-migration working reliably; script and document - https://phabricator.wikimedia.org/T106146#9860702 (10Andrew) >>! In T106146#9860611, @taavi wrote: > AIUI this is irrelevant on Ceph. agreed [16:34:41] 10Tool-gitlab-account-approval: Add error logging - https://phabricator.wikimedia.org/T361079#9860713 (10bd808) 05Open→03Resolved a:03bd808 I did this quite a while ago, but apparently forgot I had made a task about doing it. Logging and log rotation are done with jobs framework bits: `lang=yaml # http... [16:34:44] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 11): Intermittent redis connection timeouts in Toolforge - https://phabricator.wikimedia.org/T318479#9860719 (10fnegri) I wonder if it could be something that's not related to Redis at all, but instead something else that blocks the ap... [16:35:38] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-codfw: decommission cloudcontrol2001-dev.codfw.wmnet - https://phabricator.wikimedia.org/T364577#9860723 (10Andrew) [16:37:19] 06cloud-services-team, 10VPS-Projects, 10fundraising-tech-ops, 10Puppet (Puppet 7.0): Update puppet civicrm-prototype puppetmaster - https://phabricator.wikimedia.org/T361595#9860734 (10Andrew) Hello! Can I get an update on the status of this work? [16:37:41] 06cloud-services-team, 06Infrastructure-Foundations, 10Puppet-Infrastructure, 13Patch-For-Review: puppet servers run out of inodes in puppet code volume - https://phabricator.wikimedia.org/T364047#9860730 (10Andrew) 05Open→03Resolved This was resolved by https://gerrit.wikimedia.org/r/c/operations/... [16:39:05] 06cloud-services-team, 10Cloud-VPS, 05Goal: Update designate-sink handlers to catch up with upstream refactors - https://phabricator.wikimedia.org/T356515#9860736 (10Andrew) 05Open→03Resolved a:03Andrew [16:43:43] 10Data-Services: [wikireplicas] clouddb* free memory decreases over time - https://phabricator.wikimedia.org/T365164#9860777 (10fnegri) I did not restart the services, but the alerts disappeared from alerts.wikimedia.org. I can see they are still in status WARNING in Icinga though, I'm not sure why they are no l... [16:45:21] 10Tool-phab-ban, 07Technical-Debt: Deploy more recent PhabBanBot code from repository into production - https://phabricator.wikimedia.org/T366587#9860778 (10bd808) >>! In T366587#9859152, @Aklapper wrote: > @bd808: Would you have an idea how/who to achieve that, maybe? :) `lang=shell-session $ ssh login.toolf... [16:48:51] 10Toolforge (Toolforge iteration 11): Toolforge Aptfile not producing working copy of `ffmpeg` - https://phabricator.wikimedia.org/T365633#9860785 (10dcaro) It seems libpulse0 package is being pulled and installed: ` [step-build] 2024-06-04T16:40:49.714002081Z -----> Fetching .debs for libpulse0 [step-build] 202... [17:07:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:08:46] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T366588#9860817 (10Curb_Safe_Charmer) 05Open→03Resolved It is working again now - no idea why. [17:17:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:21:03] 10Toolforge (Toolforge iteration 11): Toolforge Aptfile not producing working copy of `ffmpeg` - https://phabricator.wikimedia.org/T365633#9860868 (10dcaro) It thinks it's a virtual package: ` [step-build] 2024-06-04T16:32:46.813638806Z Choosing liboss4-salsa-asound2 for virtual package libasound2 ` From... [17:37:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:47:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:23:41] (03open) 10sstefanova: openapi: refactor yaml [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/90 [18:38:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:53:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:01:16] 06cloud-services-team, 10Toolforge: [toolforge,storage] Provide per-tool access to cloud-vps object storage - https://phabricator.wikimedia.org/T358496#9861310 (10Andrew) I'm converging on a new design, which is a variant of 'Automatic creation of per-tool keystone project (projects in database)': Some persi... [20:07:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:17:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:22:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:31:16] FIRING: [3x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [20:36:16] RESOLVED: [3x] ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_main_page_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [21:11:56] 06cloud-services-team, 10Cloud-VPS, 07Epic: Wind down use of project ID and project name equivalency in OpenStack - https://phabricator.wikimedia.org/T274268#9861767 (10Andrew) >>! In T274268#6815698, @Bstorm wrote: > One of the first things to understand here is why our CLI doesn't support names. Upstre... [21:12:59] 06cloud-services-team, 10Cloud-VPS, 07Epic: Wind down use of project ID and project name equivalency in OpenStack - https://phabricator.wikimedia.org/T274268#9861771 (10Andrew) 05Open→03Resolved a:03Andrew New projects will now be created with a uuid instead of an id == name. Existing projects will... [21:19:57] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T366651 (10Ost316) 03NEW [21:35:35] (03open) 10ladsgroup: Add support for switchover of ES hosts [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/3 (https://phabricator.wikimedia.org/T365098) [21:43:47] (03update) 10ladsgroup: Add support for switchover of ES hosts [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/3 (https://phabricator.wikimedia.org/T365098) [21:45:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-35 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [21:56:44] 06cloud-services-team, 10VPS-Projects, 10fundraising-tech-ops, 10Puppet (Puppet 7.0): Update puppet civicrm-prototype puppetmaster - https://phabricator.wikimedia.org/T361595#9861953 (10Dwisehaupt) We hit a snag with the deployment to prod and are still working through an issue (T363571). I'm hoping that w... [21:57:26] 10Cloud-VPS (Project-requests), 10Performance-Device-Lab, 06Quality-and-Test-Engineering-Team, 10Synthetic-Performance-Testing: Request creation of web performance test VPS project - https://phabricator.wikimedia.org/T366569#9861958 (10Andrew) 05Open→03Resolved a:03Andrew I've created this projec... [22:00:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-35 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [22:09:00] 06cloud-services-team, 06DC-Ops, 10decommission-hardware, 10ops-codfw, 06SRE: decommission cloudcontrol2001-dev.codfw.wmnet - https://phabricator.wikimedia.org/T364577#9861989 (10Papaul) a:03Jhancock.wm @Jhancock.wm can you please proceed with this and resolve the task once done. Thanks [23:07:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:17:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks