[00:15:28] FIRING: InstanceDown: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:16:37] FIRING: [2x] ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [00:20:28] RESOLVED: InstanceDown: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [01:40:55] 06cloud-services-team: SystemdUnitDown Unit backup_glance_images.service on node cloudbackup1003 has been down for long. - https://phabricator.wikimedia.org/T366416#9853157 (10OKJ04) [01:41:30] 10Cloud-VPS: [cloudvps] 2024-05-01 cloudinfra puppetserver got out of space - https://phabricator.wikimedia.org/T366406#9853164 (10OKJ04) [01:44:09] 10Toolforge: [toolforge] [redis] Improve Puppet config - https://phabricator.wikimedia.org/T366365#9853202 (10OKJ04) [01:44:53] 10Tool-spacemedia, 10Server-side-upload-request: Server-side upload request for OptimusPrimeBot (INPE DPI) - https://phabricator.wikimedia.org/T366353#9853214 (10OKJ04) [01:45:05] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review, 10Puppet (Puppet 7.0): cloud-vps puppetservers filling up / with puppetserver reports - https://phabricator.wikimedia.org/T366357#9853210 (10OKJ04) [01:48:14] 06cloud-services-team: SystemdUnitDown Unit backup_glance_images.service on node cloudbackup1003 has been down for long. - https://phabricator.wikimedia.org/T366416#9853260 (10JJMC89) [01:48:46] 10Tool-global-search: Global Search: Language selector doesn't work and 7 languages ​​have no labels - https://phabricator.wikimedia.org/T366410#9853264 (10JJMC89) [01:48:54] 10Cloud-VPS: [cloudvps] 2024-05-01 cloudinfra puppetserver got out of space - https://phabricator.wikimedia.org/T366406#9853267 (10JJMC89) [01:51:31] 10Toolforge: [toolforge] [redis] Improve Puppet config - https://phabricator.wikimedia.org/T366365#9853304 (10JJMC89) [01:52:15] 10Tool-spacemedia, 10Server-side-upload-request: Server-side upload request for OptimusPrimeBot (INPE DPI) - https://phabricator.wikimedia.org/T366353#9853316 (10JJMC89) [01:52:31] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review, 10Puppet (Puppet 7.0): cloud-vps puppetservers filling up / with puppetserver reports - https://phabricator.wikimedia.org/T366357#9853312 (10JJMC89) [02:28:18] 06cloud-services-team, 10Horizon, 10Fiwiki-Wikidata-Commons, 07LDAP: Inconsistency between r/w and r/o ldap - https://phabricator.wikimedia.org/T366310#9853358 (10Andrew) 05Open→03Resolved a:03Andrew Thank you @bd808 ! By following Alex's weird advice on that ticket (recreate a new record with t... [02:57:23] 10Toolforge, 07LDAP: toolforge: ldap: disabling an account creates 2 entries in the LDAP tree - https://phabricator.wikimedia.org/T366263#9853362 (10Andrew) 05Open→03Resolved I've renamed the five misnamed records, and deleted the .test8 record. ` root@cloudcontrol1006:~# ldapsearch -x -E pr=5000/nop... [03:06:52] 10Tool-bub2: Integrate Gallica (Bibliothèque Nationale de France) - https://phabricator.wikimedia.org/T354381#9853365 (10theprotonade) //Adding more context for this task// [[ https://gallica.bnf.fr | Gallica ]] is the French National Library's website for copies of books, newspapers and digital records - [[ h... [03:11:25] 10Tool-bub2: Integrate Limédia Kiosque/Galeries (Sillon Lorrain, France) - https://phabricator.wikimedia.org/T354382#9853367 (10theprotonade) //Adding more context to this task// **Limédia Kiosque** provides Lorraine newspapers, journals of learned societies, directories, etc. preserved in the libraries of the... [03:41:51] (03open) 10bd808: Implement all the things [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/1 [03:45:45] (03merge) 10bd808: Implement all the things [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/1 [04:41:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:46:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:51:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:56:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:19:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-35 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [05:24:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-35 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [06:13:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:16:46] 10PAWS, 07Upstream: PAWS kills active users servers that are not connected to a user session - https://phabricator.wikimedia.org/T188684#9853451 (10Lokal_Profil) [06:23:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:55:47] 10Toolforge, 07Kubernetes: Shell pods continue running after ssh session exits - https://phabricator.wikimedia.org/T315735#9853646 (10dcaro) As a workaround for now you can delete all of them with: ` kubectl delete pods -l 'app.kubernetes.io/component=webservice-interactive' ` And one by one with: ` kubectl d... [07:58:10] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-admission/-/merge_requests/7 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:58:22] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-admission/-/merge_requests/7 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:59:26] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/20 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:59:29] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/20 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:59:42] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-admission/-/merge_requests/7 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [07:59:54] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/20 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:01:06] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/95 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:01:10] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/95 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:01:12] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/95 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:01:33] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/9 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:01:34] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/9 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:01:38] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/volume-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/volume-admission/-/merge_requests/9 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:01:58] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.23-20240603075945-18fc26ce [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/293 [08:02:58] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/4 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:03:02] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/4 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:03:02] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/ingress-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/ingress-admission/-/merge_requests/4 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:03:43] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/4 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:03:46] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/4 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:03:50] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/4 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:04:47] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: volume-admission: bump to 0.0.47-20240603080151-200d2924 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/294 [08:05:41] (03approved) 10sstefanova: nginx: increase read timeout for the logs endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/87 (https://phabricator.wikimedia.org/T359953) (owner: 10dcaro) [08:06:23] (03update) 10sstefanova: nginx: increase read timeout for the logs endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/87 (https://phabricator.wikimedia.org/T359953) (owner: 10dcaro) [08:06:31] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: ingress-admission: bump to 0.0.42-20240603080316-25cb95db [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/295 [08:07:13] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: registry-admission: bump to 0.0.40-20240603080405-5c500731 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/296 [08:07:15] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: registry-admission: bump to 0.0.40-20240603080405-5c500731 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/296 [08:08:55] (03approved) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/20 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:08:58] (03merge) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/toolforge-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-cli/-/merge_requests/20 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:10:02] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/21 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [08:11:02] (03PS1) 10Muehlenhoff: Remove obsolete stub certs [labs/private] - 10https://gerrit.wikimedia.org/r/1038222 (https://phabricator.wikimedia.org/T364622) [08:12:15] (03approved) 10dcaro: functional-tests: increase retry default time [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/292 (owner: 10aborrero) [08:12:19] (03update) 10dcaro: functional-tests: increase retry default time [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/292 (owner: 10aborrero) [08:16:38] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission [08:16:50] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission [08:19:04] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove obsolete stub certs [labs/private] - 10https://gerrit.wikimedia.org/r/1038222 (https://phabricator.wikimedia.org/T364622) (owner: 10Muehlenhoff) [08:21:29] (03CR) 10David Caro: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1037821 (owner: 10Arturo Borrero Gonzalez) [08:21:37] (03CR) 10David Caro: [C:03+1] "LGTM" [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1037822 (https://phabricator.wikimedia.org/T364113) (owner: 10Arturo Borrero Gonzalez) [08:24:25] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission [08:24:28] (03merge) 10aborrero: functional-tests: increase retry default time [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/292 [08:24:37] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission [08:24:47] (03CR) 10Arturo Borrero Gonzalez: [C:03+2] wmcs_libs: add image controller to refactor upload function [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1037821 (owner: 10Arturo Borrero Gonzalez) [08:25:15] (03update) 10sstefanova: registry-admission: bump to 0.0.40-20240603080405-5c500731 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/296 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:25:16] (03approved) 10sstefanova: registry-admission: bump to 0.0.40-20240603080405-5c500731 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/296 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:25:41] (03update) 10sstefanova: registry-admission: bump to 0.0.40-20240603080405-5c500731 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/296 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:26:07] (03merge) 10sstefanova: registry-admission: bump to 0.0.40-20240603080405-5c500731 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/296 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:27:29] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission [08:27:38] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission [08:27:59] 06cloud-services-team, 06Infrastructure-Foundations, 10Puppet-Infrastructure, 13Patch-For-Review: puppet servers run out of inodes in puppet code volume - https://phabricator.wikimedia.org/T364047#9853753 (10dcaro) This is probably related to {T366406} [08:29:23] 10Cloud-VPS: [cloudvps] 2024-05-01 cloudinfra puppetserver got out of space - https://phabricator.wikimedia.org/T366406#9853768 (10dcaro) Interestingly enough, today there's even more free space (somehow 2G of puppet reports got cleared up): ` root@cloudinfra-cloudvps-puppetserver-1:~# du -hs /var/lib/puppetserv... [08:31:37] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component ingress-admission [08:31:46] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component ingress-admission [08:31:56] (03update) 10sstefanova: ingress-admission: bump to 0.0.42-20240603080316-25cb95db [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/295 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:31:58] (03approved) 10sstefanova: ingress-admission: bump to 0.0.42-20240603080316-25cb95db [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/295 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:32:04] (03update) 10sstefanova: ingress-admission: bump to 0.0.42-20240603080316-25cb95db [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/295 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:32:35] (03merge) 10sstefanova: ingress-admission: bump to 0.0.42-20240603080316-25cb95db [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/295 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:33:09] (03update) 10sstefanova: volume-admission: bump to 0.0.47-20240603080151-200d2924 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/294 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:34:34] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission [08:34:43] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission [08:37:55] 10Cloud-VPS: [cloudvps] 2024-05-01 cloudinfra puppetserver got out of space - https://phabricator.wikimedia.org/T366406#9853803 (10dcaro) From puppet docs (https://www.puppet.com/docs/puppet/8/report#report-store): ` store Stores the yaml report in the configured reportdir. By default, this is the report proces... [08:38:26] 10Cloud-VPS: [cloudvps] 2024-05-01 cloudinfra puppetserver got out of space - https://phabricator.wikimedia.org/T366406#9853815 (10dcaro) There it is: ` Mon 2024-06-03 14:47:21 UTC 6h left Mon 2024-06-03 06:47:20 UTC 1h 50min ago remove_old_puppet_reports.timer remove_old_puppet_repor... [08:41:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:43:35] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component volume-admission [08:43:45] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component volume-admission [08:45:01] 10Cloud-VPS: [cloudvps] 2024-05-01 cloudinfra puppetserver got out of space - https://phabricator.wikimedia.org/T366406#9853857 (10dcaro) It's set to 16h: ` root@cloudinfra-cloudvps-puppetserver-1:~# systemctl status remove_old_puppet_reports.service ○ remove_old_puppet_reports.service - Clears out older puppet... [08:46:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:51:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:53:01] (03update) 10sstefanova: volume-admission: bump to 0.0.47-20240603080151-200d2924 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/294 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:53:02] (03approved) 10sstefanova: volume-admission: bump to 0.0.47-20240603080151-200d2924 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/294 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:53:07] (03merge) 10sstefanova: volume-admission: bump to 0.0.47-20240603080151-200d2924 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/294 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [08:55:06] 10Cloud Services Proposals, 10Toolforge: toolforge: introduce docker-registry.svc.toolforge.org FQDN to replace docker-registry.tools.wmflabs.org - https://phabricator.wikimedia.org/T366453 (10aborrero) 03NEW [08:56:11] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [08:56:21] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway [08:56:37] FIRING: [3x] ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [08:56:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:57:29] PROBLEM - toolschecker: Redis set/get on checker.tools.wmflabs.org is CRITICAL: HTTP CRITICAL: HTTP/1.1 504 Gateway Time-out - string OK not found on http://checker.tools.wmflabs.org:80/redis - 324 bytes in 60.010 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [08:59:27] RECOVERY - toolschecker: Redis set/get on checker.tools.wmflabs.org is OK: HTTP OK: HTTP/1.1 200 OK - 158 bytes in 54.334 second response time https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Toolschecker [09:01:38] FIRING: [3x] ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:06:38] FIRING: [3x] ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:12:59] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [09:13:11] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway [09:16:38] FIRING: [4x] ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:18:23] (03update) 10sstefanova: api-gateway: bump to 0.0.23-20240603075945-18fc26ce [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/293 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:18:29] (03update) 10sstefanova: api-gateway: bump to 0.0.23-20240603075945-18fc26ce [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/293 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:19:30] (03merge) 10sstefanova: api-gateway: bump to 0.0.23-20240603075945-18fc26ce [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/293 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:19:59] (03update) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/21 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:20:06] (03merge) 10sstefanova: poetry: Autoupdate [repos/cloud/toolforge/api-gateway] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/21 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [09:21:37] RESOLVED: [3x] ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_admin_beta_toolforge_org_ip4) - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [09:22:10] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: api-gateway: bump to 0.0.24-20240603092014-c6e82aab [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/297 [09:25:16] !log sstefanova@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [09:25:27] !log sstefanova@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway [09:26:12] (03PS2) 10Arturo Borrero Gonzalez: wmcs.toolforge.k8s: add cookbook to upload kyverno container images [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1037822 (https://phabricator.wikimedia.org/T364113) [09:26:41] (03CR) 10Arturo Borrero Gonzalez: wmcs.toolforge.k8s: add cookbook to upload kyverno container images (032 comments) [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1037822 (https://phabricator.wikimedia.org/T364113) (owner: 10Arturo Borrero Gonzalez) [09:28:59] !log arturo@nostromo tools START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry [09:28:59] !log arturo@nostromo tools END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) [09:29:01] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:29:02] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:29:23] !log sstefanova@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component api-gateway [09:29:34] !log sstefanova@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component api-gateway [09:29:41] !log arturo@nostromo tools START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry [09:29:42] !log arturo@nostromo tools END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) [09:29:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:29:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:37:40] !log arturo@nostromo tools START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry [09:37:40] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 [09:37:41] (03update) 10sstefanova: api-gateway: bump to 0.0.24-20240603092014-c6e82aab [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/297 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:37:42] (03approved) 10sstefanova: api-gateway: bump to 0.0.24-20240603092014-c6e82aab [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/297 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:37:43] !log arturo@nostromo tools END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) [09:37:43] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:37:44] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:37:45] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [09:37:47] (03merge) 10sstefanova: api-gateway: bump to 0.0.24-20240603092014-c6e82aab [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/297 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [09:40:26] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 10): [toolforge] Redis refusing connections - https://phabricator.wikimedia.org/T363709#9854046 (10fnegri) I applied the change today, by removing the config files and letting Puppet recreate them as described in the previous comment.... [10:05:18] 10Cloud Services Proposals, 10Toolforge, 13Patch-For-Review: toolforge: introduce docker-registry.svc.toolforge.org FQDN to replace docker-registry.tools.wmflabs.org - https://phabricator.wikimedia.org/T366453#9854091 (10aborrero) a:05aborrero→03None [10:11:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:13:28] !log arturo@nostromo tools START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry [10:13:28] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 [10:13:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:13:31] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:13:31] !log arturo@nostromo tools END (FAIL) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=99) [10:13:33] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:14:06] !log arturo@nostromo tools START - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry [10:14:06] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 [10:14:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:14:09] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:14:34] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-kyvernopre:v1.10.7 [10:14:35] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:14:48] (03PS3) 10Arturo Borrero Gonzalez: wmcs.toolforge.k8s: add cookbook to upload kyverno container images [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1037822 (https://phabricator.wikimedia.org/T364113) [10:14:58] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-background-controller:v1.10.7 [10:14:59] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:15:22] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-cleanup-controller:v1.10.7 [10:15:23] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:15:46] !log arturo@nostromo tools Updating container image docker-registry.tools.wmflabs.org/toolforge-kyverno-reports-controller:v1.10.7 [10:15:49] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:16:06] !log arturo@nostromo tools END (PASS) - Cookbook wmcs.toolforge.k8s.kyverno.copy_images_to_registry (exit_code=0) [10:16:08] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [10:17:06] (03CR) 10Arturo Borrero Gonzalez: [C:03+2] wmcs.toolforge.k8s: add cookbook to upload kyverno container images [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/1037822 (https://phabricator.wikimedia.org/T364113) (owner: 10Arturo Borrero Gonzalez) [10:21:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:22:46] 10Cloud Services Proposals, 10Toolforge, 13Patch-For-Review: toolforge: introduce docker-registry.svc.toolforge.org FQDN to replace docker-registry.tools.wmflabs.org - https://phabricator.wikimedia.org/T366453#9854225 (10aborrero) This change triggered a discussion about the security of the Cloud VPS opensta... [10:33:10] 06cloud-services-team, 10Toolforge: toolforge: identify and cache in our container registry all kyverno images - https://phabricator.wikimedia.org/T364113#9854246 (10aborrero) 05Open→03Resolved Done, we have now: * docker-registry.tools.wmflabs.org/toolforge-kyverno-kyverno:v1.10.7 * docker-registry.t... [10:38:13] (03open) 10dcaro: docs: add a howto create the images [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/40 [10:42:59] (03update) 10aborrero: kubernetes: introduce securityContext in the pod template [repos/cloud/toolforge/tools-webservice] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/tools-webservice/-/merge_requests/37 (https://phabricator.wikimedia.org/T362050) [10:43:34] (03approved) 10fnegri: docs: add a howto create the images [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/40 (owner: 10dcaro) [10:47:06] (03update) 10dcaro: docs: add a howto create the images [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/40 [10:47:07] (03approved) 10dcaro: docs: add a howto create the images [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/40 [10:47:10] (03merge) 10dcaro: docs: add a howto create the images [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/40 [11:18:53] 06cloud-services-team, 10Toolforge: toolforge: create a PSP migration plan - https://phabricator.wikimedia.org/T364297#9854407 (10aborrero) updated plan: [x] finish {T362872} [] finish {T362050} [x] finish {T364113} [] deploy refactored maintain-kubeusers -- {T364312} [] deploy kyverno itself -- https://gitl... [11:19:28] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [11:26:36] (03PS1) 10Muehlenhoff: Remove obsolete stub certs for labvirt-star [labs/private] - 10https://gerrit.wikimedia.org/r/1038264 [11:28:00] (03PS1) 10Muehlenhoff: Remove dummy certs for hosts long gone [labs/private] - 10https://gerrit.wikimedia.org/r/1038265 [11:39:38] 10Tool-nfp: Allow filtering by user edit count - https://phabricator.wikimedia.org/T364698#9854482 (10Ladsgroup) I will try to implement this but one thing is that if a user is experienced, they should have auto patrol right and they would be automatically excluded. [11:48:35] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [11:57:30] 06cloud-services-team, 10Toolforge (Toolforge iteration 10), 13Patch-For-Review: [maintain-kubeusers,infra,k8s]: introduce some logic to backfill maintain-kubeuser resources (like per-tool kyverno policies) - https://phabricator.wikimedia.org/T364312#9854509 (10aborrero) Deployment plan: * don't remove/updat... [12:02:21] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [12:02:31] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [12:06:34] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/4 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [12:06:38] (03approved) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/4 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [12:06:42] (03merge) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/4 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [12:07:21] (03update) 10sstefanova: pre-commit: Autoupdate [repos/cloud/toolforge/envvars-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-admission/-/merge_requests/4 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [12:15:47] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [12:17:54] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [12:18:02] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [12:18:16] (03update) 10sstefanova: [envvar-api] remove unused code [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/30 (owner: 10raymond-ndibe) [12:18:21] (03approved) 10sstefanova: [envvar-api] remove unused code [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/30 (owner: 10raymond-ndibe) [12:23:57] (03update) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [12:24:02] (03open) 10l10n-bot: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/10 [12:24:09] (03approved) 10sstefanova: [envvars-cli] remove unused code [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/41 (owner: 10raymond-ndibe) [12:24:37] (03update) 10sstefanova: [envvars-cli] remove unused code [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/41 (owner: 10raymond-ndibe) [12:26:54] (03update) 10sstefanova: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/93 (https://phabricator.wikimedia.org/T363808) [12:30:19] (03merge) 10aborrero: maintain_kubeusers: introduce resource abstraction [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/23 (https://phabricator.wikimedia.org/T364312) [12:32:47] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.134-20240603123028-c8c2ea33 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/298 (https://phabricator.wikimedia.org/T364312) [12:33:25] (03update) 10aborrero: maintain-kubeusers: deploy new resource abstraction in toolsbeta [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/287 (https://phabricator.wikimedia.org/T364312) [12:35:57] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [12:36:08] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [12:41:20] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [12:41:33] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [12:47:48] (03update) 10aborrero: maintain-kubeusers: deploy new resource abstraction in toolsbeta [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/287 (https://phabricator.wikimedia.org/T364312) [12:47:53] (03update) 10aborrero: maintain-kubeusers: deploy new resource abstraction [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/287 (https://phabricator.wikimedia.org/T364312) [12:48:40] (03update) 10sstefanova: cli: use prefixed endpoints [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/71 [12:50:25] 10Toolforge (Toolforge iteration 10), 13Patch-For-Review: [envvars-api, envvars-cli] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363809#9854627 (10Slst2020) [12:50:55] (03merge) 10aborrero: maintain-kubeusers: deploy new resource abstraction [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/287 (https://phabricator.wikimedia.org/T364312) [12:51:02] 10Toolforge (Toolforge iteration 10), 13Patch-For-Review: [envvars-api, envvars-cli] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363809#9854628 (10Slst2020) [12:53:22] (03update) 10sstefanova: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/31 (https://phabricator.wikimedia.org/T363809) [12:53:24] 10Toolforge (Toolforge iteration 10): [jobs-api, jobs-cli] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363346#9854643 (10Slst2020) [12:53:42] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/10 (owner: 10l10n-bot) [12:53:45] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/10 (owner: 10l10n-bot) [12:57:19] 06cloud-services-team, 10Cloud-VPS: [openstack] APT failing to update osbpo packages in Cloud instances - https://phabricator.wikimedia.org/T366028#9854663 (10fnegri) [13:00:14] 10Toolforge: [toolforge] [redis] Prometheus exporter logging errors - https://phabricator.wikimedia.org/T366471 (10fnegri) 03NEW [13:01:10] 06cloud-services-team, 10Toolforge: [toolforge] [redis] Prometheus exporter logging errors - https://phabricator.wikimedia.org/T366471#9854687 (10fnegri) p:05Triage→03Low [13:03:20] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Toolforge (Toolforge iteration 10): [toolforge] Redis refusing connections - https://phabricator.wikimedia.org/T363709#9854693 (10fnegri) 05In progress→03Resolved [13:04:42] (03update) 10aborrero: helpers: add foxtrot-ldap-load-users.sh [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/129 [13:06:27] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove obsolete stub certs for labvirt-star [labs/private] - 10https://gerrit.wikimedia.org/r/1038264 (owner: 10Muehlenhoff) [13:06:35] (03CR) 10Muehlenhoff: [V:03+2 C:03+2] Remove dummy certs for hosts long gone [labs/private] - 10https://gerrit.wikimedia.org/r/1038265 (owner: 10Muehlenhoff) [13:10:23] (03open) 10sstefanova: cli: use prefixed endpoints [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/43 [13:10:31] (03update) 10sstefanova: cli: use prefixed endpoints [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/43 [13:13:21] (03update) 10sstefanova: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/93 [13:21:44] (03update) 10sstefanova: cli: use prefixed endpoints [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/43 [13:24:30] (03update) 10sstefanova: cli: use prefixed endpoints [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/43 [13:37:54] (03update) 10sstefanova: cli: use prefixed endpoints [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/43 [13:46:14] (03update) 10sstefanova: cli: use prefixed endpoints [repos/cloud/toolforge/envvars-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-cli/-/merge_requests/43 [13:46:47] (03update) 10sstefanova: prefix endpoints with /tool/{toolname}/ [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/31 (https://phabricator.wikimedia.org/T363809) [13:51:12] 06cloud-services-team, 10Toolforge: [toolforge] [redis] Prometheus exporter logging errors - https://phabricator.wikimedia.org/T366471#9854872 (10dcaro) Might be blocked/hashed (iirc all the commands were hashed as CLIENT-snoteuhnstoeahnstaoeunstaoe or similar) [13:59:22] (03approved) 10dcaro: helpers: add foxtrot-ldap-load-users.sh [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/129 (owner: 10aborrero) [14:00:00] (03update) 10dcaro: helpers: add foxtrot-ldap-load-users.sh [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/129 (owner: 10aborrero) [14:01:06] 10Toolforge (Toolforge iteration 10): toolforge: deal with tools with invalid names - https://phabricator.wikimedia.org/T366477 (10aborrero) 03NEW [14:01:39] (03merge) 10aborrero: helpers: add foxtrot-ldap-load-users.sh [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/129 [14:01:49] (03open) 10aborrero: quota: fix object name [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/27 [14:07:18] (03approved) 10dcaro: quota: fix object name [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/27 (owner: 10aborrero) [14:07:29] (03update) 10dcaro: quota: fix object name [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/27 (owner: 10aborrero) [14:08:11] (03merge) 10aborrero: quota: fix object name [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/27 [14:10:15] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.134-20240603123028-c8c2ea33 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/298 (https://phabricator.wikimedia.org/T364312) [14:10:50] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [14:11:00] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [14:11:20] 10Toolforge, 07Kubernetes: Shell pods continue running after ssh session exits - https://phabricator.wikimedia.org/T315735#9854998 (10dcaro) p:05Triage→03Medium [14:11:30] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [14:11:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:11:42] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [14:12:09] 10Toolforge (Toolforge iteration 10): [builds-api] Nginx times out when tailing logs - https://phabricator.wikimedia.org/T366173#9855001 (10dcaro) 05Duplicate→03Resolved [14:13:52] (03update) 10aborrero: maintain-kubeusers: bump to 0.0.135-20240603140823-9b51acbf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/298 (https://phabricator.wikimedia.org/T364312) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [14:14:04] (03update) 10aborrero: maintain-kubeusers: bump to 0.0.135-20240603140823-9b51acbf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/298 (https://phabricator.wikimedia.org/T364312) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [14:14:14] (03merge) 10aborrero: maintain-kubeusers: bump to 0.0.135-20240603140823-9b51acbf [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/298 (https://phabricator.wikimedia.org/T364312) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [14:16:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:16:41] 10Toolforge (Toolforge iteration 10): toolforge: deal with tools with invalid names - https://phabricator.wikimedia.org/T366477#9855006 (10dcaro) Look what I just found in the proxy config: ` # T301720: redirect ru_monuments to ru-monuments if ($host ~* "^ru_monuments\.(.*)$") { return 301 https:... [14:20:03] 10Cloud-VPS, 13Patch-For-Review: [cloudvps] 2024-05-01 cloudinfra puppetserver got out of space - https://phabricator.wikimedia.org/T366406#9855014 (10dcaro) Let's see how many reports per-host we have right now: ` root@cloudinfra-cloudvps-puppetserver-1:~# ls /var/lib/puppetserver/reports/ | wc 847 84... [14:20:58] 10Toolforge: toolforge jobs load flushes out all jobs - https://phabricator.wikimedia.org/T364204#9855021 (10Raymond_Ndibe) >>! In T364204#9852951, @Multichill wrote: > @Raymond_Ndibe what is the status? I see you merged something. I just tested and the problem still exists. Yes. We are aware that the problem... [14:20:58] 10Cloud-VPS, 13Patch-For-Review: [cloudvps] 2024-05-01 cloudinfra puppetserver got out of space - https://phabricator.wikimedia.org/T366406#9855017 (10dcaro) →14Duplicate dup:03T366357 [14:21:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:22:49] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review, 10Puppet (Puppet 7.0): cloud-vps puppetservers filling up / with puppetserver reports - https://phabricator.wikimedia.org/T366357#9855019 (10dcaro) [14:23:59] 06cloud-services-team: SystemdUnitDown Unit backup_glance_images.service on node cloudbackup1003 has been down for long. - https://phabricator.wikimedia.org/T366416#9855028 (10dcaro) a:03Andrew I saw you silenced the alert, feel free to reassign to me if it's not the same issue [14:25:33] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/72 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [14:25:35] (03update) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/72 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [14:25:37] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/72 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [14:25:59] 10Toolforge (Toolforge iteration 10): [jobs-api] move jobs load feature to the backend - https://phabricator.wikimedia.org/T366209#9855034 (10Raymond_Ndibe) [14:26:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:27:16] 10Toolforge (Toolforge iteration 10): [jobs-api] move jobs load feature to the backend - https://phabricator.wikimedia.org/T366209#9855045 (10Raymond_Ndibe) [14:30:09] (03close) 10dcaro: Draft: donotmerge timeout log [repos/cloud/toolforge/builds-api] (slavina/prefix-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/94 [14:42:21] 10Toolforge (Toolforge iteration 10): toolforge: deal with tools with invalid names - https://phabricator.wikimedia.org/T366477#9855117 (10aborrero) [14:51:13] 10Toolforge: [webservice,toolforge-cli] Make `webservice shell` a standalone tool - https://phabricator.wikimedia.org/T311917#9855169 (10Wurgl) My solution for tool persondata: in ~/.profile I added the following line: ` [ -z "$PERSONDATA_PORT" ] && trap "kubectl delete pods -l 'app.kubernetes.io/component=web... [14:51:17] 10Toolforge: tools-static webserver shows directory listing instead of the index file - https://phabricator.wikimedia.org/T366482 (10Mormegil) 03NEW [15:02:18] 10Toolforge (Toolforge iteration 10): [jobs-api] Save business models in a DB - https://phabricator.wikimedia.org/T359650#9855255 (10Raymond_Ndibe) [15:23:17] 10Toolforge: tools-static webserver shows directory listing instead of the index file - https://phabricator.wikimedia.org/T366482#9855361 (10Mormegil) [15:41:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:46:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:51:41] FIRING: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:53:00] (03approved) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/88 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:53:08] (03update) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/88 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:53:12] (03merge) 10dcaro: poetry: Autoupdate [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/88 (owner: 10group_203_bot_4866fc124f4b41659f667468a6115cf3) [15:53:18] (03update) 10dcaro: nginx: increase read timeout for the logs endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/87 (https://phabricator.wikimedia.org/T359953) [15:54:57] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: jobs-api: bump to 0.0.304-20240603155213-7d1d6fec [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/299 [15:56:41] RESOLVED: [3x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:56:52] (03PS1) 10Muehlenhoff: Remove obsolete stub cert [labs/private] - 10https://gerrit.wikimedia.org/r/1038380 [15:57:45] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [15:58:08] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [16:00:29] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [16:00:39] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [16:01:47] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [16:01:56] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [16:04:52] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [16:05:03] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [16:05:50] 06cloud-services-team, 10Cloud-VPS, 06DC-Ops, 10ops-eqiad, and 2 others: Degraded RAID on cloudcephosd1031 - https://phabricator.wikimedia.org/T364060#9855591 (10Jclark-ctr) 05In progress→03Resolved [16:17:28] (03update) 10sstefanova: cli: use prefixed endpoints [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/71 [16:21:22] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [16:21:31] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [16:23:55] 10Horizon: Show quota utilization for Object Storage in Horizon - https://phabricator.wikimedia.org/T366495#9855736 (10Pppery) [16:26:16] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [16:26:28] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [16:31:08] (03approved) 10dcaro: jobs-api: bump to 0.0.304-20240603155213-7d1d6fec [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/299 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:31:10] (03update) 10dcaro: jobs-api: bump to 0.0.304-20240603155213-7d1d6fec [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/299 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:31:12] (03merge) 10dcaro: jobs-api: bump to 0.0.304-20240603155213-7d1d6fec [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/299 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:55:44] 06cloud-services-team, 10Cloud-VPS, 13Patch-For-Review, 10Puppet (Puppet 7.0): cloud-vps puppetservers filling up / with puppetserver reports - https://phabricator.wikimedia.org/T366357#9855913 (10Andrew) 05Open→03Resolved [17:38:48] 10Tool-schedule-deployment, 10Gerrit: Link to https://schedule-deployment.toolforge.org/backport from changes eligable for deployment in a backport window - https://phabricator.wikimedia.org/T366512 (10bd808) 03NEW [17:44:01] 10Tool-schedule-deployment, 10Gerrit: Link to https://schedule-deployment.toolforge.org/backport/{change-id} from changes eligable for deployment in a backport window - https://phabricator.wikimedia.org/T366512#9856303 (10bd808) [18:25:39] (03PS1) 10Eevans: Faux commons_impact_analytics Cassandra creds [labs/private] - 10https://gerrit.wikimedia.org/r/1038416 (https://phabricator.wikimedia.org/T361835) [18:28:05] (03CR) 10Eevans: [C:03+2] Faux commons_impact_analytics Cassandra creds [labs/private] - 10https://gerrit.wikimedia.org/r/1038416 (https://phabricator.wikimedia.org/T361835) (owner: 10Eevans) [18:28:07] (03CR) 10Eevans: [V:03+2 C:03+2] Faux commons_impact_analytics Cassandra creds [labs/private] - 10https://gerrit.wikimedia.org/r/1038416 (https://phabricator.wikimedia.org/T361835) (owner: 10Eevans) [18:34:07] 10Tool-toolwatch, 06Indic-MediaWiki-Developers: Sort tools based on tool Title - https://phabricator.wikimedia.org/T353579#9856517 (10marrivs) Hi @hks3333, Are you still working on this ticket? [18:38:53] 10Toolforge, 06Project-Admins: Create project tag for toolforge_i18n - https://phabricator.wikimedia.org/T365939#9856528 (10Aklapper) 05Open→03Resolved a:03Aklapper Requested public project #toolforge_i18n has been created: https://phabricator.wikimedia.org/project/view/7218/ (In case you need to ed... [18:45:54] 06cloud-services-team: SystemdUnitDown Unit backup_glance_images.service on node cloudbackup1003 has been down for long. - https://phabricator.wikimedia.org/T366416#9856538 (10Andrew) 05Open→03Resolved This seems to have been transient, we'll see if it returns. [19:11:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:21:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:21:56] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:22:11] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:42:39] 10Toolforge, 06Project-Admins: Create project tag for toolforge_i18n - https://phabricator.wikimedia.org/T365939#9856863 (10LucasWerkmeister) Great, thank you <3 [19:43:02] 10toolforge_i18n, 10Tools, 06translatewiki.net, 10Language-Team (Language-2024-April-June), and 2 others: Make Wikidata Image Positions tool translatable on translatewiki.net - https://phabricator.wikimedia.org/T363626#9856864 (10LucasWerkmeister) [19:43:04] 10toolforge_i18n, 10Tools, 07I18n: Extract Python library for Wikimedia tool i18n from Wikidata Lexeme Forms tool - https://phabricator.wikimedia.org/T283376#9856866 (10LucasWerkmeister) [20:00:48] 06cloud-services-team, 07sre-alert-triage: Alert triage: Adjust severity of backup_cinder_volumes from critical to warning - https://phabricator.wikimedia.org/T342764#9856932 (10Andrew) 05Open→03Declined This hasn't been an issue lately; also digging in code suggests that it's already not a critical wa... [20:40:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:45:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:55:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:23:26] 10Tool-schedule-deployment, 10Gerrit: Link to https://schedule-deployment.toolforge.org/backport/{change-id} from changes eligable for deployment in a backport window - https://phabricator.wikimedia.org/T366512#9857293 (10hashar) That can be added as a simple link below the commit message and the checks summar... [21:56:35] 10Toolforge: Need some help with php - https://phabricator.wikimedia.org/T366543 (10Wurgl) 03NEW [22:11:41] FIRING: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:21:41] RESOLVED: [2x] CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:35:01] 10Toolforge: Need some help with php - https://phabricator.wikimedia.org/T366543#9857462 (10bd808) * `https://viaf.org/viaf/data/viaf-20240602-clusters.xml.gz` is a 20 GB compressed file. * `https://data.dnb.de/opendata/authorities-gnd-kongress_lds.rdf.gz` is an 88 MB compressed file. I would expect the viaf.or... [22:44:28] 10Toolforge: Need some help with php - https://phabricator.wikimedia.org/T366543#9857472 (10bd808) >>! In T366543#9857462, @bd808 wrote: > I am having a hard time finding definite proof that php's `fopen()` will stream the `https://` + `compress.zlib://` response body to you by lines without first waiting for th... [22:56:33] 10Toolforge: Need some help with php - https://phabricator.wikimedia.org/T366543#9857514 (10Wurgl) Oh my god! I have written some code in C with the gzip-library and there was no such delay. I really did not expect such a behaviour. But it seems you are right, compress.zlib://https://data.dnb.de/opendata/author... [23:42:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:53:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-35 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [23:57:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:58:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-35 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses