[00:05:55] FIRING: MaxConntrack: Max conntrack at 80.2% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:10:55] RESOLVED: MaxConntrack: Max conntrack at 82.9% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:11:25] FIRING: MaxConntrack: Max conntrack at 82.97% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:11:35] chicocvenancio opened https://github.com/toolforge/paws/pull/439 [00:14:03] 10PAWS: Update labpawspublic extension to jupyterlab 4 system - https://phabricator.wikimedia.org/T358604#9967763 (10Chicocvenancio) a:03Chicocvenancio @Isaac mentioned paws public links are not working in irc/telegram, this should get the links back. [00:16:10] RESOLVED: MaxConntrack: Max conntrack at 80.07% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:33:57] chicocvenancio closed https://github.com/toolforge/paws/pull/439 [00:42:25] FIRING: MaxConntrack: Max conntrack at 80.89% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [00:57:25] RESOLVED: MaxConntrack: Max conntrack at 80.96% on cloudvirt1040:9100 - https://wikitech.wikimedia.org/wiki/Monitoring/check_conntrack - https://grafana.wikimedia.org/d/oITUqwKIk/netfilter-connection-tracking - https://alerts.wikimedia.org/?q=alertname%3DMaxConntrack [01:03:53] (03open) 10raymond-ndibe: [jobs-api] custom resource definitions deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) [01:04:14] (03update) 10raymond-ndibe: [jobs-api] custom resource definitions deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) [01:04:39] (03update) 10raymond-ndibe: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) [01:05:07] (03update) 10raymond-ndibe: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) [01:10:21] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 10Data-Services, 05Goal: Improve trove backup/restore - https://phabricator.wikimedia.org/T356291#9967930 (10Andrew) Update about this: I'm working on getting 'openstack database instance rebuild' working for all our cases (T355721); if that succeeds... [01:11:39] (03update) 10raymond-ndibe: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) [01:27:33] (03update) 10raymond-ndibe: [jobs-api] custom resource definition deployment templates [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T359650) [02:19:56] FIRING: CloudVPSDesignateLeaks: Detected 25 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:04:41] RESOLVED: CloudVPSDesignateLeaks: Detected 24 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:10:19] (03PS2) 10Pppery: ISA: New footer not included in translation message [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1053071 (https://phabricator.wikimedia.org/T340538) (owner: 10Aditya0545) [04:10:36] (03PS2) 10Pppery: ISA: Typo in Top contributor campaign page [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1053078 (https://phabricator.wikimedia.org/T358396) (owner: 10Aditya0545) [04:11:41] (03CR) 10Pppery: "recheck" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1053047 (owner: 10GauriGuptaa) [04:12:03] (03CR) 10Pppery: "recheck" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1053046 (https://phabricator.wikimedia.org/T358396) (owner: 10Hridyesh_Gupta) [04:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:04:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:31:04] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: review k8s API usage by custom components for 1.25 upgrade - https://phabricator.wikimedia.org/T369164#9968066 (10Slst2020) [05:47:46] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: review k8s API usage by custom components for 1.25 upgrade - https://phabricator.wikimedia.org/T369164#9968083 (10Slst2020) @aborrero should this be removed now that PSP is gone? ` [~/repos/work/toolforge/envvars-admission (main)] % cat d... [06:50:44] (03approved) 10arnaudb: task_template*: Replace dborch command [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/6 (owner: 10marostegui) [06:58:41] (03update) 10dcaro: auth: remove ssl header auth and use x-toolforge-user [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T367181) [06:59:44] (03Abandoned) 10Aklapper: Set description when creating Phabricator projects [labs/striker] - 10https://gerrit.wikimedia.org/r/987145 (https://phabricator.wikimedia.org/T344610) (owner: 10Aklapper) [07:00:28] (03update) 10dcaro: remove auth [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/102 (https://phabricator.wikimedia.org/T367181) [07:33:23] 10Tool-spacemedia, 10Server-side-upload-request: Server-side upload request for OptimusPrimeBot (INPE DPI) - https://phabricator.wikimedia.org/T366353#9968183 (10Bawolff) > I am sorry. I didn't want to blame the infra In fairness, the infra is often broken :) Keep in mind that wikimedia uses the followin... [08:09:28] FIRING: InstanceDown: Project metricsinfra instance metricsinfra-thanos-fe-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [08:11:33] (03CR) 10Hridyesh_Gupta: "I have added this batch because the last updated patch was submitted 4-5 months ago and had a merge conflict. Please review my patch and p" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1053046 (https://phabricator.wikimedia.org/T358396) (owner: 10Hridyesh_Gupta) [08:29:25] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: review k8s API usage by custom components for 1.25 upgrade - https://phabricator.wikimedia.org/T369164#9968243 (10aborrero) >>! In T369164#9968083, @Slst2020 wrote: > @aborrero should this be removed now that PSP is gone? > > ` > [~/repos... [08:44:28] RESOLVED: InstanceDown: Project metricsinfra instance metricsinfra-thanos-fe-1 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [08:45:26] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: [jobs-api] Save business models in a DB - https://phabricator.wikimedia.org/T359650#9968272 (10aborrero) >>! In T359650#9967925, @CodeReviewBot wrote: > raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/10... [09:07:33] 10Cloud-VPS (Debian Buster Deprecation), 10Wikispore: Rebuild Wikispore Vagrant boxes on Bullseye or Bookworm - https://phabricator.wikimedia.org/T365934#9968311 (10Tgr) A possible approach to meet the Jul 15 deadline is to build new Bullseye Cloud VPS boxes, and copy the existing Vagrant boxes over as they ar... [09:09:39] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: [jobs-api] Save business models in a DB - https://phabricator.wikimedia.org/T359650#9968309 (10dcaro) >>! In T359650#9968272, @aborrero wrote: >>>! In T359650#9967925, @CodeReviewBot wrote: >> raymond-ndibe opened https://gitlab.wikimedia.org/repos/clou... [09:13:08] 06cloud-services-team, 10Toolforge (Toolforge iteration 12): toolforge: review k8s API usage by custom components for 1.25 upgrade - https://phabricator.wikimedia.org/T369164#9968337 (10Slst2020) >>! In T369164#9968243, @aborrero wrote: > yes, I missed that one as part of {T368142} are you on it or should I g... [09:13:13] 10Toolforge (Toolforge iteration 12): [toolforge] simplify calling the different toolforge apis from within the containers - https://phabricator.wikimedia.org/T356377#9968339 (10dcaro) [09:14:20] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 06Traffic: Replace or remove Debian Buster VMs in 'traffic' cloud-vps project - https://phabricator.wikimedia.org/T360710#9968342 (10Vgutierrez) Actionable right now: * `traffic-cache-atstext-buster` can be wiped. Things that need to be fixed:... [09:14:24] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "mediawiki-vagrant" project Buster deprecation - https://phabricator.wikimedia.org/T367541#9968343 (10Tgr) I'll take a look at the box over the weekend and see if I can figure out how it works and document it for posteriority, but in any case I think it's fine t... [09:22:45] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 06Traffic: Replace or remove Debian Buster VMs in 'traffic' cloud-vps project - https://phabricator.wikimedia.org/T360710#9968417 (10Vgutierrez) and something is definitely wrong with other hosts using the new puppetmaster: ` vgutierrez@traffic-... [09:28:57] 10Toolforge (Toolforge iteration 12): [toolforge, toolforge-cli] Experiment with PyInstaller to package CLI tools for buildpack images - https://phabricator.wikimedia.org/T369693 (10Slst2020) 03NEW [09:29:05] 10Toolforge (Toolforge iteration 12): [toolforge, toolforge-cli] Experiment with PyInstaller to package CLI tools for buildpack images - https://phabricator.wikimedia.org/T369693#9968446 (10Slst2020) a:05dcaro→03None [09:44:38] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "mwv-apt" project Buster deprecation - https://phabricator.wikimedia.org/T367542#9968492 (10Tgr) [09:47:05] 10Toolforge (Toolforge iteration 12): [toolforge, toolforge-cli] Experiment with PyInstaller to package CLI tools for buildpack images - https://phabricator.wikimedia.org/T369693#9968497 (10Slst2020) [09:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [10:01:23] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api [10:01:34] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api [10:02:57] 10Cloud-VPS (Debian Buster Deprecation): Cloud VPS "mwv-apt" project Buster deprecation - https://phabricator.wikimedia.org/T367542#9968512 (10Tgr) Specifically, used here: [[https://gerrit.wikimedia.org/g/mediawiki/vagrant/+/2953fc8de2e315801f1d448059ac096f6a13b539/puppet/modules/apt/manifests/init.pp#84|puppet... [10:05:55] (03approved) 10dcaro: Cache ldap connection and config loading [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/2 [10:05:58] (03merge) 10dcaro: Cache ldap connection and config loading [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/2 [10:06:20] (03approved) 10dcaro: Add logrotate [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/11 (https://phabricator.wikimedia.org/T369601) [10:06:24] (03update) 10dcaro: Add logrotate [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/11 (https://phabricator.wikimedia.org/T369601) [10:06:45] (03merge) 10dcaro: Add logrotate [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/11 (https://phabricator.wikimedia.org/T369601) [10:08:31] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: [fourohfour,infra] Got a lot of alerts this weekend - https://phabricator.wikimedia.org/T369601#9968533 (10dcaro) Seems to be going smoother, will open a new task if it happens again [10:09:24] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: [builds-api,envvars-api,jobs-api] bump the version in the openapi definition when bumping the package version - https://phabricator.wikimedia.org/T356972#9968541 (10dcaro) [10:09:42] (03approved) 10dcaro: pre-commit: add openapi version bump check [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T356972) [10:09:52] (03merge) 10dcaro: pre-commit: add openapi version bump check [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T356972) [10:09:55] (03update) 10dcaro: pre-commit: add openapi version bump check [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T356972) [10:10:31] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api [10:10:42] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api [10:11:27] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: [fourohfour,infra] Got a lot of alerts this weekend - https://phabricator.wikimedia.org/T369601#9968535 (10dcaro) 05Open→03Resolved [10:11:37] (03approved) 10dcaro: envvars-api: bump to 0.0.52-20240705101149-aa9da2fa [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/391 (https://phabricator.wikimedia.org/T356972 https://phabricator.wikimedia.org/T368142) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:11:45] (03merge) 10dcaro: envvars-api: bump to 0.0.52-20240705101149-aa9da2fa [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/391 (https://phabricator.wikimedia.org/T356972 https://phabricator.wikimedia.org/T368142) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [10:12:10] (03update) 10sstefanova: api: remove unprefixed endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/94 (https://phabricator.wikimedia.org/T363346) [10:13:48] (03approved) 10dcaro: [builds-cli] bug fix [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/79 (owner: 10raymond-ndibe) [10:14:18] (03update) 10dcaro: [builds-cli] bug fix [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/79 (owner: 10raymond-ndibe) [10:49:32] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 06Traffic: Replace or remove Debian Buster VMs in 'traffic' cloud-vps project - https://phabricator.wikimedia.org/T360710#9968683 (10Vgutierrez) regarding traffic-cache instances: * I've tried to unify their puppet configuration in 3 prefixes:... [10:50:49] (03approved) 10ladsgroup: task_template*: Replace dborch command [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/6 (owner: 10marostegui) [10:50:54] (03update) 10ladsgroup: task_template*: Replace dborch command [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/6 (owner: 10marostegui) [10:51:17] (03merge) 10ladsgroup: task_template*: Replace dborch command [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/6 (owner: 10marostegui) [11:48:43] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: [jobs-api, jobs-cli] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363346#9968876 (10Slst2020) 05In progress→03Stalled [11:49:35] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: [jobs-api, jobs-cli] Prefix all endpoints with `/tool/` - https://phabricator.wikimedia.org/T363346#9968897 (10Slst2020) will merge the last patch when the old endpoints have been officially deprecated (Thu 18 July) [12:04:52] (03update) 10dcaro: remove auth [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/102 [12:05:38] 10Toolforge (Toolforge iteration 12): [toolforge, toolforge-cli] Experiment with PyInstaller to package CLI tools for buildpack images - https://phabricator.wikimedia.org/T369693#9968955 (10aborrero) > * Requires rebuilding images when CLI tools are updated This reminded me, this was precisely one of the main p... [12:08:18] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: [jobs-api] Save business models in a DB - https://phabricator.wikimedia.org/T359650#9968965 (10aborrero) ok! thanks. [12:08:54] (03open) 10dcaro: pre-commit: allow setting memory-optimized runners [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/44 [12:18:01] (03update) 10dcaro: pre-commit: allow setting memory-optimized runners [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/44 [12:23:11] (03update) 10aborrero: kubecerts: have certificates lifetime to be max 10 days, renew them often [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/55 (https://phabricator.wikimedia.org/T365681) [12:26:06] (03update) 10aborrero: kubecerts: have certificates lifetime to be max 10 days, renew them often [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/55 (https://phabricator.wikimedia.org/T365681) [12:34:20] (03approved) 10aborrero: pre-commit: allow setting memory-optimized runners [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/44 (owner: 10dcaro) [12:34:50] (03update) 10dcaro: pre-commit: allow setting memory-optimized runners [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/44 [12:35:03] (03update) 10dcaro: pre-commit: allow setting memory-optimized runners [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/44 [12:36:53] (03update) 10aborrero: kubecerts: have certificates lifetime to be max 10 days, renew them often [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/55 (https://phabricator.wikimedia.org/T365681) [12:39:31] (03approved) 10dcaro: kubecerts: have certificates lifetime to be max 10 days, renew them often [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/55 (https://phabricator.wikimedia.org/T365681) (owner: 10aborrero) [12:40:17] (03merge) 10aborrero: kubecerts: have certificates lifetime to be max 10 days, renew them often [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/55 (https://phabricator.wikimedia.org/T365681) [12:42:27] (03update) 10dcaro: pre-commit: allow setting memory-optimized runners [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/44 [12:42:44] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.163-20240710124028-b9245afd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/402 (https://phabricator.wikimedia.org/T365681) [12:44:26] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [12:44:36] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [12:44:48] (03update) 10dcaro: pre-commit: allow setting memory-optimized runners [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/44 [12:44:51] (03merge) 10dcaro: pre-commit: allow setting memory-optimized runners [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/44 [12:45:19] (03update) 10dcaro: remove auth [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/102 [12:45:48] (03update) 10dcaro: remove auth [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/102 (https://phabricator.wikimedia.org/T367181) [12:47:03] (03open) 10dcaro: ci: use memory-optimized runners [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/103 [12:47:08] (03update) 10dcaro: ci: use memory-optimized runners [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/103 [12:52:13] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [12:52:24] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [12:53:38] (03merge) 10aborrero: maintain-kubeusers: bump to 0.0.163-20240710124028-b9245afd [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/402 (https://phabricator.wikimedia.org/T365681) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [12:54:18] (03open) 10dcaro: ci: use memory-optimized runners [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/39 [12:56:32] (03approved) 10aborrero: ci: use memory-optimized runners [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/103 (owner: 10dcaro) [12:57:25] (03merge) 10dcaro: ci: use memory-optimized runners [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/103 [12:57:37] (03merge) 10dcaro: ci: use memory-optimized runners [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/39 [13:01:52] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: envvars-api: bump to 0.0.54-20240710125749-a758c8d7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/403 [13:01:55] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: envvars-api: bump to 0.0.54-20240710125749-a758c8d7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/403 [13:02:42] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: builds-api: bump to 0.0.161-20240710125734-9d017eeb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/404 (https://phabricator.wikimedia.org/T356972) [13:26:27] 10cloud-services-team (FY2023/2024-Q3-Q4), 06DC-Ops, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, and 2 others: [ceph] Getting rack level HA - https://phabricator.wikimedia.org/T297083#9969270 (10dcaro) I think this is done already, we have: * the three mons in different racks {F563... [13:28:11] 10cloud-services-team (FY2023/2024-Q3-Q4), 06DC-Ops, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, and 2 others: [ceph] Getting rack level HA - https://phabricator.wikimedia.org/T297083#9969280 (10dcaro) 05Open→03Resolved Will reopen a specific task if needed if we end up getting... [13:45:07] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#9969343 (10dcaro) [13:50:59] 10cloud-services-team (Hardware), 06DC-Ops, 10ops-eqiad, 06SRE: Q4:rack/setup/install new cloudcephmon hosts - https://phabricator.wikimedia.org/T364870#9969363 (10dcaro) >>! In T364870#9942790, @Jclark-ctr wrote: > @Andrew @dcaro thank you for providing update did you have host names for this and please... [13:54:25] 06cloud-services-team: cloudceph: reimage some servers - https://phabricator.wikimedia.org/T328011#9969369 (10dcaro) 05In progress→03Resolved Done [13:54:32] FIRING: ToolsToolsDBWritableState: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [13:59:32] RESOLVED: ToolsToolsDBWritableState: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [14:05:05] (03update) 10sstefanova: consolidate prefixes [repos/cloud/toolforge/jobs-api] (slavina/remove-unprefixed-endpoints) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/95 (https://phabricator.wikimedia.org/T363346) [14:13:07] 06cloud-services-team, 05Cloud-Services-Origin-Alert, 07Cloud-Services-Worktype-Unplanned, 10Sustainability (Incident Followup): [toolsdb] 2024-07-10 ToolsToolsDBWritableState alert page - prometheus-mysql-exporter down - https://phabricator.wikimedia.org/T369722#9969489 (10dcaro) a:03dcaro [14:13:53] 06cloud-services-team, 10Cloud-VPS, 10Data-Services, 05Goal: Update all trove VMs to a modern guest image - https://phabricator.wikimedia.org/T369723 (10Andrew) 03NEW [14:17:56] 06cloud-services-team, 05Cloud-Services-Origin-Alert, 07Cloud-Services-Worktype-Unplanned, 10Sustainability (Incident Followup): [toolsdb] 2024-07-10 ToolsToolsDBWritableState alert page - prometheus-mysql-exporter down - https://phabricator.wikimedia.org/T369722#9969520 (10dcaro) It seems a new change add... [14:18:22] 06cloud-services-team, 05Cloud-Services-Origin-Alert, 07Cloud-Services-Worktype-Unplanned, 10Sustainability (Incident Followup): [toolsdb] 2024-07-10 ToolsToolsDBWritableState alert page - prometheus-mysql-exporter down - https://phabricator.wikimedia.org/T369722#9969522 (10dcaro) p:05Triage→03High [14:27:04] 06cloud-services-team, 05Cloud-Services-Origin-Alert, 07Cloud-Services-Worktype-Unplanned, 10Sustainability (Incident Followup): [toolsdb] 2024-07-10 ToolsToolsDBWritableState alert page - prometheus-mysql-exporter down - https://phabricator.wikimedia.org/T369722#9969560 (10dcaro) We need at least exporter... [14:28:31] (03open) 10sstefanova: openapi: consolidate metrics and healthz endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/102 (https://phabricator.wikimedia.org/T365014) [14:31:58] 06cloud-services-team, 05Cloud-Services-Origin-Alert, 07Cloud-Services-Worktype-Unplanned, 10Sustainability (Incident Followup): [toolsdb] 2024-07-10 ToolsToolsDBWritableState alert page - prometheus-mysql-exporter down - https://phabricator.wikimedia.org/T369722#9969579 (10dcaro) For the alert not alertin... [14:37:56] (03open) 10aborrero: kubecerts: only randomly select certs for renew in case they are old [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/57 (https://phabricator.wikimedia.org/T365681) [14:40:01] 10Toolforge (Toolforge iteration 12): [toolforge, toolforge-cli] Experiment with PyInstaller to package CLI tools for buildpack images - https://phabricator.wikimedia.org/T369693#9969631 (10Slst2020) >>! In T369693#9968955, @aborrero wrote: >> * Requires rebuilding images when CLI tools are updated > > This rem... [14:43:35] (03update) 10sstefanova: openapi: consolidate metrics and healthz endpoints [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/102 (https://phabricator.wikimedia.org/T365014) [14:45:30] (03update) 10aborrero: kubecerts: only randomly select certs for renew in case they are old [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/57 (https://phabricator.wikimedia.org/T365681) [15:10:40] (03merge) 10aborrero: kubecerts: only randomly select certs for renew in case they are old [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/57 (https://phabricator.wikimedia.org/T365681) [15:13:06] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.164-20240710151051-a518392e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/405 (https://phabricator.wikimedia.org/T365681) [15:15:03] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [15:15:12] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [15:15:20] (03PS3) 10Pppery: Fixed the typo in Top contributor campaign page [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1053046 (https://phabricator.wikimedia.org/T358396) (owner: 10Hridyesh_Gupta) [15:15:57] (03CR) 10Pppery: "Looks fine to me I guess but I don't have permission to merge changes to this repository. (After updating the commit message to not have a" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1053046 (https://phabricator.wikimedia.org/T358396) (owner: 10Hridyesh_Gupta) [15:16:26] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [15:16:37] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [15:18:25] 10Toolforge: `webservice` requires effective user to be the tool user and listed in NSS passwd data - https://phabricator.wikimedia.org/T369569#9969879 (10bd808) [15:18:32] 10Toolforge (Toolforge iteration 12): [toolforge, toolforge-cli] Experiment with PyInstaller to package CLI tools for buildpack images - https://phabricator.wikimedia.org/T369693#9969880 (10bd808) [15:18:38] 10Toolforge: `toolforge jobs` requires current user to be the tool user and listed in NSS passwd data - https://phabricator.wikimedia.org/T369573#9969881 (10bd808) [15:18:46] 10Toolforge (Toolforge iteration 12): [toolforge, toolforge-cli] Experiment with PyInstaller to package CLI tools for buildpack images - https://phabricator.wikimedia.org/T369693#9969882 (10bd808) [15:18:52] (03merge) 10aborrero: maintain-kubeusers: bump to 0.0.164-20240710151051-a518392e [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/405 (https://phabricator.wikimedia.org/T365681) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [15:21:08] 10Toolforge (Toolforge iteration 12): [toolforge, toolforge-cli] Experiment with PyInstaller to package CLI tools for buildpack images - https://phabricator.wikimedia.org/T369693#9969898 (10aborrero) >>! In T369693#9969631, @Slst2020 wrote: > > What alternatives are there? Is there any option right now other th... [15:25:05] 10Toolforge (Toolforge iteration 12): [toolforge, toolforge-cli] Experiment with PyInstaller to package CLI tools for buildpack images - https://phabricator.wikimedia.org/T369693#9969935 (10bd808) >>! In T369693#9969898, @aborrero wrote: > I think previously the narrative was: use the raw API directly. If the A... [15:28:37] 10Toolforge (Toolforge iteration 12): [toolforge, toolforge-cli] Experiment with PyInstaller to package CLI tools for buildpack images - https://phabricator.wikimedia.org/T369693#9969957 (10aborrero) >>! In T369693#9969935, @bd808 wrote: >> Strict API deprecation policies and backwards compatibility shims are po... [15:40:49] 10Toolforge (Toolforge iteration 12): [toolforge, toolforge-cli] Experiment with PyInstaller to package CLI tools for buildpack images - https://phabricator.wikimedia.org/T369693#9970050 (10dcaro) >>! In T369693#9969957, @aborrero wrote: >>>! In T369693#9969935, @bd808 wrote: >>> Strict API deprecation policies... [15:41:11] 06cloud-services-team, 06DC-Ops, 10ops-codfw, 06SRE: Test new hardware candidate for cloudbackup replacement - https://phabricator.wikimedia.org/T353746#9970063 (10Jhancock.wm) a:05Jhancock.wm→03None [15:43:29] 10Toolforge: `toolforge jobs` requires current user to be the tool user and listed in NSS passwd data - https://phabricator.wikimedia.org/T369573#9970076 (10dcaro) p:05Triage→03High [15:43:34] 10Toolforge (Toolforge iteration 12): [toolforge, toolforge-cli] Experiment with PyInstaller to package CLI tools for buildpack images - https://phabricator.wikimedia.org/T369693#9970079 (10dcaro) p:05Triage→03High [15:47:25] (03CR) 10Hridyesh_Gupta: "Okay thanks, so who will be merging this commit in the repository, please guide!" [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1053046 (https://phabricator.wikimedia.org/T358396) (owner: 10Hridyesh_Gupta) [15:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [15:52:05] (03update) 10aborrero: registryadmission: allow to exclude namespaces from checks [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/9 (https://phabricator.wikimedia.org/T369527) [15:53:26] (03update) 10aborrero: registryadmission: allow to exclude namespaces from checks [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/9 (https://phabricator.wikimedia.org/T369527) [15:54:15] (03merge) 10aborrero: registryadmission: allow to exclude namespaces from checks [repos/cloud/toolforge/registry-admission] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/registry-admission/-/merge_requests/9 (https://phabricator.wikimedia.org/T369527) [15:57:05] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: registry-admission: bump to 0.0.46-20240710155426-ae29a6f9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/406 (https://phabricator.wikimedia.org/T369527) [15:58:09] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission [15:58:19] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission [15:59:41] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:01:30] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission [16:01:40] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission [16:03:17] (03merge) 10aborrero: registry-admission: bump to 0.0.46-20240710155426-ae29a6f9 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/406 (https://phabricator.wikimedia.org/T369527) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:17:39] 06cloud-services-team, 05Cloud-Services-Origin-Alert, 07Cloud-Services-Worktype-Unplanned, 13Patch-For-Review, 10Sustainability (Incident Followup): [toolsdb] 2024-07-10 ToolsToolsDBWritableState alert page - prometheus-mysql-exporter down - https://phabricator.wikimedia.org/T369722#9970273 (10dcaro) The... [16:17:48] 06cloud-services-team, 05Cloud-Services-Origin-Alert, 07Cloud-Services-Worktype-Unplanned, 13Patch-For-Review, 10Sustainability (Incident Followup): [toolsdb] 2024-07-10 ToolsToolsDBWritableState alert page - prometheus-mysql-exporter down - https://phabricator.wikimedia.org/T369722#9970275 (10dcaro)... [16:19:55] (03update) 10dcaro: envvars-api: bump to 0.0.54-20240710125749-a758c8d7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/403 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [16:45:40] (03CR) 10Agamyasamuel: "It will be Merged, if its approved by the Maintainer who has access to this repo." [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1053046 (https://phabricator.wikimedia.org/T358396) (owner: 10Hridyesh_Gupta) [16:46:57] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api [16:47:07] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api [16:49:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:52:59] 10Tool-spacemedia, 10Server-side-upload-request: Server-side upload request for OptimusPrimeBot (INPE DPI) - https://phabricator.wikimedia.org/T366353#9970512 (10Don-vip) I found a way to solve the issue :) I removed the problematic Photosphop-specific TIFF tag 37724 called ImageSourceData with exiftool:... [16:57:42] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component envvars-api [16:57:53] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component envvars-api [17:04:41] (03update) 10dcaro: envvars-api: bump to 0.0.54-20240710125749-a758c8d7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/403 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [17:04:43] (03merge) 10dcaro: envvars-api: bump to 0.0.54-20240710125749-a758c8d7 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/403 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [17:04:53] (03update) 10dcaro: builds-api: bump to 0.0.161-20240710125734-9d017eeb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/404 (https://phabricator.wikimedia.org/T356972) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [17:05:11] !log dcaro@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [17:05:21] !log dcaro@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [17:09:34] !log dcaro@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api [17:09:46] !log dcaro@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api [17:22:31] (03update) 10dcaro: builds-api: bump to 0.0.161-20240710125734-9d017eeb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/404 (https://phabricator.wikimedia.org/T356972) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [17:22:33] (03approved) 10dcaro: builds-api: bump to 0.0.161-20240710125734-9d017eeb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/404 (https://phabricator.wikimedia.org/T356972) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [17:22:39] (03merge) 10dcaro: builds-api: bump to 0.0.161-20240710125734-9d017eeb [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/404 (https://phabricator.wikimedia.org/T356972) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [17:22:43] 10Tool-dabfix, 06Indic MediaWiki Developers UG, 06Indic-TechCom: Fix Text Overflow - https://phabricator.wikimedia.org/T369380#9970637 (10Jayprakash12345) 05Open→03Resolved [17:23:26] 10Tool-dabfix, 06Indic MediaWiki Developers UG, 06Indic-TechCom: Improve Footer Styling and Fix Position - https://phabricator.wikimedia.org/T369379#9970642 (10Jayprakash12345) 05Open→03Resolved [17:32:36] 10Toolforge (Toolforge iteration 12): `webservice` requires effective user to be the tool user and listed in NSS passwd data - https://phabricator.wikimedia.org/T369569#9970682 (10dcaro) p:05Triage→03High a:03dcaro [17:32:48] (03open) 10dcaro: cli: get the toolname from the k8s cert [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/49 (https://phabricator.wikimedia.org/T369569) [17:33:13] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: `webservice` requires effective user to be the tool user and listed in NSS passwd data - https://phabricator.wikimedia.org/T369569#9970689 (10dcaro) 05Open→03In progress [17:34:22] (03update) 10dcaro: cli: get the toolname from the k8s cert [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/49 (https://phabricator.wikimedia.org/T369569) [17:36:00] (03update) 10dcaro: cli: get the toolname from the k8s cert [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/49 (https://phabricator.wikimedia.org/T369569) [17:39:34] (03update) 10dcaro: cli: get the toolname from the k8s cert [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/49 (https://phabricator.wikimedia.org/T369569) [17:49:55] (03update) 10dcaro: cli: get the toolname from the k8s cert [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/49 (https://phabricator.wikimedia.org/T369569) [18:15:53] 06cloud-services-team, 10Cloud-VPS, 10Data-Services, 05Goal: Update all trove VMs to a modern guest image - https://phabricator.wikimedia.org/T369723#9970815 (10Andrew) [18:48:57] FIRING: HarborComponentDown: A Harbor component is down. #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborComponentDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborComponentDown [18:58:57] RESOLVED: HarborComponentDown: A Harbor component is down. #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborComponentDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborComponentDown [19:13:57] FIRING: HarborComponentDown: A Harbor component is down. #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborComponentDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborComponentDown [19:23:57] RESOLVED: HarborComponentDown: A Harbor component is down. #page - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/HarborComponentDown - https://prometheus-alerts.wmcloud.org/?q=alertname%3DHarborComponentDown [20:00:13] 10VPS-project-Codesearch, 06collaboration-services: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199#9971232 (10bd808) >>! In T268199#9967699, @Ladsgroup wrote: > Not to mention that converting [[https://gerrit.wikimedia.org/r/plugins/gitiles/labs/codesearch/+/refs/heads/master/fro... [20:49:56] FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [20:58:02] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 06Research: Replace or remove Debian Buster VMs in 'wmf-research-tools' cloud-vps project - https://phabricator.wikimedia.org/T367444#9971389 (10Isaac) 05Open→03Resolved The other three have been migrated over / deprecated now too (https... [22:50:03] FIRING: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-9 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [23:00:03] RESOLVED: ToolforgeKubernetesWorkerTooManyDProcesses: Kubernetes worker tools-k8s-worker-nfs-9 has many processes stuck on IO (probably NFS) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolforgeKubernetesWorkerTooManyDProcesses - https://grafana.wmcloud.org/d/3jhWxB8Vk/toolforge-general-overview - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolforgeKubernetesWorkerTooManyDProcesses [23:19:59] (03update) 10bd808: Make image useful for Brad [toolforge-repos/bd808-buildpack-perl-bastion] - 10https://gitlab.wikimedia.org/toolforge-repos/bd808-buildpack-perl-bastion/-/merge_requests/1 [23:41:56] (03update) 10bd808: Make image useful for Brad [toolforge-repos/bd808-buildpack-perl-bastion] - 10https://gitlab.wikimedia.org/toolforge-repos/bd808-buildpack-perl-bastion/-/merge_requests/1