[00:05:22] 10Tool-yearinreview, 07good first task: Please attribute the original, add disclaimer and add the LICENSE - https://phabricator.wikimedia.org/T366114#9963718 (10theprotonade) Hi @Jdlrobson, I agree with you completely. We'll get down to work on this as a top priority and get it closed in the next few days. Hop... [00:11:25] 10Tool-yearinreview, 07good first task: Please attribute the original, add disclaimer and add the LICENSE - https://phabricator.wikimedia.org/T366114#9963733 (10theprotonade) a:03theprotonade [00:31:53] 10Tool-spacemedia, 10Server-side-upload-request: Server-side upload request for OptimusPrimeBot (INPE DPI) - https://phabricator.wikimedia.org/T366353#9963748 (10Bawolff) Lack of thumbnails is probably because mediawiki thinks the dimension of the image is 0x0. MediaWiki reports an error of "no page data f... [01:33:57] 10Tool-spacemedia, 10Server-side-upload-request: Server-side upload request for OptimusPrimeBot (INPE DPI) - https://phabricator.wikimedia.org/T366353#9963787 (10Don-vip) Tried Brasilia 20170312a0416 Mediana B151413.tif tiffinfo is able to read it: ` === TIFF directory 0 === TIFF Directory at offset 0xcd... [01:50:42] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:00:42] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [02:18:39] 10Tool-spacemedia, 10Server-side-upload-request: Server-side upload request for OptimusPrimeBot (INPE DPI) - https://phabricator.wikimedia.org/T366353#9963830 (10Bawolff) If MediaWiki can't determine the dimensions of the file, it will fail before thumbor gets involved [03:19:42] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:29:42] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:49:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [04:59:42] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:08:28] (03CR) 10Abijeet Patro: [V:03+2] Localisation updates from https://translatewiki.net. [labs/tools/massmailer] - 10https://gerrit.wikimedia.org/r/1052725 (owner: 10L10n-bot) [06:44:08] 10Toolforge: Regression dev.toolforge.org is missing a C compiler - https://phabricator.wikimedia.org/T369408#9964038 (10Count_Count) @bd808 I remember now. Thanks for reminding me. [06:58:31] (03PS11) 10David Caro: toolsdb: load the inventory dynamically [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/992932 [06:59:56] (03CR) 10CI reject: [V:04-1] toolsdb: load the inventory dynamically [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/992932 (owner: 10David Caro) [07:06:55] (03PS12) 10David Caro: toolsdb: load the inventory dynamically [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/992932 [07:24:47] (03CR) 10David Caro: [C:03+2] toolsdb: load the inventory dynamically [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/992932 (owner: 10David Caro) [07:28:10] (03Merged) 10jenkins-bot: toolsdb: load the inventory dynamically [cloud/wmcs-cookbooks] - 10https://gerrit.wikimedia.org/r/992932 (owner: 10David Caro) [07:43:04] (03update) 10dcaro: ci: add check-openapi-version-bump pre-commit hook [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T356972) [08:03:07] (03open) 10dcaro: check_openapi_version: ensure that the new version is greater [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/43 [08:04:33] (03update) 10dcaro: check_openapi_version: ensure that the new version is greater [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/43 [08:30:16] (03update) 10dcaro: ci: add check-openapi-version-bump pre-commit hook [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T356972) [08:30:33] (03update) 10dcaro: ci: add check-openapi-version-bump pre-commit hook [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T356972) [08:38:12] 10Toolforge (Toolforge iteration 12): [envvars-api] Remove authentication and use api-gateway provided headers - https://phabricator.wikimedia.org/T367181#9964362 (10dcaro) 05Open→03In progress [08:47:15] (03open) 10dcaro: auth: remove ssl header auth and use x-toolforge-user [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T367181) [08:49:42] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:51:11] (03update) 10dcaro: Cache ldap connection and config loading [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/2 [08:51:26] 10Toolforge (Toolforge iteration 12): [fourohfour,infra] Got a lot of alerts this weekend - https://phabricator.wikimedia.org/T369601#9964404 (10dcaro) p:05Triage→03High [08:56:30] 10Toolforge (Toolforge iteration 12): [fourohfour,infra] Got a lot of alerts this weekend - https://phabricator.wikimedia.org/T369601#9964413 (10dcaro) Found that the logs were growing too much (>300GB in tools), so added a logrotate job: ` tools.fourohfour@tools-bastion-13:~$ toolforge jobs list -o long +------... [08:57:57] 10Toolforge (Toolforge iteration 12): [fourohfour,infra] Got a lot of alerts this weekend - https://phabricator.wikimedia.org/T369601#9964439 (10dcaro) [08:58:29] 10Toolforge (Toolforge iteration 12): [fourohfour,infra] Got a lot of alerts this weekend - https://phabricator.wikimedia.org/T369601#9964434 (10dcaro) The traffic in the last two days does not seem suspicious, though there was one worker (nfs-37) that got nfs issues, so might have affected the tool too: {F5630... [08:59:42] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:59:56] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:00:11] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [09:13:40] 10Toolforge (Toolforge iteration 12): [fourohfour,infra] Got a lot of alerts this weekend - https://phabricator.wikimedia.org/T369601#9964507 (10dcaro) Looking closer at the logs, there's a lot of logs like: ` [WARNING] unable to add HTTP_ACCEPT_ENCODING=gzip, deflate, br to uwsgi packet, consider increasing buf... [09:20:39] (03update) 10dcaro: Cache ldap connection and config loading [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/2 [09:27:18] (03open) 10dcaro: Add logrotate [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/11 (https://phabricator.wikimedia.org/T369601) [09:27:19] (03update) 10dcaro: Add logrotate [toolforge-repos/fourohfour] - 10https://gitlab.wikimedia.org/toolforge-repos/fourohfour/-/merge_requests/11 (https://phabricator.wikimedia.org/T369601) [09:31:30] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: [fourohfour,infra] Got a lot of alerts this weekend - https://phabricator.wikimedia.org/T369601#9964578 (10dcaro) Changed the uwsgi config, let's see the log growth in an hour (the 10MB grew in 2h since the previous restart 09/Jul/2024:07:22:20): ` tool... [10:11:31] (03update) 10dcaro: check_openapi_version: ensure that the new version is greater [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/43 [10:13:50] (03update) 10dcaro: check_openapi_version: ensure that the new version is greater [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/43 [10:18:45] (03update) 10dcaro: ci: add check-openapi-version-bump pre-commit hook [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T356972) [10:19:53] 10cloud-services-team (FY2023/2024-Q3-Q4), 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance, and 2 others: [ceph] Upgrade hosts to bullseye - https://phabricator.wikimedia.org/T309789#9964846 (10dcaro) [10:21:13] 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance: [reimage,ceph] reimaging cloudcephosd hosts gets stuck in network configuration screen - https://phabricator.wikimedia.org/T369026#9964853 (10dcaro) [10:21:34] 10Cloud-VPS, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Maintenance: [reimage,ceph] reimaging cloudcephosd hosts gets stuck in network configuration screen - https://phabricator.wikimedia.org/T369026#9964847 (10dcaro) 05In progress→03Resolved Yep, upgrading the firmware works [10:24:10] 10Tool-spacemedia, 10Server-side-upload-request: Server-side upload request for OptimusPrimeBot (INPE DPI) - https://phabricator.wikimedia.org/T366353#9964858 (10Don-vip) I've setup a local mediawiki instance (1.39, using Ubuntu package) with default config: `php // Use exiv2? if false, MediaWiki's intern... [10:41:34] (03open) 10marostegui: task_template*: Replace dborch command [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/5 [10:43:01] (03close) 10marostegui: task_template*: Replace dborch command [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/5 [10:48:19] (03open) 10marostegui: task_template*: Replace dborch command [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/6 [11:19:42] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:29:44] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [11:38:27] (03approved) 10aborrero: check_openapi_version: ensure that the new version is greater [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/43 (owner: 10dcaro) [11:40:41] (03update) 10dcaro: check_openapi_version: ensure that the new version is greater [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/43 [11:40:46] (03merge) 10dcaro: check_openapi_version: ensure that the new version is greater [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/43 [11:40:47] (03update) 10dcaro: check_openapi_version: ensure that the new version is greater [repos/cloud/cicd/gitlab-ci] - 10https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/43 [11:41:33] (03update) 10dcaro: ci: add check-openapi-version-bump pre-commit hook [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T356972) [11:42:22] (03update) 10dcaro: ci: add check-openapi-version-bump pre-commit hook [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T356972) [11:42:33] (03update) 10dcaro: ci: add check-openapi-version-bump pre-commit hook [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T356972) [11:44:57] (03open) 10dcaro: pre-commit: add openapi version bump check [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/101 (https://phabricator.wikimedia.org/T356972) [11:45:03] (03merge) 10dcaro: ci: add check-openapi-version-bump pre-commit hook [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/37 (https://phabricator.wikimedia.org/T356972) [11:49:23] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: envvars-api: bump to 0.0.52-20240705101149-aa9da2fa [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/391 (https://phabricator.wikimedia.org/T368142) [11:49:28] (03update) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: envvars-api: bump to 0.0.52-20240705101149-aa9da2fa [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/391 (https://phabricator.wikimedia.org/T356972 https://phabricator.wikimedia.org/T368142) [11:52:27] (03update) 10aborrero: kubecerts: have certificates lifetime to be max 10 days, renew them often [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/55 (https://phabricator.wikimedia.org/T365681) [12:03:49] (03open) 10dcaro: remove auth [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/102 (https://phabricator.wikimedia.org/T367181) [12:06:35] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation): Replace or remove Debian Buster VMs in 'monitoring' cloud-vps project - https://phabricator.wikimedia.org/T360703#9965164 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This is done (the `monitoring` project is no longer a thing) [12:06:53] (03update) 10dcaro: auth: remove ssl header auth and use x-toolforge-user [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T367181) [12:11:16] 10Tool-spacemedia, 10Server-side-upload-request: Server-side upload request for OptimusPrimeBot (INPE DPI) - https://phabricator.wikimedia.org/T366353#9965169 (10Urbanecm_WMF) >>! In T366353#9964858, @Don-vip wrote: > I've setup a local mediawiki instance (1.39, using Ubuntu package) with default config:... [12:23:26] 10Toolforge (Toolforge iteration 12), 13Patch-For-Review: [fourohfour,infra] Got a lot of alerts this weekend - https://phabricator.wikimedia.org/T369601#9965203 (10dcaro) Yep, logs seems way less busy: ` tools.fourohfour@tools-bastion-13:~$ date; ls -lrth uwsgi.log Tue Jul 9 12:21:20 UTC 2024 -rw-r----- 1 to... [12:49:42] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [12:59:05] (03open) 10aborrero: maintain-kubeusers: add .gitattributes file to don't print cassettes diff [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/56 [13:00:57] 06cloud-services-team, 10Data-Services, 06Data-Engineering, 10Data-Platform-SRE (2024.07.08 - 2024.07.28): Some wikibase tables not available in commonswiki_p - https://phabricator.wikimedia.org/T298452#9965277 (10Gehel) [13:04:42] RESOLVED: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:07:29] (03update) 10aborrero: maintain-kubeusers: add .gitattributes file to don't print cassettes diff [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/56 [13:09:18] (03merge) 10aborrero: maintain-kubeusers: add .gitattributes file to don't print cassettes diff [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/56 [13:10:01] (03update) 10aborrero: kubecerts: have certificates lifetime to be max 10 days, renew them often [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/55 (https://phabricator.wikimedia.org/T365681) [13:11:38] (03open) 10project_1317_bot_df3177307bed93c3f34e421e26c86e38: maintain-kubeusers: bump to 0.0.162-20240709130929-f214eda4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/401 [13:14:30] (03update) 10aborrero: kubecerts: have certificates lifetime to be max 10 days, renew them often [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/55 (https://phabricator.wikimedia.org/T365681) [14:14:57] (03update) 10marostegui: task_template*: Replace dborch command [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/6 [14:15:11] (03update) 10marostegui: task_template*: Replace dborch command [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/6 [14:15:23] (03update) 10marostegui: task_template*: Replace dborch command [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/6 [14:15:40] (03update) 10marostegui: task_template*: Replace dborch command [toolforge-repos/switchmaster] - 10https://gitlab.wikimedia.org/toolforge-repos/switchmaster/-/merge_requests/6 [14:17:22] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [14:17:32] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [14:18:57] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component maintain-kubeusers [14:19:07] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component maintain-kubeusers [14:19:24] (03merge) 10aborrero: maintain-kubeusers: bump to 0.0.162-20240709130929-f214eda4 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/401 (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [14:19:41] FIRING: CloudVPSDesignateLeaks: Detected 13 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [14:20:00] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission [14:20:10] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission [14:21:27] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component registry-admission [14:21:37] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component registry-admission [14:22:05] (03update) 10aborrero: registry-admission: bump to 0.0.45-20240708145115-17015d83 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/400 (https://phabricator.wikimedia.org/T368142) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [14:22:27] (03merge) 10aborrero: registry-admission: bump to 0.0.45-20240708145115-17015d83 [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/400 (https://phabricator.wikimedia.org/T368142) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [14:22:50] (03update) 10aborrero: envvars-api: bump to 0.0.52-20240705101149-aa9da2fa [repos/cloud/toolforge/toolforge-deploy] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/391 (https://phabricator.wikimedia.org/T356972 https://phabricator.wikimedia.org/T368142) (owner: 10project_1317_bot_df3177307bed93c3f34e421e26c86e38) [14:36:16] FIRING: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:41:16] RESOLVED: ProbeDown: Service tools-legacy-redirector-2:443 has failed probes (http_tools_wmflabs_org_tool_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-legacy-redirector-2:443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [14:58:54] (03update) 10aborrero: kubecerts: have certificates lifetime to be max 10 days, renew them often [repos/cloud/toolforge/maintain-kubeusers] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/maintain-kubeusers/-/merge_requests/55 (https://phabricator.wikimedia.org/T365681) [15:32:22] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 06Traffic: Replace or remove Debian Buster VMs in 'traffic' cloud-vps project - https://phabricator.wikimedia.org/T360710#9965865 (10Vgutierrez) [16:44:35] 10Tool-spacemedia, 10Server-side-upload-request: Server-side upload request for OptimusPrimeBot (INPE DPI) - https://phabricator.wikimedia.org/T366353#9966215 (10Bawolff) 600 mb! I imagine mediawiki is cutting off the output way before that, probably losing somd important data. [18:19:56] FIRING: CloudVPSDesignateLeaks: Detected 18 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:24:16] 06cloud-services-team, 10Cloud-VPS (Debian Buster Deprecation), 06Traffic: Replace or remove Debian Buster VMs in 'traffic' cloud-vps project - https://phabricator.wikimedia.org/T360710#9966519 (10BCornwall) 05Open→03In progress p:05Triage→03Medium a:03BCornwall @Vgutierrez: traffic-cache-atstext-b... [18:54:38] (03approved) 10raymond-ndibe: remove auth [repos/cloud/toolforge/builds-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/102 (https://phabricator.wikimedia.org/T367181) (owner: 10dcaro) [18:56:52] (03update) 10raymond-ndibe: auth: remove ssl header auth and use x-toolforge-user [repos/cloud/toolforge/envvars-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/envvars-api/-/merge_requests/38 (https://phabricator.wikimedia.org/T367181) (owner: 10dcaro) [19:19:45] (03update) 10raymond-ndibe: [lima-kilo] fix toolforge_deploy_mr restore [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/163 [19:23:29] (03update) 10raymond-ndibe: [lima-kilo] fix toolforge_deploy_mr restore [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/163 [19:23:36] (03approved) 10raymond-ndibe: [lima-kilo] fix toolforge_deploy_mr restore [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/163 [19:23:40] (03merge) 10raymond-ndibe: [lima-kilo] fix toolforge_deploy_mr restore [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/163 [19:25:00] (03update) 10raymond-ndibe: kind: upgrade to k8s 1.25 [repos/cloud/toolforge/lima-kilo] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/lima-kilo/-/merge_requests/161 (https://phabricator.wikimedia.org/T369165) (owner: 10sstefanova) [19:46:19] (03open) 10gergesshamon: V1.1 [toolforge-repos/links-monitor] - 10https://gitlab.wikimedia.org/toolforge-repos/links-monitor/-/merge_requests/2 [19:46:34] (03merge) 10gergesshamon: V1.1 [toolforge-repos/links-monitor] - 10https://gitlab.wikimedia.org/toolforge-repos/links-monitor/-/merge_requests/2 [19:46:56] (03open) 10gergesshamon: V1.2 [toolforge-repos/links-monitor] - 10https://gitlab.wikimedia.org/toolforge-repos/links-monitor/-/merge_requests/3 [19:47:11] (03merge) 10gergesshamon: V1.2 [toolforge-repos/links-monitor] - 10https://gitlab.wikimedia.org/toolforge-repos/links-monitor/-/merge_requests/3 [19:48:18] (03update) 10raymond-ndibe: [jobs-cli] refactor error handling [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/48 [19:51:17] (03update) 10raymond-ndibe: [jobs-cli] refactor error handling [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/48 [19:51:50] (03update) 10raymond-ndibe: [jobs-cli] refactor error handling [repos/cloud/toolforge/jobs-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/48 [20:00:20] (03update) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] (refactor_error_handling) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) [20:06:29] (03PS1) 10Hridyesh_Gupta: Fixed the typo in Top contributor campaign page [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1053046 [20:11:31] (03update) 10raymond-ndibe: [jobs-cli] move jobs load to backend [repos/cloud/toolforge/jobs-cli] (refactor_job_validation) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44 (https://phabricator.wikimedia.org/T366209) [20:13:09] (03update) 10raymond-ndibe: [jobs-cli] refactor job validation [repos/cloud/toolforge/jobs-cli] (refactor_error_handling) - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/45 (https://phabricator.wikimedia.org/T366209) [20:13:31] (03PS2) 10Hridyesh_Gupta: Fixed the typo in Top contributor campaign page [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1053046 (https://phabricator.wikimedia.org/T358396) [20:14:34] (03PS1) 10GauriGuptaa: Typo in Top contributor campaign page is resolved [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1053047 [20:21:35] (03open) 10raymond-ndibe: [builds-cli] bug fix [repos/cloud/toolforge/builds-cli] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/79 [20:29:08] 06cloud-services-team, 10Cloud-VPS, 10Data-Services, 13Patch-For-Review: Fix 'openstack database instance rebuild' - https://phabricator.wikimedia.org/T355721#9967082 (10Andrew) Rebuilding now works fairly reliably for mariadb and mysql -- the only hitch I've run into is disagreement about where to mount t... [22:19:56] FIRING: CloudVPSDesignateLeaks: Detected 23 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [23:15:53] (03PS1) 10Aditya0545: ISA: New footer not included in translation message Add missing footer message to messages.pot - Updated messages.pot to include the new footer message "Powered by SWITCH/WMSE/Bern Uni". - Corrected line references in messages.pot to align with source code. [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1053071 (https://phabricator.wikimedia.org/T340538) [23:20:40] 10VPS-project-Codesearch, 06collaboration-services: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199#9967661 (10Ladsgroup) >>! In T268199#9905399, @Dzahn wrote: > One thing to be done here: > > add a systemd unit for the frontend. See T367479#9904556 I looked at this a bit. It's... [23:33:10] 10VPS-project-Codesearch, 06collaboration-services: Graduate codesearch to production - https://phabricator.wikimedia.org/T268199#9967699 (10Ladsgroup) Making these images pushed to docker registry via CI is actually quite complicated because codesearch pushes via blubber (I don't think there is any other way)... [23:53:42] 10Tool-spacemedia, 10Server-side-upload-request: Server-side upload request for OptimusPrimeBot (INPE DPI) - https://phabricator.wikimedia.org/T366353#9967731 (10Don-vip) >>! In T366353#9965169, @Urbanecm_WMF wrote: > Let's avoid prematurely jumping into conclusions :). As showed above, your comment can be... [23:57:32] (03PS1) 10Aditya0545: ISA: Typo in Top contributor campaign page Fix typo in Top contributor campaign page [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1053078 (https://phabricator.wikimedia.org/T358396)