[02:47:28] (PuppetCertificateAboutToExpire) firing: Puppet CA certificate Puppet CA: cloudinfra-internal-puppetmaster01.cloudinfra.eqiad.wmflabs is about to expire in 26d 17h 48m 11s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [03:45:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [03:55:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:15:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:25:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [05:47:28] (PuppetCertificateAboutToExpire) firing: Puppet CA certificate Puppet CA: cloudinfra-internal-puppetmaster01.cloudinfra.eqiad.wmflabs is about to expire in 26d 14h 48m 11s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [06:15:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [06:25:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:10:56] (ProbeDown) firing: Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:15:56] (ProbeDown) resolved: Service tools-k8s-haproxy-3:30000 has failed probes (http_admin_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-k8s-haproxy-3:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [07:44:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [07:54:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [08:47:28] (PuppetCertificateAboutToExpire) firing: Puppet CA certificate Puppet CA: cloudinfra-internal-puppetmaster01.cloudinfra.eqiad.wmflabs is about to expire in 26d 11h 48m 11s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [09:30:04] 10Wikibugs: Set a timeout for phorge requests - https://phabricator.wikimedia.org/T359145 (10taavi) [09:35:16] 10Toolforge, 06cloud-services-team, 07Documentation, 07Kubernetes: Figure out and document how to call the Kubernetes API as your tool user from inside a pod - https://phabricator.wikimedia.org/T321919#9599668 (10dcaro) [09:35:48] 10Toolforge (Toolforge iteration 06), 13Patch-For-Review: Upgrade Toolforge image builder to Bookworm - https://phabricator.wikimedia.org/T358483#9599670 (10dcaro) p:05Triage→03Medium [09:35:54] 10Cloud-VPS (Debian Buster Deprecation), 10Toolforge, 10cloud-services-team (FY2023/2024-Q3-Q4), 07Epic, 05Goal: Toolforge: migrate to Debian Bullseye or later - https://phabricator.wikimedia.org/T311897#9599673 (10dcaro) [09:36:01] 10Toolforge (Toolforge iteration 06), 13Patch-For-Review: [builds-api,jobs-api,envvars-api,api-gateway] FIgure out and document how to do non-backwards compatible changes - https://phabricator.wikimedia.org/T356974#9599667 (10dcaro) 05Open→03In progress [09:37:50] 10Toolforge (Toolforge iteration 06), 13Patch-For-Review: Upgrade Toolforge image builder to Bookworm - https://phabricator.wikimedia.org/T358483#9599672 (10dcaro) 05Open→03In progress [09:38:00] 10Toolforge (Toolforge iteration 06), 10Toolforge Jobs framework, 13Patch-For-Review: Support job health checks - https://phabricator.wikimedia.org/T335592#9599675 (10dcaro) p:05Triage→03Medium [09:39:45] 10Toolforge (Toolforge iteration 06), 10Toolforge Build Service, 13Patch-For-Review: [maintain-harbor] Improvements to subcommands and config validation - https://phabricator.wikimedia.org/T353059#9599679 (10dcaro) 05Open→03In progress [10:14:50] (ProbeDown) firing: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:19:50] (ProbeDown) resolved: Service tools-static-14:80 has failed probes (http_tools_static_wmflabs_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#tools-static-14:80 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [10:30:05] 10Toolforge Build Service: [tbs] Explore adding caching support - https://phabricator.wikimedia.org/T350689#9599917 (10dcaro) [10:30:52] 10Toolforge Build Service: [tbs] Explore adding caching support - https://phabricator.wikimedia.org/T350689#9599920 (10dcaro) [10:31:00] 10Toolforge Build Service: [tbs] Explore adding caching support - https://phabricator.wikimedia.org/T350689#9599923 (10dcaro) [10:31:09] 10Toolforge Build Service: [buildservice] Cache .m2 folder (local maven repository) between builds - https://phabricator.wikimedia.org/T350307#9599924 (10dcaro) [10:31:37] 10Toolforge: [builds-builder] Explore adding caching support - https://phabricator.wikimedia.org/T350689#9599928 (10dcaro) p:05Triage→03Medium [10:40:41] 10Toolforge Build Service: [tbs][builder] Explore adding support for third-party buildpacks - https://phabricator.wikimedia.org/T352389#9599960 (10dcaro) 05Open→03Resolved a:03dcaro I think we can close this for now as we are not probably be working on it for the next FY, and reopen if it becomes urgent or... [10:41:28] 10Toolforge: [buildservice] Cache .m2 folder (local maven repository) between builds - https://phabricator.wikimedia.org/T350307#9599968 (10dcaro) p:05Triage→03Low [11:14:11] 10Toolforge (Toolforge iteration 06): Rust compilation fails on jobs framework - https://phabricator.wikimedia.org/T358090#9599991 (10dcaro) p:05Triage→03High Did that help? Do you need more guidance? [11:15:32] 10Toolforge (Toolforge iteration 06): Rust compilation fails on jobs framework - https://phabricator.wikimedia.org/T358090#9599997 (10Magnus) 05Open→03Resolved a:03Magnus Yes, works for me, thanks [11:16:04] 10Toolforge Build Service: [tbs] Unable to get pywikibot + wget on a python build service image - https://phabricator.wikimedia.org/T354157#9600000 (10dcaro) @YFdyh000 did that help? do you need more guidance? We have fixed a few bugs also in the meantime, including now `toolforge jobs run test --command "pytho... [11:16:12] 10Toolforge Build Service: [tbs] Unable to get pywikibot + wget on a python build service image - https://phabricator.wikimedia.org/T354157#9600001 (10dcaro) p:05Triage→03High [11:16:36] 10Toolforge (Toolforge iteration 06): [tbs] Unable to get pywikibot + wget on a python build service image - https://phabricator.wikimedia.org/T354157#9600003 (10dcaro) [11:17:12] 10Toolforge (Toolforge iteration 06), 07Documentation: [tbs] Improve Harbor quota handling and docs - https://phabricator.wikimedia.org/T351092#9600008 (10dcaro) [11:18:04] 10Toolforge, 06cloud-services-team, 07Kubernetes: [toolforge k8s] Support Cinder volumes - https://phabricator.wikimedia.org/T275555#9600017 (10dcaro) [11:18:12] 10Toolforge Build Service, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [tbs]Add storage capabilities for buildpack services - https://phabricator.wikimedia.org/T293670#9600018 (10dcaro) [11:18:20] 10Toolforge Build Service, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [tbs]Add storage capabilities for buildpack services - https://phabricator.wikimedia.org/T293670#9600019 (10dcaro) [11:21:29] 10Toolforge, 06cloud-services-team, 07Kubernetes: [toolforge,storage] Support Cinder volumes - https://phabricator.wikimedia.org/T275555#9600047 (10dcaro) [11:21:37] 10Toolforge, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [toolforge,storage] Add storage capabilities for tools - https://phabricator.wikimedia.org/T293670#9600043 (10dcaro) p:05Triage→03Medium [11:25:48] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9600071 (10MBH) Since I have `.sln` file in every my bot/project folder on my PC, like `.csproj` file, maybe it will be better to use this native sln files, one for every tool... [11:32:19] 05Grid-Engine-to-K8s-Migration: Migrate mbh from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319883#9600141 (10dcaro) >>! In T319883#9600071, @MBH wrote: > Since I have `.sln` file in every my bot/project folder on my PC, like `.csproj` file, maybe it will be better to use t... [11:33:42] 10Toolforge, 07Upstream: Python buildpack does not detect requirements from pyproject.toml - https://phabricator.wikimedia.org/T353762#9600149 (10dcaro) 05Open→03Stalled Waiting for upstream https://github.com/heroku/buildpacks-python/issues/7 [11:34:18] 10Toolforge: [apt-buildpack] Installed python scripts with a hardcoded shebang to the python binary will not work when installing new pythons - https://phabricator.wikimedia.org/T356500#9600157 (10dcaro) [11:34:34] 10Wikibugs: Frequent "Redis listener crashed; restarting in a few seconds." errors logged - https://phabricator.wikimedia.org/T359097#9600159 (10taavi) >>! In T359097#9597794, @bd808 wrote: > * Is connectivity to Redis actually lost this often? Yes, in the past 15 minutes the worker node where the `irc` task is... [11:34:42] 10Toolforge, 10cloud-services-team (FY2023/2024-Q3-Q4), 05Goal, 15User-Raymond_Ndibe, 15User-aborrero: [harbor] Deploy with Helm - https://phabricator.wikimedia.org/T356301#9600165 (10dcaro) [11:35:00] 10Toolforge: Add a container for Swift - https://phabricator.wikimedia.org/T354815#9600170 (10dcaro) [11:35:24] 10Toolforge Build Service: [apt-buildpack] Add local Ubuntu mirror or package cache - https://phabricator.wikimedia.org/T357251#9600175 (10dcaro) [11:35:32] 10Toolforge: [builds-builder] Explore adding caching support - https://phabricator.wikimedia.org/T350689#9600176 (10dcaro) [11:36:05] 10Toolforge Build Service: [build-service,apt-buildpack] Add local Ubuntu mirror or package cache - https://phabricator.wikimedia.org/T357251#9600188 (10dcaro) [11:36:13] 10Toolforge: [build-service,apt-buildpack] Add local Ubuntu mirror or package cache - https://phabricator.wikimedia.org/T357251#9600189 (10dcaro) [11:39:57] 10Toolforge, 06cloud-services-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [builds-builder,harbor,bulid-service] user-story 11: Add section to admin docs on how to debug the service, how to pin-point the failing c... - https://phabricator.wikimedia.org/T325174#9600221 [11:40:40] 10Toolforge, 06cloud-services-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [builds-builder,harbor,bulid-service] user-story 11: Add section to admin docs on how to debug the service, how to pin-point the failing c... - https://phabricator.wikimedia.org/T325174#9600213 [11:40:43] 10Toolforge Build Service, 06cloud-services-team: [builds-cli] --debug option behaviour is confusing - https://phabricator.wikimedia.org/T354726#9600226 (10dcaro) [11:42:30] 10Toolforge, 06cloud-services-team: [builds-cli] --debug option behaviour is confusing - https://phabricator.wikimedia.org/T354726#9600237 (10dcaro) I think this might be a regression, we should be using environment variables from the main toolforge cli to pass down the debug flags too, though `toolforge build... [11:42:49] 10Toolforge, 06cloud-services-team: [builds-cli] --debug option behaviour is confusing - https://phabricator.wikimedia.org/T354726#9600240 (10dcaro) p:05Triage→03Low [11:45:59] 10Toolforge, 06cloud-services-team: [builds-api] log stack trace for 5xx errors - https://phabricator.wikimedia.org/T354731#9600256 (10dcaro) p:05Triage→03Medium [11:46:07] 10Toolforge, 06cloud-services-team: [builds-api] log stack trace for 5xx errors - https://phabricator.wikimedia.org/T354731#9600258 (10dcaro) [11:47:28] (PuppetCertificateAboutToExpire) firing: Puppet CA certificate Puppet CA: cloudinfra-internal-puppetmaster01.cloudinfra.eqiad.wmflabs is about to expire in 26d 8h 48m 11s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [11:47:55] 10Toolforge, 06cloud-services-team: [harbor] Update HarborDown runbook with the incident debugging details - https://phabricator.wikimedia.org/T354739#9600274 (10dcaro) p:05Triage→03Medium [11:49:42] 10Toolforge (Toolforge iteration 06): [apt-buildpak] Some APT packages are not installed during the image build - https://phabricator.wikimedia.org/T355252#9600282 (10dcaro) p:05Triage→03High a:03dcaro @Dapete there were a bunch of improvements on the apt-buildpack about dependency resolution and similar,... [11:51:44] 10Toolforge (Toolforge iteration 06): Support monorepos with the Multi Procfile buildpack - https://phabricator.wikimedia.org/T355329#9600312 (10dcaro) p:05Triage→03High a:03dcaro @Count_Count did it work? [11:52:51] 10Toolforge: [ci][builds-cli][envvars-cli] Investigate discrepancy between different CI envs - https://phabricator.wikimedia.org/T353044#9600322 (10dcaro) p:05Triage→03Medium [11:53:11] 10Toolforge: [ci][builds-cli][envvars-cli] Investigate discrepancy between different CI envs - https://phabricator.wikimedia.org/T353044#9600326 (10dcaro) We should double check that this is still an issue, and that we are using the same image for CI and run_local_ci (I think we are on python 3.9 already) [11:54:48] 10Toolforge Build Service: [tbs][dev] decide on which kubernetes bootstrapper to focus on between minikube and kind - https://phabricator.wikimedia.org/T347723#9600328 (10dcaro) 05Open→03Resolved a:03dcaro We went for kind on lima VM. [11:55:04] 10Toolforge, 06cloud-services-team, 13Patch-For-Review: [harbor,trove] Trove DB filled disk and caused toolforge-build to fail as a result - https://phabricator.wikimedia.org/T354714#9600333 (10dcaro) [12:12:03] 10Tool-Global-user-contributions, 06Stewards-and-global-tools, 10Temporary accounts, 10XTools, and 2 others: [Design] Synthesise user testing results - https://phabricator.wikimedia.org/T358098#9600406 (10KColeman-WMF) [12:23:44] 10PAWS: remove jupyter-dash - https://phabricator.wikimedia.org/T358621#9600457 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/382 [12:23:55] vivian-rook opened https://github.com/toolforge/paws/pull/382 [12:26:09] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T359159 (10Curb_Safe_Charmer) [12:28:30] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T359159#9600476 (10Curb_Safe_Charmer) 05Open→03In progress a:05Curb_Safe_Charmer→03TheresNoTime Any ideas Sammy? [12:46:04] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T359159#9600537 (10TheresNoTime) partial of `kubectl get pod refill-api-scheduler-5d67b8bc49-fqz5q --output=yaml`; ` lastState: terminated: containerID: containerd://0a83afbffbe834b8a63080ebc... [13:08:45] 10Toolforge Build Service: [build-service] Add builder support for Perl runtime projects - https://phabricator.wikimedia.org/T353744#9600621 (10dcaro) [13:09:09] 10Toolforge: [build-service] Add builder support for Perl runtime projects - https://phabricator.wikimedia.org/T353744#9600622 (10dcaro) p:05Triage→03Low [13:09:44] 10Toolforge: [webservice-cli,builds-api] `webservice restart` sometimes timing out for buildservice images - https://phabricator.wikimedia.org/T341057#9600627 (10dcaro) [13:10:00] 10Toolforge: [webservice-cli,builds-api] `webservice restart` sometimes timing out for buildservice images - https://phabricator.wikimedia.org/T341057#9600625 (10dcaro) p:05Triage→03Low [13:10:47] 10Toolforge, 07Documentation: [tbs] Create a tutorial on compiling static frontend assets at build time - https://phabricator.wikimedia.org/T351082#9600629 (10dcaro) p:05Triage→03Low [13:12:21] 10Toolforge, 06cloud-services-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [build-service] user-story 14: Run a set of security checks on the full service - https://phabricator.wikimedia.org/T325208#9600635 (10dcaro) p:05Triage→03Medium [13:12:26] 10Toolforge, 06cloud-services-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [build-service] user-story 14: Run a set of security checks on the full service - https://phabricator.wikimedia.org/T325208#9600637 (10dcaro) [13:12:37] 10Toolforge (Toolforge iteration 06), 13Patch-For-Review, 07Upstream: [maintain-harbor] Manage project quotas via maintain-harbor - https://phabricator.wikimedia.org/T352417#9600640 (10dcaro) [13:13:39] 10Toolforge (Toolforge iteration 06): `build quota` fails if tool has no builds - https://phabricator.wikimedia.org/T353701#9600647 (10dcaro) [13:13:50] 10Toolforge (Toolforge iteration 06): [harbor] upgrade to 2.10.x - https://phabricator.wikimedia.org/T354507#9600648 (10dcaro) [13:14:02] 10Toolforge (Toolforge iteration 06): Build service: Calling nontrivial Procfile commands with arguments results in confusing error (“no such file or directory”) - https://phabricator.wikimedia.org/T356016#9600649 (10dcaro) [13:14:28] 10Toolforge (Toolforge iteration 06): [builds-builder,jobs-api] Calling nontrivial Procfile commands with arguments results in confusing error (“no such file or directory”) - https://phabricator.wikimedia.org/T356016#9600650 (10dcaro) [13:14:29] 10Toolforge: [builds-api] Improve error message when logs time out - https://phabricator.wikimedia.org/T354755#9600644 (10dcaro) p:05Triage→03Low [13:15:16] 05Grid-Engine-to-K8s-Migration: Migrate wd-shex-infer from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320140#9600653 (10dcaro) [13:15:42] 10Toolforge (Toolforge iteration 06), 10Toolforge Build Service, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [maintain-harbor] Document current setup and admin procedures - https://phabricator.wikimedia.org/T329176#9600654 (10dcaro) [13:16:13] 10Toolforge (Toolforge iteration 06): [builds-builder,jobs-api] Calling nontrivial Procfile commands with arguments results in confusing error (“no such file or directory”) - https://phabricator.wikimedia.org/T356016#9600651 (10dcaro) 05In progress→03Stalled [13:16:43] 10Toolforge (Toolforge iteration 06), 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [maintain-harbor] Document current setup and admin procedures - https://phabricator.wikimedia.org/T329176#9600659 (10dcaro) [13:16:59] 10Toolforge (Toolforge iteration 06), 10cloud-services-team (FY2023/2024-Q3-Q4): [tbs] Create a tutorial on how to deploy a Node.js app using Build Service - https://phabricator.wikimedia.org/T353313#9600662 (10dcaro) [13:17:15] 10Toolforge (Toolforge iteration 06), 10cloud-services-team (FY2023/2024-Q3-Q4): [tbs] Add dashboards with the new statistics - https://phabricator.wikimedia.org/T352764#9600664 (10dcaro) [13:17:40] 10Toolforge (Toolforge iteration 06), 10cloud-services-team (FY2023/2024-Q3-Q4): [builds-api] Add dashboards with the new statistics - https://phabricator.wikimedia.org/T352764#9600665 (10dcaro) [13:18:04] 10Toolforge (Toolforge iteration 06), 13Patch-For-Review: [maintain-harbor] Improvements to subcommands and config validation - https://phabricator.wikimedia.org/T353059#9600666 (10dcaro) [13:18:39] 10Toolforge (Toolforge iteration 06), 06cloud-services-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [builds-api] Automatically deploy the webservice when the image is built - https://phabricator.wikimedia.org/T341065#9600671 (10dcaro) [13:18:50] 10Toolforge (Toolforge iteration 06), 06cloud-services-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [builds-api,orchestration] Automatically deploy the webservice when the image is built - https://phabricator.wikimedia.org/T341065#9600674 (10dcaro) [13:19:25] 10Cloud Services Proposals, 10Toolforge, 10cloud-services-team (FY2023/2024-Q3-Q4), 05Cloud-Services-Origin-Team, and 4 others: [Epic] Make Toolforge a proper platform as a service with push-to-deploy and build packs - https://phabricator.wikimedia.org/T194332#9600678 (10dcaro) [13:20:04] 06cloud-services-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 07Epic, 15User-dcaro: tbs: user-story 11: I want to know how to debug the service - https://phabricator.wikimedia.org/T325172#9600681 (10dcaro) p:05Triage→03Medium [13:20:23] 06cloud-services-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 07Epic, 15User-dcaro: [builds-api,harbor,builds-builder] user-story 11: I want to know how to debug the service - https://phabricator.wikimedia.org/T325172#9600683 (10dcaro) [13:20:59] 10Toolforge, 06cloud-services-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, and 2 others: [builds-api,harbor,builds-builder] user-story 11: I want to know how to debug the service - https://phabricator.wikimedia.org/T325172#9600684 (10dcaro) [13:21:53] 10Toolforge, 06cloud-services-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, and 2 others: [builds-cli,builds-api,builds-builder,jobs-cli,josb-api] user-story 14: I want have some certainty that the service is secure - https://phabricator.wikimedia.org/T325207#9600691 (10dcaro) [13:22:44] 10Toolforge, 06cloud-services-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, and 2 others: [builds-cli,builds-api,builds-builder,jobs-cli,josb-api] user-story 14: I want have some certainty that the service is secure - https://phabricator.wikimedia.org/T325207#9600688 (10dcaro) p:0... [13:25:44] 10PAWS: remove jupyter-dash - https://phabricator.wikimedia.org/T358621#9600708 (10github-toolforge-bot) vivian-rook closed https://github.com/toolforge/paws/pull/382 [13:25:57] vivian-rook closed https://github.com/toolforge/paws/pull/382 [13:26:04] 10PAWS: remove jupyter-dash - https://phabricator.wikimedia.org/T358621#9600712 (10rook) 05Open→03Resolved [13:30:05] 10Toolforge, 07Epic, 15User-Raymond_Ndibe: Run webservices via the jobs framework - https://phabricator.wikimedia.org/T348755#9600742 (10Raymond_Ndibe) a:03Raymond_Ndibe [13:34:19] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T359159#9600756 (10TheresNoTime) nb, needed to add `toolforge: tool` label to `worker-deployment.yml` to resolve above, now looking at ` Could not load cache: EOFError('Ran out of input') [2024-03-05 13:33:50,7... [13:39:18] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T359159#9600785 (10TheresNoTime) Some restarts later and this is resolved. Found a bug when doing my normal test article (`Depths of wikipedia`): ` [2024-03-05 13:37:22,384: ERROR/ForkPoolWorker-8] Task refill... [13:39:40] 10Toolforge Jobs framework, 06cloud-services-team: Special consideration needed for toolforge-jobs when performing kubernetes cluster upgrades? - https://phabricator.wikimedia.org/T292981#9600783 (10dcaro) 05Open→03Invalid We have been running jobs already on k8s for a while without issues, I think this do... [13:40:12] 10Toolforge: [jobs-emailer] job emails should have timestamps in events - https://phabricator.wikimedia.org/T306309#9600800 (10dcaro) [13:40:33] 10Toolforge: [jobs-emailer] job emails should have timestamps in events - https://phabricator.wikimedia.org/T306309#9600787 (10dcaro) p:05Triage→03Low [13:47:18] 10Toolforge, 07Kubernetes: Allow Toolforge scheduled jobs to have a maximum runtime - https://phabricator.wikimedia.org/T306391#9600847 (10dcaro) We are working on adding health checks to the jobs {T335592} and accessing the APIs from the containers themselves {T321919}, I'll add the checks as dependent as tha... [13:47:24] 10Toolforge, 07Kubernetes: Allow Toolforge scheduled jobs to have a maximum runtime - https://phabricator.wikimedia.org/T306391#9600850 (10dcaro) [13:47:32] 10Toolforge (Toolforge iteration 06), 10Toolforge Jobs framework, 13Patch-For-Review: Support job health checks - https://phabricator.wikimedia.org/T335592#9600851 (10dcaro) [13:48:06] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T359159#9600856 (10TheresNoTime) 05In progress→03Resolved [13:48:59] 10Toolforge, 07Kubernetes: Allow Toolforge scheduled jobs to have a maximum runtime - https://phabricator.wikimedia.org/T306391#9600857 (10dcaro) 05Open→03Stalled [13:49:58] 10Toolforge, 07Kubernetes: Allow Toolforge scheduled jobs to have a maximum runtime - https://phabricator.wikimedia.org/T306391#9600860 (10dcaro) p:05Triage→03Medium [13:50:13] 10Toolforge (Toolforge iteration 06), 15User-aborrero: [toolforge] simplify calling the different toolforge apis from within the containers - https://phabricator.wikimedia.org/T356377#9600871 (10dcaro) [13:50:16] 10Toolforge Jobs framework: Make it possible to start/restart jobs from other k8s jobs - https://phabricator.wikimedia.org/T315729#9600869 (10dcaro) [13:50:18] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T359159#9600873 (10Curb_Safe_Charmer) Brilliant, @TheresNoTime, thank you! Any indication of root cause? [13:51:28] 10Cloud-Services: Outdated repository data for - https://phabricator.wikimedia.org/T359169 (10MoritzMuehlenhoff) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to th... [13:52:26] 10Toolforge (Toolforge iteration 06), 07Epic: Consolidate the Toolforge CLIs - https://phabricator.wikimedia.org/T356262#9600919 (10dcaro) [13:52:35] 10Toolforge, 13Patch-For-Review: Add tfj as a shortcut for toolforge-jobs command - https://phabricator.wikimedia.org/T309308#9600918 (10dcaro) [13:54:07] 10Toolforge, 15User-Raymond_Ndibe: [jobs-cli,orchestrator] Provide YAML schema file for toolforge-jobs definition files - https://phabricator.wikimedia.org/T314729#9600971 (10dcaro) [13:54:15] 10Toolforge, 13Patch-For-Review: Add tfj as a shortcut for toolforge-jobs command - https://phabricator.wikimedia.org/T309308#9600903 (10dcaro) 05Open→03Stalled p:05Triage→03Low [13:54:23] 10Toolforge, 15User-Raymond_Ndibe: [jobs-cli,orchestrator] Provide YAML schema file for toolforge-jobs definition files - https://phabricator.wikimedia.org/T314729#9600955 (10dcaro) p:05Triage→03Low We are planning on introducing a higher level yaml file that will include this one, so this might end up app... [13:55:11] 10Toolforge (Toolforge iteration 06): Outdated repository data for - https://phabricator.wikimedia.org/T359169#9601000 (10taavi) a:03taavi This is https://kubernetes.io/blog/2023/08/31/legacy-package-repository-deprecation/, this is the old repository which is now gone and the next Kubernetes version (1.24) is... [13:55:20] 10Tool-refill: Refill tool stuck "waiting for an available worker" - https://phabricator.wikimedia.org/T359159#9601003 (10TheresNoTime) >>! In T359159#9600873, @Curb_Safe_Charmer wrote: > Brilliant, @TheresNoTime, thank you! > > Any indication of root cause? I've not looked for the root cause, but will note t... [13:56:42] 10Toolforge: [jobs-api,jobs-cli] API read timeout exception crashes `toolforge jobs logs --follow NAME` after a few seconds - https://phabricator.wikimedia.org/T358534#9601004 (10dcaro) p:05Triage→03Medium [13:57:30] 10Toolforge: [jobs-api,jobs-cli] API read timeout exception crashes `toolforge jobs logs --follow NAME` after a few seconds - https://phabricator.wikimedia.org/T358534#9601014 (10dcaro) [14:02:38] 10Toolforge (Toolforge iteration 06), 13Patch-For-Review: Outdated repository data for - https://phabricator.wikimedia.org/T359169#9601032 (10taavi) 05Open→03Resolved [14:11:10] 10Tool-global-search, 06Data-Platform-SRE, 06Discovery-Search: 400 - Bad Request on any Global Search - https://phabricator.wikimedia.org/T358541#9601075 (10bking) [14:12:46] 10Striker, 10Toolforge (Toolforge iteration 06), 13Patch-For-Review: ACCOUNT_SSH.html links to obsolete help page - https://phabricator.wikimedia.org/T358615#9601071 (10taavi) 05Open→03Resolved [14:36:02] 10Toolforge (Toolforge iteration 06), 15User-Raymond_Ndibe: [toolforge-cd] remove duplicated run on tag and push to master (just do one if possible) - https://phabricator.wikimedia.org/T353563#9601192 (10dcaro) [14:38:15] 10Toolforge (Toolforge iteration 06), 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [maintain-harbor] Document current setup and admin procedures - https://phabricator.wikimedia.org/T329176#9601204 (10dcaro) [14:38:30] 10Toolforge (Toolforge iteration 06), 13Patch-For-Review: [maintain-harbor] Improvements to subcommands and config validation - https://phabricator.wikimedia.org/T353059#9601205 (10dcaro) [14:39:46] 10Toolforge (Toolforge iteration 06), 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [maintain-harbor] Document current setup and admin procedures - https://phabricator.wikimedia.org/T329176#9601208 (10dcaro) 05In progress→03Stalled [14:39:51] 10Toolforge (Toolforge iteration 06), 07Documentation: [tbs] Improve Harbor quota handling and docs - https://phabricator.wikimedia.org/T351092#9601209 (10dcaro) [14:41:05] 10Cloud Services Proposals, 10Toolforge, 10cloud-services-team (FY2023/2024-Q3-Q4), 05Cloud-Services-Origin-Team, and 4 others: [Epic] Make Toolforge a proper platform as a service with push-to-deploy and build packs - https://phabricator.wikimedia.org/T194332#9601220 (10dcaro) [14:42:22] 10Toolforge (Toolforge iteration 06), 06cloud-services-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [builds-api,orchestration] Automatically deploy the webservice when the image is built - https://phabricator.wikimedia.org/T341065#9601219 (10dcaro) 05In progress→... [14:43:31] 10Toolforge (Toolforge iteration 06), 13Patch-For-Review, 15User-aborrero: toolforge: introduce OpenAPI to jobs framework - https://phabricator.wikimedia.org/T356523#9601247 (10CodeReviewBot) aborrero merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/59 jobs-api: add openap... [14:44:18] 10Toolforge (Toolforge iteration 06), 07Documentation: [tbs] Improve Harbor quota handling and docs - https://phabricator.wikimedia.org/T351092#9601245 (10dcaro) 05Open→03Stalled [14:44:43] 10Toolforge (Toolforge iteration 06), 13Patch-For-Review, 15User-aborrero: toolforge: introduce OpenAPI to jobs framework - https://phabricator.wikimedia.org/T356523#9601258 (10aborrero) follow up: * make sure all jobs (regardless of type) accept the same length for the job name * review the /healthz endpoin... [14:45:53] 10Toolforge (Toolforge iteration 06), 06cloud-services-team, 07Kubernetes, 13Patch-For-Review, 15User-aborrero: Toolforge k8s: Migrate workers to Containerd and Bookworm - https://phabricator.wikimedia.org/T284656#9601274 (10aborrero) [14:46:07] 10Toolforge, 10cloud-services-team (FY2023/2024-Q3-Q4), 05Goal, 13Patch-For-Review: Toolforge: Decommission the Grid Engine infrastructure - https://phabricator.wikimedia.org/T314664#9601275 (10taavi) a:03taavi [14:46:35] 10Toolforge (Toolforge iteration 06), 06cloud-services-team, 07Kubernetes, 15User-aborrero: toolforge k8s: some static pods needs manual restart - https://phabricator.wikimedia.org/T358476#9601272 (10aborrero) 05Open→03Resolved a:03aborrero [14:52:28] (PuppetCertificateAboutToExpire) firing: Puppet CA certificate Puppet CA: cloudinfra-internal-puppetmaster01.cloudinfra.eqiad.wmflabs is about to expire in 26d 5h 44m 11s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [14:57:25] 10PAWS: Add a simple script to connect to a replica database - https://phabricator.wikimedia.org/T358984#9601339 (10github-toolforge-bot) vivian-rook opened https://github.com/toolforge/paws/pull/383 [14:57:35] vivian-rook opened https://github.com/toolforge/paws/pull/383 [14:59:15] 10Toolforge Jobs framework, 15User-aborrero: [jobs-api] Support services in jobs - https://phabricator.wikimedia.org/T348758#9601341 (10dcaro) p:05Triage→03High [14:59:20] 10Toolforge Jobs framework, 15User-aborrero: [jobs-api,jobs-cli] Support services in jobs - https://phabricator.wikimedia.org/T348758#9601343 (10dcaro) [15:13:46] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q3-Q4), 05Goal, 13Patch-For-Review, 10Puppet (Puppet 7.0): Migrate Cloud VPS puppet infrastructure to Puppet 7 - https://phabricator.wikimedia.org/T351450#9601401 (10Andrew) puppet7 servers need > 1 Gb of RAM or they swap [15:17:35] 10Tool-global-search, 06Discovery-Search, 10Internet-Archive, 10Data-Platform-SRE (2024.03.04 - 2024.03.24): Global-search is showing duplicate results - https://phabricator.wikimedia.org/T359136#9601415 (10bking) 05Open→03Resolved a:03bking This has been fixed. Cloudelastic has slightly different s... [15:17:40] 10Tool-global-search, 06Discovery-Search, 10Internet-Archive, 10Data-Platform-SRE (2024.03.04 - 2024.03.24): Global-search is showing duplicate results - https://phabricator.wikimedia.org/T359136#9601420 (10bking) [15:54:50] 10Toolforge, 13Patch-For-Review: [jobs-cli,toolforge-cli] Add tfj as a shortcut for toolforge-jobs command - https://phabricator.wikimedia.org/T309308#9601659 (10dcaro) [15:54:58] 10Toolforge, 07Kubernetes: [jobs-api] Allow Toolforge scheduled jobs to have a maximum runtime - https://phabricator.wikimedia.org/T306391#9601662 (10dcaro) [15:57:28] 10Cloud Services Proposals, 10Toolforge, 10cloud-services-team (FY2023/2024-Q3-Q4), 05Cloud-Services-Origin-Team, and 4 others: [Epic,builds-api,orchestrator,webservice,jobs-api] Make Toolforge a proper platform as a service with push-to-deploy and build pa... - https://phabricator.wikimedia.org/T194332#9601670 [15:57:36] 10Toolforge, 07Documentation: [builds-builder,docs] Create a tutorial on compiling static frontend assets at build time - https://phabricator.wikimedia.org/T351082#9601674 (10dcaro) [15:59:04] 10Toolforge, 06cloud-services-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [builds-builder,harbor,bulid-service,docs] user-story 11: Add section to admin docs on how to debug the service, how to pin-point the faili... - https://phabricator.wikimedia.org/T325174#9601685 [15:59:20] 10Toolforge, 07Documentation: [docs] Record video tutorial(s) of basic Toolforge access and use - https://phabricator.wikimedia.org/T162654#9601688 (10dcaro) [15:59:48] 10Toolforge (Toolforge iteration 06), 07Documentation: [harbor,docs] Improve Harbor quota handling and docs - https://phabricator.wikimedia.org/T351092#9601692 (10dcaro) [16:00:49] 10Toolforge (Toolforge iteration 06), 07Epic: [jobs-cli,builds-cli,toolforge-cli,webservice] Consolidate the Toolforge CLIs - https://phabricator.wikimedia.org/T356262#9601697 (10dcaro) [16:00:57] 10Toolforge (Toolforge iteration 06): [builds-cli,builds-api] `build quota` fails if tool has no builds - https://phabricator.wikimedia.org/T353701#9601698 (10dcaro) [16:01:05] 10Toolforge (Toolforge iteration 06): [jobs-cli,jobs-api] Allow using file logs with build service images - https://phabricator.wikimedia.org/T353537#9601699 (10dcaro) [16:01:33] 10Toolforge (Toolforge iteration 06), 10cloud-services-team (FY2023/2024-Q3-Q4): [docs] Create a tutorial on how to deploy a Node.js app using Build Service - https://phabricator.wikimedia.org/T353313#9601701 (10dcaro) [16:01:57] 10Toolforge (Toolforge iteration 06), 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [maintain-harbor,docs] Document current setup and admin procedures - https://phabricator.wikimedia.org/T329176#9601703 (10dcaro) [16:04:19] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud [16:05:26] !log taavi@cloudcumin1001 tools END (FAIL) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=99) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud [16:06:35] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.openstack.quota_increase (T357901) [16:06:35] !log taavi@cloudcumin1001 tools END (ERROR) - Cookbook wmcs.openstack.quota_increase (exit_code=97) (T357901) [16:06:39] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.openstack.quota_increase [16:06:47] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) [16:06:57] T357901: Request increased server-group-members quota for tools - https://phabricator.wikimedia.org/T357901 [16:08:28] (PuppetAgentStaleLastRun) firing: Last Puppet run was over 24 hours ago on instance tools-imagebuilder-2 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [16:08:29] !log aborrero@cloudcumin1001 toolsbeta START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [16:08:38] !log aborrero@cloudcumin1001 toolsbeta END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [16:09:16] !log aborrero@cloudcumin1001 tools START - Cookbook wmcs.toolforge.k8s.component.deploy for component jobs-api [16:09:26] !log aborrero@cloudcumin1001 tools END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component jobs-api [16:11:44] !log taavi@cloudcumin1001 tools START - Cookbook wmcs.vps.refresh_puppet_certs on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud [16:12:55] !log taavi@cloudcumin1001 tools END (PASS) - Cookbook wmcs.vps.refresh_puppet_certs (exit_code=0) on tools-imagebuilder-2.tools.eqiad1.wikimedia.cloud [16:13:28] (PuppetAgentStaleLastRun) resolved: Last Puppet run was over 24 hours ago on instance tools-imagebuilder-2 in project tools - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [16:15:03] 10Toolforge: [k8s,builds-api,harbor] Tool (k8s-status or a new one) to display details about buildservice pipelines and Harbor images - https://phabricator.wikimedia.org/T336133#9601817 (10dcaro) [16:25:17] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q3-Q4): [wmcs-backup] exclude_volumes is matching on IDs instead of names - https://phabricator.wikimedia.org/T359192 (10fnegri) [16:26:01] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q3-Q4): [wmcs-backup] exclude_volumes is matching on IDs instead of names - https://phabricator.wikimedia.org/T359192#9601962 (10fnegri) 05Open→03In progress p:05Triage→03Medium a:03fnegri [16:27:02] 10Data-Services, 10cloud-services-team (FY2023/2024-Q3-Q4), 05Goal: [toolsdb] test creating a new replica host - https://phabricator.wikimedia.org/T344717#9601981 (10fnegri) [16:27:50] 10Cloud-VPS, 10Data-Services, 10cloud-services-team (FY2023/2024-Q3-Q4), 13Patch-For-Review: [toolsdb] [cinder] [ceph] Deleting snapshot does not work - https://phabricator.wikimedia.org/T356904#9601970 (10fnegri) The patch above to exclude temp volumes from backup did not work, because of a bug in wmcs-ba... [16:28:06] 10cloud-services-team (FY2023/2024-Q3-Q4), 05Cloud-Services-Origin-Alert, 07Cloud-Services-Worktype-Unplanned, 15User-dcaro: [toolsdb] Copy s51698__yetkin.wanted_items on the replica from the primary - https://phabricator.wikimedia.org/T344420#9601985 (10fnegri) [16:28:22] 14cloud-services-team (FY2023/2024-Q1-Q2), 05Cloud-Services-Origin-Alert, 07Cloud-Services-Worktype-Unplanned, 13Patch-For-Review, 15User-dcaro: [toolsdb] ToolsDB replication is broken on tools-db-2 (errno 1032) - 2023-08-17 - https://phabricator.wikimedia.org/T344411#9601986 (10fnegri) [16:28:30] 10Data-Services, 06cloud-services-team: ToolsDB: simplify volume chain - https://phabricator.wikimedia.org/T335593#9601987 (10fnegri) [16:28:40] 10Data-Services, 10cloud-services-team (FY2023/2024-Q3-Q4), 05Goal: [toolsdb] test creating a new replica host - https://phabricator.wikimedia.org/T344717#9601984 (10fnegri) 05In progress→03Stalled [16:28:56] 10Cloud-VPS, 10Data-Services, 10cloud-services-team (FY2023/2024-Q3-Q4), 13Patch-For-Review: [toolsdb] [cinder] [ceph] Deleting snapshot does not work - https://phabricator.wikimedia.org/T356904#9601980 (10fnegri) 05In progress→03Stalled [16:30:25] 10Toolforge (Toolforge iteration 06), 13Patch-For-Review, 15User-aborrero: toolforge: introduce OpenAPI to jobs framework - https://phabricator.wikimedia.org/T356523#9601972 (10CodeReviewBot) aborrero opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/65 job: adjust max job... [16:37:52] 10Cloud-VPS (Quota-requests): Request for more compute and storage for the GLAMS dashboard project - https://phabricator.wikimedia.org/T358477#9602074 (10dcaro) +1 [16:38:54] 10Toolforge (Software install/update): [builds-builder] Request for supporting Deno on Toolforge - https://phabricator.wikimedia.org/T253470#9602076 (10dcaro) [16:41:11] !log fnegri@cloudcumin1001 glamwikidashboard START - Cookbook wmcs.openstack.quota_increase (T358477) [16:41:16] T358477: Request for more compute and storage for the GLAMS dashboard project - https://phabricator.wikimedia.org/T358477 [16:41:20] !log fnegri@cloudcumin1001 glamwikidashboard END (PASS) - Cookbook wmcs.openstack.quota_increase (exit_code=0) (T358477) [16:43:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [16:53:06] 10Cloud-VPS (Quota-requests): Request for more compute and storage for the GLAMS dashboard project - https://phabricator.wikimedia.org/T358477#9602170 (10fnegri) 05Open→03Resolved a:03fnegri [16:57:51] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q3-Q4): [wmcs-cookbook] increase_quota cookbook fails - https://phabricator.wikimedia.org/T352840#9602226 (10fnegri) 05Open→03Resolved This was fixed in https://gerrit.wikimedia.org/r/c/cloud/wmcs-cookbooks/+/992085 [17:08:50] 10Toolforge (Toolforge iteration 06), 13Patch-For-Review: [harbor, maintain-harbor] We seem to be cleaning up image tags that should not be cleaned up for the toolforge project - https://phabricator.wikimedia.org/T359052#9602299 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/... [17:12:49] 10Toolforge (Toolforge iteration 07), 10cloud-services-team (FY2023/2024-Q3-Q4), 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [maintain-kubeusers] Allow setting the requests cpu and mem quota - https://phabricator.wikimedia.org/T357881#9602354 (10dcaro) [17:12:52] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [builds-api,jobs-api,envvars-api,api-gateway] FIgure out and document how to do non-backwards compatible changes - https://phabricator.wikimedia.org/T356974#9602352 (10dcaro) [17:12:57] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [harbor, maintain-harbor] We seem to be cleaning up image tags that should not be cleaned up for the toolforge project - https://phabricator.wikimedia.org/T359052#9602350 (10dcaro) [17:13:37] 10Toolforge (Toolforge iteration 07): [jobs-cli,jobs-api] Allow using file logs with build service images - https://phabricator.wikimedia.org/T353537#9602356 (10dcaro) [17:13:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:14:09] 10Toolforge (Toolforge iteration 07), 07Documentation: [harbor,docs] Improve Harbor quota handling and docs - https://phabricator.wikimedia.org/T351092#9602358 (10dcaro) [17:14:17] 10Toolforge (Toolforge iteration 07), 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [maintain-harbor,docs] Document current setup and admin procedures - https://phabricator.wikimedia.org/T329176#9602362 (10dcaro) [17:14:20] 10Toolforge (Toolforge iteration 07), 06cloud-services-team, 05Cloud-Services-Origin-Team, 07Cloud-Services-Worktype-Project, 15User-dcaro: [builds-api,orchestration] Automatically deploy the webservice when the image is built - https://phabricator.wikimedia.org/T341065#9602360 (10dcaro) [17:14:29] 10Toolforge (Toolforge iteration 07): Support monorepos with the Multi Procfile buildpack - https://phabricator.wikimedia.org/T355329#9602366 (10dcaro) [17:14:39] 10Toolforge (Toolforge iteration 07): [builds-builder,jobs-api] Calling nontrivial Procfile commands with arguments results in confusing error (“no such file or directory”) - https://phabricator.wikimedia.org/T356016#9602364 (10dcaro) [17:14:52] 10Toolforge (Toolforge iteration 07): [apt-buildpak] Some APT packages are not installed during the image build - https://phabricator.wikimedia.org/T355252#9602368 (10dcaro) [17:15:04] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review, 07Upstream: [maintain-harbor] Manage project quotas via maintain-harbor - https://phabricator.wikimedia.org/T352417#9602377 (10dcaro) [17:15:21] 10Toolforge (Toolforge iteration 07): [builds-cli,builds-api] `build quota` fails if tool has no builds - https://phabricator.wikimedia.org/T353701#9602372 (10dcaro) [17:15:38] 10Toolforge (Toolforge iteration 07): [harbor] upgrade to 2.10.x - https://phabricator.wikimedia.org/T354507#9602374 (10dcaro) [17:15:47] 10Toolforge (Toolforge iteration 07): [tbs] Unable to get pywikibot + wget on a python build service image - https://phabricator.wikimedia.org/T354157#9602370 (10dcaro) [17:16:21] 10Toolforge (Toolforge iteration 07), 10Toolforge Jobs framework, 13Patch-For-Review: Support job health checks - https://phabricator.wikimedia.org/T335592#9602385 (10dcaro) [17:16:23] 10Toolforge (Toolforge iteration 07): Upgrade Toolforge image builder to Bookworm - https://phabricator.wikimedia.org/T358483#9602381 (10dcaro) [17:16:27] 10Toolforge (Toolforge iteration 07): dbreps job pending to start for 2d16h on Toolforge - https://phabricator.wikimedia.org/T358175#9602379 (10dcaro) [17:16:48] 10Toolforge (Toolforge iteration 07), 15User-Raymond_Ndibe: [toolforge-cd] remove duplicated run on tag and push to master (just do one if possible) - https://phabricator.wikimedia.org/T353563#9602391 (10dcaro) [17:17:24] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review: [maintain-harbor] Improvements to subcommands and config validation - https://phabricator.wikimedia.org/T353059#9602383 (10dcaro) [17:17:36] 10Toolforge (Toolforge iteration 07), 06cloud-services-team: Harbor uploads sometimes fail due to tmpfs space on project-proxy - https://phabricator.wikimedia.org/T354116#9602395 (10dcaro) [17:17:42] 10Toolforge (Toolforge iteration 07), 06cloud-services-team, 07Kubernetes, 15User-aborrero: toolforge k8s: some static pods needs manual restart - https://phabricator.wikimedia.org/T358476#9602396 (10dcaro) [17:17:44] 10Toolforge (Toolforge iteration 07), 15User-aborrero: [toolforge] several tools get periods of connection refused (104) when connecting to wikis - https://phabricator.wikimedia.org/T356164#9602389 (10dcaro) [17:17:46] 10Toolforge (Toolforge iteration 07): [k8s] Add node anti-affinity topologySpreadConstraints to infrastructure components where relevant - https://phabricator.wikimedia.org/T358203#9602398 (10dcaro) [17:17:54] 10Toolforge (Toolforge iteration 07), 06cloud-services-team, 15User-aborrero: Upgrade Toolforge Kubernetes to version 1.24 - https://phabricator.wikimedia.org/T307651#9602397 (10dcaro) [17:18:02] 10Toolforge (Toolforge iteration 07): [builds-api,envvars-api] bump the version in the openapi definition when bumping the package version - https://phabricator.wikimedia.org/T356972#9602400 (10dcaro) [17:18:10] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review, 15User-aborrero: [toolforge API] expose all backend APIs OpenAPI specs - https://phabricator.wikimedia.org/T358100#9602399 (10dcaro) [17:18:18] 10Toolforge (Toolforge iteration 07), 15User-aborrero: [toolforge] simplify calling the different toolforge apis from within the containers - https://phabricator.wikimedia.org/T356377#9602401 (10dcaro) [17:18:26] 10Toolforge (Toolforge iteration 07), 07Epic: [jobs-cli,builds-cli,toolforge-cli,webservice] Consolidate the Toolforge CLIs - https://phabricator.wikimedia.org/T356262#9602402 (10dcaro) [17:18:42] 10Toolforge (Toolforge iteration 07): [Toolforge CLI consolidation] Explore OpenAPI SDK tooling - https://phabricator.wikimedia.org/T356261#9602405 (10dcaro) [17:18:58] 10Toolforge (Toolforge iteration 07), 10cloud-services-team (FY2023/2024-Q3-Q4): [docs] Create a tutorial on how to deploy a Node.js app using Build Service - https://phabricator.wikimedia.org/T353313#9602406 (10dcaro) [17:19:06] 10Toolforge (Toolforge iteration 07), 10cloud-services-team (FY2023/2024-Q3-Q4): [builds-api] Add dashboards with the new statistics - https://phabricator.wikimedia.org/T352764#9602407 (10dcaro) [17:19:14] 10Toolforge (Toolforge iteration 07), 10cloud-services-team (FY2023/2024-Q3-Q4), 05Cloud-Services-Origin-User, 07Cloud-Services-Worktype-Maintenance, 15User-dcaro: [webservice] Error shown when restarting buildpack-based tool - https://phabricator.wikimedia.org/T348312#9602408 (10dcaro) [17:19:22] 10Toolforge (Toolforge iteration 07), 06cloud-services-team: Migrate remaining tools off Gridengine - https://phabricator.wikimedia.org/T313405#9602409 (10dcaro) [17:19:30] 10Wikibugs: Set a timeout for phorge requests - https://phabricator.wikimedia.org/T359145#9602412 (10bd808) The [[https://pypi.org/project/fab/|upstream "fab" library]] that is currently being used is backed by the well-known Requests library, so adding both connect and read timeouts should be possible. See http... [17:20:10] 10Toolforge (Toolforge iteration 07): [k8s] Add node anti-affinity topologySpreadConstraints to infrastructure components where relevant - https://phabricator.wikimedia.org/T358203#9602423 (10dcaro) [17:20:28] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review, 15User-aborrero: [toolforge API] Investigate ways to present our multiple Openapi definitions to a future consolidated CLI client - https://phabricator.wikimedia.org/T354745#9602387 (10dcaro) [17:20:36] 10Wikibugs, 15User-bd808: Frequent "Redis listener crashed; restarting in a few seconds." errors logged - https://phabricator.wikimedia.org/T359097#9602419 (10bd808) 05Open→03In progress a:03bd808 [17:20:44] 10Toolforge, 06cloud-services-team: [infra] Fix the mis-named k8s service in tools and toolsbeta projects - https://phabricator.wikimedia.org/T262562#9602433 (10dcaro) [17:20:53] 10Toolforge, 06cloud-services-team, 07Kubernetes: [infra] Upgrade Toolforge K8s haproxies to Bookworm - https://phabricator.wikimedia.org/T349206#9602434 (10dcaro) [17:21:01] 10Toolforge, 06cloud-services-team, 07Kubernetes: [infra] Remove TTLAfterFinished from config before upgrade to 1.25 - https://phabricator.wikimedia.org/T349197#9602436 (10dcaro) [17:21:09] 10Toolforge, 06cloud-services-team, 07Kubernetes: [infra] Upgrade Toolforge K8s etcd nodes to Bookworm - https://phabricator.wikimedia.org/T349207#9602435 (10dcaro) [17:21:17] 10Toolforge: [infra,builds-api,harbor] Tool (k8s-status or a new one) to display details about buildservice pipelines and Harbor images - https://phabricator.wikimedia.org/T336133#9602437 (10dcaro) [17:22:01] 10Toolforge, 07Kubernetes: [infra] kubectl is quite slow the “first time” per user account - https://phabricator.wikimedia.org/T358976#9602449 (10dcaro) [17:23:05] 10Toolforge: [envvars-cli] Enable use of `toolforge envvar` managed data on bastions - https://phabricator.wikimedia.org/T358537#9602455 (10dcaro) [17:23:41] (CloudVPSDesignateLeaks) firing: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:23:57] 10Toolforge (Toolforge iteration 07), 13Patch-For-Review, 15User-aborrero: toolforge: introduce OpenAPI to jobs framework - https://phabricator.wikimedia.org/T356523#9602393 (10dcaro) [17:28:41] (CloudVPSDesignateLeaks) resolved: (2) Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [17:29:47] 10Wikibugs: Add support for alternate channels files to make testing/debugging easier - https://phabricator.wikimedia.org/T359202 (10bd808) [17:47:52] 10Tool-global-search, 06Discovery-Search, 10Internet-Archive, 10Data-Platform-SRE (2024.03.04 - 2024.03.24): Global-search is showing duplicate results - https://phabricator.wikimedia.org/T359136#9602588 (10EBernhardson) We are in the process of deploying a new updater for CirrusSearch, with cloudelastic a... [17:52:28] (PuppetCertificateAboutToExpire) firing: Puppet CA certificate Puppet CA: cloudinfra-internal-puppetmaster01.cloudinfra.eqiad.wmflabs is about to expire in 26d 2h 44m 11s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [17:54:29] 05Grid-Engine-to-K8s-Migration: Migrate huggle from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319797#9602617 (10Petrb) yes it works now [17:56:00] 05Grid-Engine-to-K8s-Migration: Migrate huggle from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319797#9602621 (10Petrb) 05Open→03Resolved I believe it's migrated now [18:36:05] 10Wikibugs, 13Patch-For-Review, 15User-bd808: Frequent "Redis listener crashed; restarting in a few seconds." errors logged - https://phabricator.wikimedia.org/T359097#9602780 (10CodeReviewBot) bd808 opened https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/9 irc: Mark Redis2Irc.privmsg... [18:39:25] 10Tool-bub2, 10Internet-Archive, 10Outreach-Programs-Projects, 10Outreachy (Round 27), 13Patch-For-Review: Integrate Wikimedia Ecosystem within BUB2 tool - https://phabricator.wikimedia.org/T346386#9602819 (10Pppery) [18:43:41] (CloudVPSDesignateLeaks) firing: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:44:29] 10Wikibugs, 13Patch-For-Review, 15User-bd808: Frequent "Redis listener crashed; restarting in a few seconds." errors logged - https://phabricator.wikimedia.org/T359097#9602845 (10CodeReviewBot) bd808 merged https://gitlab.wikimedia.org/toolforge-repos/wikibugs2/-/merge_requests/9 irc: Mark Redis2Irc.privmsg... [18:53:41] (CloudVPSDesignateLeaks) resolved: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [18:58:55] 10Wikibugs, 13Patch-For-Review, 15User-bd808: Frequent "Redis listener crashed; restarting in a few seconds." errors logged - https://phabricator.wikimedia.org/T359097#9602874 (10bd808) 05In progress→03Resolved [18:59:06] 10Wikibugs, 13Patch-For-Review, 15User-bd808: Frequent "Redis listener crashed; restarting in a few seconds." errors logged - https://phabricator.wikimedia.org/T359097#9602880 (10bd808) >>! In T359097#9599234, @bd808 wrote: > Now the burning question is if the proper fix is marking `bot.privmsg_many` as asyn... [19:03:48] 10Tool-phab-ban: Add a "reason" field in phab-ban tool - https://phabricator.wikimedia.org/T359211 (10Bugreporter) [19:05:23] 10Tool-phab-ban: Add a "reason" field in phab-ban tool - https://phabricator.wikimedia.org/T359211#9602896 (10Bugreporter) [19:10:31] 10Tool-phab-ban: Add a "reason" field in phab-ban tool - https://phabricator.wikimedia.org/T359211#9602928 (10bd808) @Bugreporter There is no similar comment in the log for blocking done directly in Phabricator, so I guess I'm wondering what you think people will write as a reason other than "spamming". [19:11:23] 10Tool-phab-ban: Add a "reason" field in phab-ban tool - https://phabricator.wikimedia.org/T359211#9602931 (10Bugreporter) I will say more in a following task, but see also T102576#1718402 [19:12:28] (PuppetAgentNoResources) firing: No Puppet resources found on instance metricsinfra-puppet-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [19:16:11] 10Tool-phab-ban: Add a "reason" field in phab-ban tool - https://phabricator.wikimedia.org/T359211#9602953 (10jrbs) >>! In T359211#9602928, @bd808 wrote: > @Bugreporter There is no similar comment in the log for blocking done directly in Phabricator, so I guess I'm wondering what you think people will write as a... [19:18:01] 10Tool-phab-ban: Add a "reason" field in phab-ban tool - https://phabricator.wikimedia.org/T359211#9602961 (10Bugreporter) >>! In T359211#9602953, @jrbs wrote: >>>! In T359211#9602928, @bd808 wrote: >> @Bugreporter There is no similar comment in the log for blocking done directly in Phabricator, so I guess I'm w... [19:21:08] 10Tool-phab-ban, 10Phabricator: Phabricator admins should have access to phab-ban tool - https://phabricator.wikimedia.org/T359212 (10Bugreporter) [19:21:35] 10Tool-phab-ban, 10Phabricator: Phabricator admins should have access to phab-ban tool - https://phabricator.wikimedia.org/T359212#9602985 (10Bugreporter) [19:23:03] 10Tool-phab-ban, 10Phabricator: Phabricator admins should have access to phab-ban tool - https://phabricator.wikimedia.org/T359212#9602991 (10Bugreporter) [19:41:48] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q3-Q4), 06DC-Ops, 06SRE, 10ops-eqiad: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643#9603028 (10Andrew) Notes from today's (unproductive) meeting: We met with several Dell reps including an engineer n... [19:51:14] 10Tool-phab-ban, 10Phabricator: Phabricator admins should have access to phab-ban tool - https://phabricator.wikimedia.org/T359212#9603045 (10Bugreporter) [19:52:00] 10Tool-phab-ban, 10Phabricator: Phabricator admins should have access to phab-ban tool - https://phabricator.wikimedia.org/T359212#9603046 (10Bugreporter) [20:13:57] 10Data-Services: Make Dispenser's principle_links table accessible in new Wiki replica cluster - https://phabricator.wikimedia.org/T180636#9603085 (10Superyetkin) [20:14:07] 10Data-Services, 14cloud-services-team (FY2017-18), 06DBA, 05Goal, and 2 others: Migrate all users to new Wiki Replica cluster and decommission old hardware - https://phabricator.wikimedia.org/T142807#9603086 (10Superyetkin) [20:14:35] 10Data-Services, 06cloud-services-team, 06Data-Engineering-Icebox: Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#9603083 (10Superyetkin) 05Declined→03Open This needs to stay open. What do we need to do in order to create the "da... [20:16:25] 10Tool-phab-ban: Add a "reason" field in phab-ban tool - https://phabricator.wikimedia.org/T359211#9603087 (10bd808) [20:16:29] 10Tool-phab-ban, 10Phabricator: Phabricator admins should have access to phab-ban tool - https://phabricator.wikimedia.org/T359212#9603088 (10bd808) [20:24:30] 10Tool-phab-ban, 10Phabricator: Phabricator admins should have access to phab-ban tool - https://phabricator.wikimedia.org/T359212#9603094 (10Pppery) Nit: Blocking an account on MediaWiki.org does not AFAIK disable an account on Phabricator. This proposal to make everyone beat around the bush by using the bot... [20:32:49] 10Data-Services, 06cloud-services-team, 06Data-Engineering-Icebox: Implement technical details and process for "datasets_p" on wikireplica hosts - https://phabricator.wikimedia.org/T173511#9603122 (10bd808) >>! In T173511#9603083, @Superyetkin wrote: > This needs to stay open. I find your one drive-by quest... [20:36:10] 10Tool-phab-ban, 10Phabricator: Phabricator admins should have access to phab-ban tool - https://phabricator.wikimedia.org/T359212#9603136 (10Bugreporter) [20:39:38] 10Tool-phab-ban, 10Phabricator: Phabricator admins should have access to phab-ban tool - https://phabricator.wikimedia.org/T359212#9603144 (10Bugreporter) I have changed the description (though "Vandalism or spamming on Phabricator" [[https://www.mediawiki.org/wiki/MediaWiki:Ipbreason-dropdown|is listed in Med... [20:46:49] (03PS1) 10Majavah: Fix MediaWiki dependency installation issues [labs/striker] - 10https://gerrit.wikimedia.org/r/1008948 [20:57:28] (PuppetCertificateAboutToExpire) firing: Puppet CA certificate Puppet CA: cloudinfra-internal-puppetmaster01.cloudinfra.eqiad.wmflabs is about to expire in 25d 23h 38m 11s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [21:01:23] 10Tool-phab-ban, 10Phabricator: Phabricator admins should have access to phab-ban tool - https://phabricator.wikimedia.org/T359212#9603250 (10Peachey88) >>! In T359212#9603144, @Bugreporter wrote: > (though "Vandalism or spamming on Phabricator" [[https://www.mediawiki.org/wiki/MediaWiki:Ipbreason-dropdown|is... [21:18:12] 10Striker: Update Django version used in Striker - https://phabricator.wikimedia.org/T359217 (10taavi) [21:27:28] (PuppetAgentNoResources) resolved: No Puppet resources found on instance metricsinfra-puppet-2 on project metricsinfra - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentNoResources [21:31:20] 10Tool-phab-ban, 10Phabricator, 06collaboration-services: Phabricator admins should have access to phab-ban tool - https://phabricator.wikimedia.org/T359212#9603393 (10Dzahn) [21:51:22] (03PS1) 10Majavah: Drop Nose testing setup [labs/striker] - 10https://gerrit.wikimedia.org/r/1008957 [21:51:24] (03PS1) 10Majavah: Upgrade base container to Debian Bookworm [labs/striker] - 10https://gerrit.wikimedia.org/r/1008958 [21:53:18] (03PS1) 10Majavah: Require SUL/Phab links before applying for access [labs/striker] - 10https://gerrit.wikimedia.org/r/1008960 (https://phabricator.wikimedia.org/T172899) [21:54:47] 10Tool-phab-ban, 10Phabricator, 06collaboration-services: Phabricator admins should have access to phab-ban tool - https://phabricator.wikimedia.org/T359212#9603429 (10bd808) >>! In T359212#9603091, @Pppery wrote: > Nit: Blocking an account on MediaWiki.org does not AFAIK disable an account on Phabricator. T... [22:01:25] 10Striker: Update Django version used in Striker - https://phabricator.wikimedia.org/T359217#9603458 (10bd808) This has been "on my list" for a long time. The newest 4.2 LTS would be my preference for the target version. This upgrade will likely require quite a bit of manual testing. [22:12:24] 10Wikibugs, 15User-bd808: wikibugs having a hard time staying connected to libera.chat IRC network - https://phabricator.wikimedia.org/T357729#9603506 (10bd808) >>! In T357729#9599627, @MoritzMuehlenhoff wrote: > Some random observation: Some IRC notifications are notably delayed. I merged https://gerrit.wikim... [22:26:05] 10Wikibugs, 15User-bd808: Set a timeout for phorge requests - https://phabricator.wikimedia.org/T359145#9603530 (10bd808) 05Open→03In progress a:03bd808 [22:39:54] 10Wikibugs, 15User-bd808: Set a timeout for phorge requests - https://phabricator.wikimedia.org/T359145#9603560 (10bd808) >>! In T359145#9602411, @bd808 wrote: > The `timeout=...` argument would be passed to `self.phab.request(...)` calls in wikibugs2.phorge.PhorgeFeedReader. Ugh. I thought at first glance th... [22:50:35] 10Toolforge (Toolforge iteration 07), 06cloud-services-team: Harbor uploads sometimes fail due to tmpfs space on project-proxy - https://phabricator.wikimedia.org/T354116#9603580 (10Raymond_Ndibe) @dcaro do you have any idea on how to reproduce this issue? [23:06:04] (03PS1) 10LWatson: releases: Bump Codex to 1.3.4 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1008972 [23:12:20] 10Cloud-VPS (Quota-requests): Floating IP request for project Openvas - https://phabricator.wikimedia.org/T356830#9603666 (10bd808) [23:12:24] 10Cloud-VPS (Project-requests): Request creation of OpenVAS VPS project - https://phabricator.wikimedia.org/T354192#9603667 (10bd808) [23:15:52] (03CR) 10Eric Gardner: [C: 03+2] releases: Bump Codex to 1.3.4 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1008972 (owner: 10LWatson) [23:16:26] (03Merged) 10jenkins-bot: releases: Bump Codex to 1.3.4 [labs/libraryupgrader/config] - 10https://gerrit.wikimedia.org/r/1008972 (owner: 10LWatson) [23:57:28] (PuppetCertificateAboutToExpire) firing: Puppet CA certificate Puppet CA: cloudinfra-internal-puppetmaster01.cloudinfra.eqiad.wmflabs is about to expire in 25d 20h 38m 11s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire