[00:12:03] (InstanceDown) firing: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [00:17:03] (InstanceDown) resolved: Project tf-infra-test instance tf-infra-test is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [02:10:16] (NodeTextfileStale) firing: Stale textfile for cloudcontrol2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [02:40:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [02:45:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [04:18:56] (ToolsToolsDBWritableState) firing: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [04:18:56] (ToolsToolsDBReplicationError) firing: ToolsDB replication is broken on tools-db-2 (errno 2003) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationError [04:28:56] (ToolsToolsDBReplicationError) resolved: ToolsDB replication is broken on tools-db-2 (errno 2003) - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsDBReplication - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBReplicationError [04:33:56] (ToolsToolsDBWritableState) resolved: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [04:37:21] 10cloud-services-team (FY2023/2024-Q1-Q2), 10Cloud-Services-Origin-Alert, 10Cloud-Services-Worktype-Unplanned, 10Patch-For-Review, 10User-dcaro: [toolsdb] MariaDB process is killed by OOM killer (December 2023) - https://phabricator.wikimedia.org/T353093 (10fnegri) Another crash a few minutes ago: ` [Fr... [04:57:54] 10Grid-Engine-to-K8s-Migration: Migrate wordpile from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T320184 (10komla) The maintainer (@Asaf), has requested for more time. [04:58:23] 10Grid-Engine-to-K8s-Migration: Migrate glamify from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319772 (10komla) The maintainer (@Asaf), has requested for more time. [04:58:36] 10Grid-Engine-to-K8s-Migration: Migrate erex-yomi from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319727 (10komla) The maintainer (@Asaf), has requested for more time. [05:40:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [05:45:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [06:10:16] (NodeTextfileStale) firing: Stale textfile for cloudcontrol2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [07:43:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [07:48:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1006.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [08:45:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [08:45:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [09:07:56] 10Toolforge, 10Tools: Toolforge: grid: sqid tool using a lot of resources - https://phabricator.wikimedia.org/T330146 (10Mmarx) 05Open→03Resolved a:03Mmarx As the job is no longer running on the grid (nor using the JVM, for that matter), I guess this is now resolved. [09:54:56] (ToolsToolsDBWritableState) firing: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [09:59:56] (ToolsToolsDBWritableState) resolved: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [10:10:17] (NodeTextfileStale) firing: Stale textfile for cloudcontrol2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [10:23:25] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review, 10User-Raymond_Ndibe: [gitlab,toolforge-deploy] Create a process to open an MR to toolforge-deploy when a new release ofa component happens - https://phabricator.wikimedia.org/T347392 (10Raymond_Ndibe) [10:23:27] 10Toolforge (Toolforge iteration 02): [toolforge,gitlab] ensure we have a release before creating the mr on toolforge-deploy - https://phabricator.wikimedia.org/T353425 (10Raymond_Ndibe) 05Open→03In progress [10:54:56] (ToolsToolsDBWritableState) firing: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [10:59:56] (ToolsToolsDBWritableState) resolved: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [11:05:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [11:08:19] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q1-Q2), 10DC-Ops, 10SRE, 10ops-eqiad: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643 (10dcaro) I got some graphs here with the increases of the error count: https://grafana-rw.wikimedia.org/d/P1tFnn3Mk... [11:10:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [11:11:21] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q1-Q2), 10DC-Ops, 10SRE, 10ops-eqiad: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643 (10dcaro) That might be all of them since nov 28th: ` dcaro@urcuchillay$ head -n 1 Hard\ drives\ health\ -\ increase... [11:21:33] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [toolforge,gitlab] ensure we have a release before creating the mr on toolforge-deploy - https://phabricator.wikimedia.org/T353425 (10CodeReviewBot) raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/24 [toolf... [11:45:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [11:45:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [11:54:11] 10Toolforge (Toolforge iteration 02): [toolforge-cd] minor gitlab-ci refactor - https://phabricator.wikimedia.org/T353514 (10Raymond_Ndibe) [11:54:25] 10Toolforge (Toolforge iteration 02): [toolforge-cd] minor gitlab-ci refactor - https://phabricator.wikimedia.org/T353514 (10Raymond_Ndibe) 05Open→03In progress [12:07:21] 10Cloud-VPS, 10cloud-services-team (FY2023/2024-Q1-Q2), 10DC-Ops, 10SRE, 10ops-eqiad: cloudcephosd1021-1034: hard drive sector errors increasing - https://phabricator.wikimedia.org/T348643 (10dcaro) Got all the increments since we have data in prometheus 29th of Nov (data dump from grafana + a bit of pan... [12:08:37] (CephSlowOps) firing: Ceph cluster in eqiad has 130 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [12:08:43] 10cloud-services-team: CephSlowOps Ceph cluster in eqiad has slow ops, which might be blocking some writes - https://phabricator.wikimedia.org/T352570 (10phaultfinder) [12:13:37] (CephSlowOps) resolved: Ceph cluster in eqiad has 9 slow ops - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/CephSlowOps - https://grafana.wikimedia.org/d/P1tFnn3Mk/wmcs-ceph-eqiad-health?orgId=1&search=open&tag=ceph&tag=health&tag=WMCS - https://alerts.wikimedia.org/?q=alertname%3DCephSlowOps [12:46:32] 10Toolforge (Toolforge iteration 02): [toolforge-cd] minor gitlab-ci refactor - https://phabricator.wikimedia.org/T353514 (10Raymond_Ndibe) context here https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/24#987f84935efdd598ba3f9ac7f7041068b72ab57b [12:46:49] 10Toolforge (Toolforge iteration 02): [toolforge-cd] gitlab-ci refactor - https://phabricator.wikimedia.org/T353514 (10Raymond_Ndibe) [13:06:12] 10cloud-services-team, 10Infrastructure-Foundations, 10SRE, 10netbox, 10Patch-For-Review: Netbox: Add support for our complex host network setups in provision script - https://phabricator.wikimedia.org/T346428 (10cmooney) @ayounsi, @Volans I have uploaded the above patch to add the functionality as descr... [13:06:58] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [builds-cli,builds-api] Allow build service to cleanup images to free quota - https://phabricator.wikimedia.org/T341067 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-api/-/merge_requests/66 clean: add new endp... [13:08:24] (NodeTextfileStale) resolved: Stale textfile for cloudcontrol2004-dev:9100 - https://wikitech.wikimedia.org/wiki/Prometheus#Stale_file_for_node-exporter_textfile - https://grafana.wikimedia.org/d/knkl4dCWz/node-exporter-textfile - https://alerts.wikimedia.org/?q=alertname%3DNodeTextfileStale [13:15:20] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [builds-cli,builds-api] Allow build service to cleanup images to free quota - https://phabricator.wikimedia.org/T341067 (10CodeReviewBot) project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolfor... [13:17:14] 10cloud-services-team, 10Infrastructure-Foundations, 10SRE, 10netbox, 10Patch-For-Review: Netbox: Add support for our complex host network setups in provision script - https://phabricator.wikimedia.org/T346428 (10cmooney) p:05Triage→03Medium [13:17:49] !log toolsbeta dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T341067) [13:17:52] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [13:17:53] T341067: [builds-cli,builds-api] Allow build service to cleanup images to free quota - https://phabricator.wikimedia.org/T341067 [13:18:20] !log toolsbeta dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T341067) [13:18:24] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Toolsbeta/SAL [13:29:58] 10Toolforge (Toolforge iteration 02): [builds-cli] delete --all gets 0 builds to delete - https://phabricator.wikimedia.org/T353519 (10dcaro) [13:31:12] 10Toolforge (Toolforge iteration 02): [builds-cli] delete --all gets 0 builds to delete - https://phabricator.wikimedia.org/T353519 (10dcaro) Deleting just one with id works, and it does not seem to matter the status of the build (I had two other builds before in `ok` status, that I deleted by id). [13:32:51] !log tools dcaro@urcuchillay START - Cookbook wmcs.toolforge.k8s.component.deploy for component builds-api (T341067) [13:32:56] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:32:56] T341067: [builds-cli,builds-api] Allow build service to cleanup images to free quota - https://phabricator.wikimedia.org/T341067 [13:33:27] !log tools dcaro@urcuchillay END (PASS) - Cookbook wmcs.toolforge.k8s.component.deploy (exit_code=0) for component builds-api (T341067) [13:33:30] Logged the message at https://wikitech.wikimedia.org/wiki/Nova_Resource:Tools/SAL [13:38:33] 10Toolforge (Software install/update), 10Wikimedia-SVG-rendering: Install svgo - https://phabricator.wikimedia.org/T295694 (10taavi) 05Open→03Declined I'm closing all grid engine install/update tasks as the [[ grid engine is being removed | https://wikitech.wikimedia.org/wiki/News/Toolforge_Grid_Engine_dep... [13:38:35] 10Toolforge (Software install/update): Install ipython3 on Toolforge - https://phabricator.wikimedia.org/T288876 (10taavi) 05Open→03Declined I'm closing all grid engine install/update tasks as the [[ grid engine is being removed | https://wikitech.wikimedia.org/wiki/News/Toolforge_Grid_Engine_deprecation ]].... [13:38:48] 10Toolforge (Software install/update): Install wget2 on Toolforge hosts - https://phabricator.wikimedia.org/T306144 (10taavi) 05Open→03Declined I'm closing all grid engine install/update tasks as the [[ grid engine is being removed | https://wikitech.wikimedia.org/wiki/News/Toolforge_Grid_Engine_deprecation... [13:39:07] 10Toolforge (Software install/update): Install libmariadbclient-dev-compat - https://phabricator.wikimedia.org/T304848 (10taavi) 05Open→03Declined I'm closing all grid engine install/update tasks as the [[ grid engine is being removed | https://wikitech.wikimedia.org/wiki/News/Toolforge_Grid_Engine_deprecati... [13:39:19] 10Toolforge, 10Tracking-Neverending, 10User-bd808: Packages to be added to toollabs puppet - https://phabricator.wikimedia.org/T55704 (10taavi) The #Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it wit... [13:39:25] 10Toolforge (Software install/update): Please add libgnutls-dev to toolforge - https://phabricator.wikimedia.org/T219219 (10taavi) 05Open→03Declined I'm closing all grid engine install/update tasks as the [[ grid engine is being removed | https://wikitech.wikimedia.org/wiki/News/Toolforge_Grid_Engine_depreca... [13:39:34] 10Toolforge (Software install/update): toolforge: Consider installing python-passlib on grid hosts - https://phabricator.wikimedia.org/T246048 (10taavi) 05Open→03Declined I'm closing all grid engine install/update tasks as the [[ grid engine is being removed | https://wikitech.wikimedia.org/wiki/News/Toolfor... [13:39:40] 10Toolforge (Software install/update): Installer Bundler for managing Ruby application dependencies - https://phabricator.wikimedia.org/T127685 (10taavi) 05Open→03Declined I'm closing all grid engine install/update tasks as the [[ grid engine is being removed | https://wikitech.wikimedia.org/wiki/News/Toolfo... [13:40:19] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [builds-cli,builds-api] Allow build service to cleanup images to free quota - https://phabricator.wikimedia.org/T341067 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/37 clean: add subcomma... [13:40:24] 10Toolforge (Software install/update), 10Pywikibot: Pywikibot 8 requires mwparserfromhell, which is not present on the Toolforge grid - https://phabricator.wikimedia.org/T327600 (10taavi) 05Open→03Declined I'm closing all grid engine install/update tasks as the [[ grid engine is being removed | https://wik... [13:40:31] 10Toolforge (Software install/update), 10cloud-services-team, 10Pywikibot: Install setuptools >= 49.4.0 on Toolforge - https://phabricator.wikimedia.org/T284299 (10taavi) 05Open→03Declined I'm closing all grid engine install/update tasks as the [[ grid engine is being removed | https://wikitech.wikimedia... [13:40:35] 10Toolforge (Software install/update), 10cloud-services-team: Headless browser - https://phabricator.wikimedia.org/T282597 (10taavi) 05Open→03Declined I'm closing all grid engine install/update tasks as the [[ grid engine is being removed | https://wikitech.wikimedia.org/wiki/News/Toolforge_Grid_Engine_dep... [13:40:37] 10Toolforge (Software install/update): Support ASGI on Toolforge - https://phabricator.wikimedia.org/T296729 (10taavi) 05Open→03Resolved [13:43:14] 10Toolforge (Software install/update): Provide a PHP 8.2 image for Kubernetes toolforge-jobs and webservice - https://phabricator.wikimedia.org/T335352 (10taavi) [13:43:40] 10Toolforge (Software install/update): Build Bookworm based Toolforge Kubernetes images - https://phabricator.wikimedia.org/T335507 (10taavi) 05Open→03Resolved This is all done, unless I'm missing something. [13:45:48] 10Toolforge (Software install/update): Update requests to 2.31.0 on toolforge - https://phabricator.wikimedia.org/T347040 (10taavi) 05Open→03Resolved a:03taavi I'm closing all grid engine install/update tasks as the [[ https://wikitech.wikimedia.org/wiki/News/Toolforge_Grid_Engine_deprecation | grid engine... [13:45:58] 10Toolforge (Software install/update): Update requests to 2.31.0 on toolforge - https://phabricator.wikimedia.org/T347040 (10taavi) 05Resolved→03Invalid [13:46:52] 10Toolforge (Software install/update): Update setuptools to 65.5.1+ on toolforge - https://phabricator.wikimedia.org/T347051 (10taavi) 05Open→03Invalid I'm closing all grid engine install/update tasks as the [[ https://wikitech.wikimedia.org/wiki/News/Toolforge_Grid_Engine_deprecation | grid engine is being... [13:56:47] 10Toolforge (Software install/update): Kubernetes containers do not include /usr/bin/mysql - https://phabricator.wikimedia.org/T239715 (10taavi) 05Open→03Declined In most cases I would recommend using a native mariadb client for the programming language used, and if that's not viable [[ https://wikitech.wiki... [13:59:54] 10Toolforge (Software install/update): Add build-essential to php Toolforge images - https://phabricator.wikimedia.org/T313550 (10taavi) 05Open→03Invalid The images that have `build-essential` installed are languages where the native package manager needs to be able to compile dependencies on demand, this is... [14:01:24] 10Toolforge (Software install/update), 10Kubernetes: toolforge-jobs: add logrotate - https://phabricator.wikimedia.org/T327165 (10taavi) 05Open→03Resolved [14:01:28] 10Toolforge, 10cloud-services-team, 10Epic: Provide modern, non-NFS error log solution for Toolforge webservices and bots - https://phabricator.wikimedia.org/T127367 (10taavi) [14:02:48] 10Grid-Engine-to-K8s-Migration: Migrate mathis-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319880 (10Mathis_Benguigui) Hi, I'm a bit surprised because I didn't receive notifications of this task. Will do it. [14:07:48] 10Grid-Engine-to-K8s-Migration: Migrate mathis-bot from Toolforge GridEngine to Toolforge Kubernetes - https://phabricator.wikimedia.org/T319880 (10Mathis_Benguigui) 05Open→03In progress p:05Triage→03High a:03Mathis_Benguigui [14:14:15] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [builds-cli,builds-api] Allow build service to cleanup images to free quota - https://phabricator.wikimedia.org/T341067 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/39 d/changelog: bump t... [14:14:27] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [builds-cli,builds-api] Allow build service to cleanup images to free quota - https://phabricator.wikimedia.org/T341067 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/157 builds-api:... [14:15:09] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [builds-cli,builds-api] Allow build service to cleanup images to free quota - https://phabricator.wikimedia.org/T341067 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/39 d/changelog: bump t... [14:15:18] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [builds-cli,builds-api] Allow build service to cleanup images to free quota - https://phabricator.wikimedia.org/T341067 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/40 fix bump version [14:16:36] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [builds-cli,builds-api] Allow build service to cleanup images to free quota - https://phabricator.wikimedia.org/T341067 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/41 d/changelog: bump t... [14:36:29] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [builds-cli,builds-api] Allow build service to cleanup images to free quota - https://phabricator.wikimedia.org/T341067 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/41 d/changelog: bump t... [14:45:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [14:45:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [14:47:55] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [builds-cli,builds-api] Allow build service to cleanup images to free quota - https://phabricator.wikimedia.org/T341067 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/42 clean: add user not... [15:31:29] 10Toolforge (Software install/update): toolforge: Install ffmpeg in kubernetes - https://phabricator.wikimedia.org/T243564 (10taavi) 05Open→03Declined I don't think ffmpeg should be in the shared base images. I recommend you [[ https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service | build a custom... [15:31:49] 10Toolforge, 10Kubernetes, 10Tracking-Neverending, 10User-bd808: Packages to be installed in Toolforge Kubernetes Images (Tracking) - https://phabricator.wikimedia.org/T140110 (10taavi) [15:32:01] 10Toolforge (Software install/update), 10Tools-Kubernetes, 10cloud-services-team, 10Kubernetes: Install DjVuLibre and XPDF packages for Kubernetes containers on Toolforge - https://phabricator.wikimedia.org/T166985 (10taavi) 05Stalled→03Declined This feels like a good use case for [[ https://wikitech.w... [15:32:39] 10Toolforge (Software install/update): Install headless browser dependencies on node16 image - https://phabricator.wikimedia.org/T316286 (10taavi) 05Open→03Declined This feels like a good use case for [[ https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service | a custom image ]] instead of growing t... [15:34:11] 10Toolforge (Software install/update): Install ImageMagick on Toolforge - https://phabricator.wikimedia.org/T343313 (10taavi) 05Open→03Declined This feels like a good use case for [[ https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service | a custom image ]] instead of growing the shared base images. [15:34:36] 10Toolforge (Software install/update), 10cloud-services-team: libnss3-dev in toolforge-node10-sssd-web Docker image? - https://phabricator.wikimedia.org/T271082 (10taavi) 05Open→03Declined This feels like a good use case for [[ https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service | a custom ima... [15:36:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [15:36:49] 10Toolforge (Software install/update): Request for supporting Deno on Toolforge - https://phabricator.wikimedia.org/T253470 (10taavi) Offering a deno buildpack on [[ https://wikitech.wikimedia.org/wiki/Help:Toolforge/Build_Service | the build service ]] might be an option if there's interest for that. [15:38:14] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [builds-cli,builds-api] Allow build service to cleanup images to free quota - https://phabricator.wikimedia.org/T341067 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/42 clean: add user not... [15:38:57] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [builds-cli,builds-api] Allow build service to cleanup images to free quota - https://phabricator.wikimedia.org/T341067 (10CodeReviewBot) dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/43 d/changelog: bump t... [15:40:19] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [builds-cli,builds-api] Allow build service to cleanup images to free quota - https://phabricator.wikimedia.org/T341067 (10CodeReviewBot) dcaro merged https://gitlab.wikimedia.org/repos/cloud/toolforge/builds-cli/-/merge_requests/43 d/changelog: bump t... [16:38:14] 10cloud-services-team (FY2023/2024-Q1-Q2), 10Cloud-Services-Origin-Alert, 10Cloud-Services-Worktype-Unplanned, 10User-dcaro: [toolsdb] MariaDB process is killed by OOM killer (December 2023) - https://phabricator.wikimedia.org/T353093 (10fnegri) cc-ing @Marostegui as I'm running out of ideas here. As descr... [16:47:18] 10Toolforge (Toolforge iteration 02): [maintain-harbor] Improvements to subcommands and config validation - https://phabricator.wikimedia.org/T353059 (10Raymond_Ndibe) a:03Raymond_Ndibe [16:48:02] 10Toolforge (Toolforge iteration 02): [ci] Add shellcheck to pre-commit where missing - https://phabricator.wikimedia.org/T353052 (10Raymond_Ndibe) a:03Raymond_Ndibe [16:51:28] 10Toolforge (Toolforge iteration 02): [tbs][builds-api] Refactor `internal/builds.go` - https://phabricator.wikimedia.org/T352762 (10Raymond_Ndibe) a:03Raymond_Ndibe [16:54:50] 10Toolforge (Toolforge iteration 02), 10User-Raymond_Ndibe: [tbs] Give a meaningful error message when a user exceeds their Harbor quota - https://phabricator.wikimedia.org/T351178 (10Raymond_Ndibe) [16:55:56] (ToolsToolsDBWritableState) firing: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [17:00:08] 10Toolforge (Toolforge iteration 02): [tbs] cleanup robot account related code - https://phabricator.wikimedia.org/T352763 (10Raymond_Ndibe) a:03Raymond_Ndibe [17:05:56] (ToolsToolsDBWritableState) resolved: There should be exactly one writable MariaDB instance instead of 0 - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/ToolsToolsDBWritableState - https://prometheus-alerts.wmcloud.org/?q=alertname%3DToolsToolsDBWritableState [17:09:29] 10Toolforge (Toolforge iteration 02), 10User-Raymond_Ndibe: alert users when they are about to exceed their harbor quota - https://phabricator.wikimedia.org/T353535 (10Raymond_Ndibe) [17:20:29] 10Cloud-VPS, 10cloud-services-team, 10ops-eqiad: cloudrabbit: connect them via cloudsw and cloud-private - https://phabricator.wikimedia.org/T345610 (10taavi) This can move forward now, although due to the nature of Rabbit this needs to be coordinated to avoid unnecessary downtime. It's fine if this goes to... [17:42:37] 10Toolforge Jobs framework, 10Pywikibot: Pywikibot image for Toolforge jobs does not capture output - https://phabricator.wikimedia.org/T353537 (10russblau) [17:45:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [17:45:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [17:51:29] 10Toolforge Jobs framework, 10Pywikibot: Pywikibot image for Toolforge jobs does not capture output - https://phabricator.wikimedia.org/T353537 (10taavi) p:05Triage→03High a:03taavi [18:01:19] 10Toolforge Jobs framework: Allow using file logs with build service images - https://phabricator.wikimedia.org/T353537 (10taavi) p:05High→03Medium [18:36:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [19:50:16] 10Toolforge Jobs framework, 10Patch-For-Review: Allow using file logs with build service images - https://phabricator.wikimedia.org/T353537 (10CodeReviewBot) taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/51 api: allow file logging on build service jobs with all mounts [19:50:27] 10Toolforge Jobs framework, 10Patch-For-Review: Allow using file logs with build service images - https://phabricator.wikimedia.org/T353537 (10taavi) The [[ https://wikitech.wikimedia.org/wiki/Help:Toolforge/Jobs_framework#Internal_log_storage | internal log storage ]] does function with the pywikibot image an... [19:51:28] 10Toolforge Jobs framework, 10Patch-For-Review: Allow using file logs with build service images - https://phabricator.wikimedia.org/T353537 (10CodeReviewBot) taavi opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/50 command: Always use absolute paths for file logs [19:56:25] 10Toolforge (Toolforge iteration 02), 10Toolforge Jobs framework, 10Patch-For-Review: Allow using file logs with build service images - https://phabricator.wikimedia.org/T353537 (10taavi) [19:57:46] 10Toolforge (Toolforge iteration 02), 10Toolforge Jobs framework, 10Patch-For-Review: Allow using file logs with build service images - https://phabricator.wikimedia.org/T353537 (10taavi) 05Open→03In progress [20:36:48] 10Toolforge: Tools violating the connection handling policy - https://phabricator.wikimedia.org/T353551 (10taavi) [20:37:41] 10Data-Services, 10cloud-services-team: Tools violating the connection handling policy - https://phabricator.wikimedia.org/T353551 (10taavi) [20:39:50] 10Tools: wdumps tool violates the Toolforge database connection handling policy - https://phabricator.wikimedia.org/T353553 (10taavi) [20:44:06] 10Tools: montage tool violates the Toolforge database connection handling policy - https://phabricator.wikimedia.org/T353554 (10taavi) [20:45:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [20:45:32] 10Tools: xlinks tool violates the Toolforge database connection handling policy - https://phabricator.wikimedia.org/T353555 (10taavi) [20:47:20] 10Tools: zhdeletionpedia tool violates the Toolforge database connection handling policy - https://phabricator.wikimedia.org/T353556 (10taavi) [20:50:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [20:51:14] 10Toolforge (Toolforge iteration 02): [envvars-cli] move pytest from tox to pre-commit - https://phabricator.wikimedia.org/T351476 (10Raymond_Ndibe) >>! In T351476#9395817, @dcaro wrote: > I think we might want to keep pytest outside of pre-commit, we kept it out for the reasons mentioned here: https://github.co... [20:57:10] 10Toolforge (Toolforge iteration 02), 10Patch-For-Review: [toolforge,gitlab] ensure we have a release before creating the mr on toolforge-deploy - https://phabricator.wikimedia.org/T353425 (10CodeReviewBot) raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/cicd/gitlab-ci/-/merge_requests/24 [toolf... [21:00:37] 10Tool-hitaden, 10Toolforge Build Service: Toolforge Build Service doesn't pass envvars to the buildpack - https://phabricator.wikimedia.org/T353557 (10Lofhi) [21:04:12] 10Toolforge (Toolforge iteration 02), 10cloud-services-team (FY2023/2024-Q1-Q2): [tbs] Create a tutorial on how to deploy a Node.js app using Build Service - https://phabricator.wikimedia.org/T353313 (10Lofhi) >>! In T353313#9407391, @bd808 wrote: >>>! In T353313#9406261, @Lofhi wrote: >> I don't know where to... [21:10:19] (HAProxyBackendUnavailable) firing: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [21:15:19] (HAProxyBackendUnavailable) resolved: HAProxy service neutron-api_backend backend cloudcontrol1007.private.eqiad.wikimedia.cloud is down - https://wikitech.wikimedia.org/wiki/HAProxy - TODO - https://alerts.wikimedia.org/?q=alertname%3DHAProxyBackendUnavailable [21:23:02] 10Tool-hitaden, 10Toolforge Build Service: Toolforge Build Service doesn't pass envvars to the buildpack - https://phabricator.wikimedia.org/T353557 (10dcaro) Do you have the logs of the build when it did not set the environment variables? (or the tool/build id) That would help debugging it. [21:23:14] 10Tool-hitaden, 10Toolforge Build Service: [buildservice] Not passing envvars to buildpack (or atleast to the nodejs one) - https://phabricator.wikimedia.org/T353557 (10Lofhi) [21:28:25] 10Tool-hitaden, 10Toolforge Build Service: [buildservice] Not passing envvars to buildpack (or atleast to the nodejs one) - https://phabricator.wikimedia.org/T353557 (10Lofhi) Here you go: P54485 [21:36:03] (InstanceDown) firing: Project toolsbeta instance toolsbeta-bastion-6 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [21:37:30] 10Toolforge (Toolforge iteration 02): [builds-cli] delete --all gets 0 builds to delete - https://phabricator.wikimedia.org/T353519 (10Raymond_Ndibe) a:03Raymond_Ndibe [21:39:47] 10Tool-hitaden, 10Toolforge Build Service: [buildservice] Not passing envvars to buildpack (or atleast to the nodejs one) - https://phabricator.wikimedia.org/T353557 (10dcaro) I can see the preparation script putting the environment variables in the platform directory: ` [step-prepare] 2023-12-15T18:39:21.3760... [21:53:08] 10Tool-bridgebot: Figure out how to deploy ZNC using buildpacks - https://phabricator.wikimedia.org/T353559 (10bd808) [21:55:26] 10Tool-hitaden, 10Toolforge Build Service: [buildservice] Not passing envvars to buildpack (or atleast to the nodejs one) - https://phabricator.wikimedia.org/T353557 (10dcaro) I was able to add the execution of `env` in one of the other buildpacks, and the variables are loaded correctly in the environment: ` [... [22:25:16] 10Tool-hitaden, 10Toolforge Build Service: [buildservice] Not passing envvars to buildpack (or atleast to the nodejs one) - https://phabricator.wikimedia.org/T353557 (10dcaro) By adding a build script to my package.json, I was able to run `env` as part of the nodejs buildpack, and the environment variables are... [22:28:18] 10Tool-hitaden, 10Toolforge Build Service: [buildservice] Not passing envvars to buildpack (or atleast to the nodejs one) - https://phabricator.wikimedia.org/T353557 (10dcaro) I get the same behavior using latest `heroku/builder:22` builder and the `pack` tool, the vars are there, but the buildpack still uses... [22:31:55] 10cloud-services-team (FY2023/2024-Q1-Q2), 10Cloud-Services-Origin-Alert, 10Cloud-Services-Worktype-Unplanned, 10User-dcaro: [toolsdb] MariaDB process is killed by OOM killer (December 2023) - https://phabricator.wikimedia.org/T353093 (10Andrew) Looks like there are quite a few memory issues that have been... [22:32:23] 10Tool-hitaden, 10Toolforge Build Service: [buildservice,nodejs] nodejs buildpack does not take envvars into account - https://phabricator.wikimedia.org/T353557 (10dcaro) [22:44:04] 10Tool-hitaden, 10Toolforge Build Service: [buildservice,nodejs] nodejs buildpack does not take envvars into account - https://phabricator.wikimedia.org/T353557 (10dcaro) Opened https://github.com/heroku/heroku-buildpack-nodejs/issues/1188 upstream, let's see how it goes :) [22:46:14] 10Toolforge (Toolforge iteration 02), 10User-Raymond_Ndibe: [toolforge-cd] find out why we run two gitlab ci/cd pipelines after merge - https://phabricator.wikimedia.org/T353563 (10Raymond_Ndibe) [22:46:30] 10Toolforge (Toolforge iteration 02), 10User-Raymond_Ndibe: [toolforge-cd] find out why we run two gitlab ci/cd pipelines after merge - https://phabricator.wikimedia.org/T353563 (10Raymond_Ndibe) a:03Raymond_Ndibe [22:46:46] 10Toolforge (Toolforge iteration 02), 10User-Raymond_Ndibe: [toolforge-cd] find out why we run two gitlab ci/cd pipelines after merge - https://phabricator.wikimedia.org/T353563 (10Raymond_Ndibe) 05Open→03In progress [22:47:50] 10Toolforge (Toolforge iteration 02), 10User-Raymond_Ndibe: [toolforge-cd] find out why we run two gitlab ci/cd pipelines after merge - https://phabricator.wikimedia.org/T353563 (10Raymond_Ndibe) [22:47:56] 10Tool-hitaden, 10Toolforge Build Service: [buildservice,nodejs] nodejs buildpack does not take envvars into account - https://phabricator.wikimedia.org/T353557 (10Lofhi) A nice xmas-gift I see! Anyway, I can't go any further on the subject until next week, there's no hurry. Thanks for the expertise! [22:49:17] 10Toolforge (Toolforge iteration 02), 10User-Raymond_Ndibe: [toolforge-cd] find out why we run two gitlab ci/cd pipelines after merge - https://phabricator.wikimedia.org/T353563 (10Raymond_Ndibe) [22:49:21] 10Toolforge (Toolforge iteration 02): [toolforge-cd] gitlab-ci refactor - https://phabricator.wikimedia.org/T353514 (10Raymond_Ndibe) [22:49:27] 10Tool-hitaden, 10Toolforge Build Service: [buildservice,nodejs] nodejs buildpack does not take envvars into account - https://phabricator.wikimedia.org/T353557 (10taavi) > Context: I am using pnpm. Problem: Heroku's builder-22 doesn't support any buildpack PNPM compatible. I know that it is planned to support... [22:52:51] 10Toolforge Build Service, 10cloud-services-team, 10Documentation: Build service admin documentation is lacking on how build process works - https://phabricator.wikimedia.org/T353566 (10taavi) [22:57:15] 10Toolforge Build Service, 10cloud-services-team, 10Documentation: Build service admin documentation is lacking on how build process works - https://phabricator.wikimedia.org/T353566 (10taavi) It seems like the buildpack versions are embedded in [[ https://github.com/heroku/cnb-builder-images/blob/main/build... [22:58:49] 10Tools: xlinks tool violates the Toolforge database connection handling policy - https://phabricator.wikimedia.org/T353555 (10Count_Count) Oversight on my part, should be fixed now. Please confirm. [23:02:08] 10Data-Services, 10cloud-services-team: Tools violating the connection handling policy - https://phabricator.wikimedia.org/T353551 (10taavi) [23:02:13] 10Tools: xlinks tool violates the Toolforge database connection handling policy - https://phabricator.wikimedia.org/T353555 (10taavi) 05Open→03Resolved a:03Count_Count ` MariaDB [(none)]> select user, count(*) from information_schema.processlist where user = 's55543' group by user; Empty set (0.001 sec) `... [23:03:06] 10Tool-hitaden, 10Toolforge Build Service: [buildservice,nodejs] nodejs buildpack does not take envvars into account - https://phabricator.wikimedia.org/T353557 (10dcaro) >>! In T353557#9410507, @taavi wrote: >> Context: I am using pnpm. Problem: Heroku's builder-22 doesn't support any buildpack PNPM compatibl... [23:04:41] 10Toolforge Build Service, 10cloud-services-team, 10Documentation: Build service admin documentation is lacking on how build process works - https://phabricator.wikimedia.org/T353566 (10bd808) There is a bunch of stuff in https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Previous_Efforts/Toolforge_Build_S... [23:05:33] 10Toolforge Build Service, 10cloud-services-team, 10Documentation: Document how build service build process pulls in buildpacks and how to update them - https://phabricator.wikimedia.org/T353566 (10taavi) [23:06:01] 10Toolforge Build Service, 10cloud-services-team, 10Documentation: Document how build service build process pulls in buildpacks and how to update them - https://phabricator.wikimedia.org/T353566 (10taavi) [23:06:03] 10Tool-hitaden, 10Toolforge Build Service: [buildservice,nodejs] nodejs buildpack does not take envvars into account - https://phabricator.wikimedia.org/T353557 (10taavi) [23:08:24] 10Toolforge Build Service, 10cloud-services-team, 10Documentation: Document how build service build process pulls in buildpacks and how to update them - https://phabricator.wikimedia.org/T353566 (10taavi) https://wikitech.wikimedia.org/wiki/Wikimedia_Cloud_Services_team/EnhancementProposals/Decision_record_T... [23:33:54] 10Cloud Services Proposals, 10Toolforge Build Service (Iteration 11): Decision request - What buildpacks to allow and include for toolforge build service beta - https://phabricator.wikimedia.org/T330102 (10taavi) Was a custom builder actually ever implemented? If not, do we want to re-visit that decision or sh... [23:35:02] 10Toolforge Build Service, 10cloud-services-team, 10Documentation: Document how build service build process pulls in buildpacks and how to update them - https://phabricator.wikimedia.org/T353566 (10taavi) Ok, as far as I can tell that was never implemented. (Left a comment on that task asking if we still wan... [23:35:28] 10Toolforge Build Service, 10cloud-services-team, 10Documentation: Document how to update heroku-builder:22 container image - https://phabricator.wikimedia.org/T353566 (10taavi) [23:45:04] (TfInfraTestDestroyFailed) firing: Terraform failed to destroy the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestDestroyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestDestroyFailed [23:50:04] (TfInfraTestApplyFailed) firing: Terraform failed to apply/create the resounces on tf-bastion - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/TfInfraTestApplyFailed - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTfInfraTestApplyFailed [23:58:15] 10Cloud Services Proposals, 10Toolforge Build Service (Iteration 11): Decision request - What buildpacks to allow and include for toolforge build service beta - https://phabricator.wikimedia.org/T330102 (10dcaro) We discovered how to add custom buildpacks to the upstream builder, so we changed to use the upstr...